Screen Scraper Mitigation and Dogfooding Included in Riot Games' Textbook API Justifications

Riot Games senior software engineer Leigh Estes (@schmik) has published a three part series (more parts could be coming) regarding the League of Legends publisher's ongoing digital transformation to a microservices-driven software infrastructure. The company's journey involves some of the most textbook business and technical justifications for rolling out its Web APIs and serves as a great case study for others looking to sell such a strategy up the foodchain. 

One technical reason (one that we at ProgrammableWeb have seen come up before) has to do with mitigating the effects of screen scraping or what we in the biz sometimes call "ScrAPI" (pronounced scrape--pee--eye). Many websites that lack an API for getting at the valuable information on their pages will often find that their pages are getting heavily crawled by screen scrapers. These screen scrapers are often built by developers who, in the absence of an API, build their own scripts for parsing through every page and extracting the meaningful data. In essence, it's an API albeit an unofficial one; thus ScrAPI. But such screen scraping can negatively impact a website. Wrote Estes:

"During the first several years after the launch of League of Legends, fan sites and stats services obtained data by scraping Riot websites or imitating client traffic to Riot servers. Inefficient scraping methods increased the load on our server platform, which in turn put live services at risk. This sort of traffic caused a degradation of players’ experience....We needed to find a solution, and a public API seemed like the right one. When starting the API program, we had several specific goals [one of which was to] protect our live services by offering legitimate and regulated access to League of Legends data...Third-party developers have embraced and adopted the API as the de facto standard for obtaining Leagues of Legends data, and scraping is no longer an issue for our live services."

Another classic reason that Riot's APIs are paying off has to do with the company's dogfooding of them, as more of its own developers are making use of them. Here on ProgrammableWeb and across the API economy, many advocates of APIs and services-oriented infrastructures like to talk about the need for business agility in a fast-changing market. In a traditional monolithic software model, making changes to code or functionality can take forever because you sometimes have no idea what else -- particularly something else that's mission critical -- will break. In a services-oriented model, all functionality is provided and consumed through services-oriented interfaces like APIs. Changes are easier and faster to make so long as the API's contract (the expectation of how to work the API from the consuming application side) doesn't change. As we like to say, such loose coupling can separate a lot of concerns

But another form of agility achieved by Riot by moving to an API-led strategy had to do with the simple problem of Riot's own developers getting programmatic access to the resources they need to get their jobs done. Without APIs, getting access that that dataset living on that server in some other satellite data center can require an act of the administrative Gods. But now, with APIs, the process for getting such access is greatly simplified. Estes continued by stating:

"Most public API programs are focused on external collaboration, but we've found plenty of cases in which collaboration between developers and teams internally at Riot has been improved. Simply having a reliable registry of available APIs in the developer portal helps Riot software engineers onboard more quickly and in some cases has started conversations about accessing data that a team may not have otherwise known was available. The edge layer makes it possible for more teams to easily gain access to those APIs.  Our global offices, in particular, have taken advantage of the public API, using it as a fast, reliable way to access player data without needing to navigate internal technical hurdles and team structures."

The second and third parts of Estes' series go into greater detail, explaining how Riot standardized on some of Netflix's open source software like Zuul Proxy Server and how it optimized that software to deliver the necessary scale and performance to support the company's many customers (all those gamers!).  Here are the different parts to Estes' series: Part 1, Part 2, Part 3

Original Article

The Riot Games API: Goals and Design

David Berlind is the editor-in-chief of ProgrammableWeb.com. You can reach him at david.berlind@programmableweb.com. Connect to David on Twitter at @dberlind or on LinkedIn, put him in a Google+ circle, or friend him on Facebook.
 

Comments

Comments(1)