Despite currently boasting an architecture that has a reputation for flexibility, continuous experimentation, and regular daily deploys, Etsy’s system was suffering from significant performance problems just a few years ago. In this article on InfoQ, Thomas Betts discusses Etsy software engineer Stefanie Schirmer’s talk at QCon New York 2016 about how and why the company moved to an API-first architecture.
The need arose from two different projects that both required a new design philosophy without alienating the development team from the API. One was the need for increased extensibility from the platform to facilitate new features for mobile apps, and the other was the team’s attempts to overcome the performance limitations of PHP for allowing concurrent API calls.
The Etsy team adopted a pattern similar to that used by Netflix and created a two-layer API using meta-endpoints that are aggregations of several other endpoints. This approach allows the composition of low-level, general-purpose resources into view- or device-specific resources on the server side.
The full stack is a multi-level tree with a customer-facing bespoke homepage tailored to a specific view, and this consumes a layer of concurrent meta-endpoints that reduce the complexity for composing these bespoke views. These meta-endpoints call atomic component endpoints which make up the only level that can talk to the database.
The team then used cURL to enable parallel HTTP calls to improve performance, with visualization tools for locating trouble areas as requests move through the framework. The first automatically-generated API clients simplified the learning curve by being accompanied by documentation and already knowing the parameters for each endpoint.
The company then took a dynamic approach to training that involved lunch and learn workshops, pilot groups, code labs, and pairing with experienced developers. Schirmer and her colleagues are maintaining open conversations with development teams to ensure the framework continues to evolve in a way that benefits everyone.