Zapier is the Web app automation service that allows even non-technical users to define custom workflows, or ‘Zaps’, that integrate multiple apps to ease the burden of repetitive tasks. In this guest post on the High Scalability site, Zapier Co-founder and CTO Bryan Helmig explains how his team built the stack to handle over 60m API calls every day.
Helmig admits that they used some fairly standard tools, laying out the basic architecture in under 54 hours of a Startup weekend. While Zapier has enhanced capabilities since then, their approach to keeping things simple continued
With the basic structure in place, Helmig and his team decided to focus on application design to support horizontal scalability, as well as redundant infrastructure to ensure high availability. A key decision early on was to stick with technologies that the team was already familiar with, allowing them to use the same solutions multiple times.
As an example, large chunks of the primary database’s IO and storage was being consumed by session data in a key/value arrangement. Rather than apply the best tool for the job (they were considering Cassandra or Riak), the team chose to stand up a dedicated MySQL instance with a single session table. This decision was based on the system’s existing support for MySQL, its ability to handle simple key/value storage, and the team’s familiarity with it.
Helmig’s team also designed the application to easily handle horizontal scaling by using MySQL read-only replicas as the primary touch point, with replica lag concerns nullified by the static nature of Zaps. They also improved reliability by designing for failure from the beginning by incorporating auto-scaling groups with auto-replacement, allowing instances to be killed since they will be replaced swiftly.
This reliability and scalability mean that new technologies and capabilities, such as the upgrade from two-step Zaps to multi-step Zaps, can be added to the platform without issue. Through this architecture, the platform currently handles over 60m API calls as Zapier automates over 8m tasks every day.