The last decade has seen substantial technological progress in a wide range of fields, and one of those is the scalable object storage of Amazon S3 that heralded the arrival of AWS. First launched on March 14, 2006, the secure and reliable storage service has certainly evolved. In this post on All Things Distributed, Amazon.com CTO Werner Vogels discusses 10 lessons from the last 10 years.
Firstly, Vogels highlighted the importance of building evolvable systems right from the start, helping the process of upgrading the architecture without disrupting the service. This foresight also led the team to expect the unexpected and prepare for the many possible system failures that could occur, even those they couldn’t anticipate, by intelligently managing the “blast radius” of failure occurrences to maintain overall system health.
Vogels also mentioned the importance of accommodating the unknown needs of users as the system’s use cases diversified by providing a collection of primitives and tools to develop bespoke systems, rather than rigid frameworks. The team realised the importance of automation and built management APIs that control key operational functions, extending these to serve customers who can define their own automation rules. Once launched, APIs could not be changed as this would impact customer operations, so careful consideration went into designing them.
Developing the resource usage and charging model was an important phase and it was vital to strike the right balance, especially since the service was operating at scale with low margins. Respecting the trust of these many customers could not be an afterthought, and so security was fundamental to development by bringing the security team in from day one.
The S3 team took several years to integrate proper encryption once appropriate tools became available, eventually turning over encryption key management to customers through Amazon KMS. The diversity of AWS’s deployment is supported by the team’s unique innovations around datacenter layout and operations, providing vital flexibility of the network infrastructure.
Finally, AWS benefits from a lack of gatekeepers that ensures the ecosystem keeps innovating and evolving as customers design and provide new services that often feed back into the ecosystem. With these lessons learned in the first 10 years, AWS has a solid foundation from which to continue evolving over the next 10.