Presentation: With Great Scalability Comes Great Responsibility
Abstract
This is a story of how I took down one of our vendor’s services with an innocent serverless application. I wanted to retrieve data from one of our monitoring platforms to analyze SPS Commerce’s software performance. Initially, I wrote a script to collect the data using python multiprocessing. To gather this data in a more scalable, fast, and efficient way, I decided to pivot to a serverless architecture. Unfortunately, my solution ended up spawning requests faster than the REST API could handle. In this talk, we will cover the contextual pros and cons of a number of architectural patterns given real world scalability constraints; from orchestrating Lambdas with AWS step functions to multiprocessing with S3 triggers to rate limiting with queues like SQS.
Similar Talks
Stateful Programming Models in Serverless Functions
![Qcon](https://archive.qconsf.com/sites/qcon_sf/files/styles/similar_talks/public/profile_2013_400.jpg?itok=33G0EO-o)
Principal Engineering Manager @Microsoft, helping lead the Azure Functions Team
Chris Gillum
User & Device Identity for Microservices @ Netflix Scale
![Qcon](https://archive.qconsf.com/sites/qcon_sf/files/styles/similar_talks/public/0_14.jpeg?itok=SnAyVkDL)
Senior Software Engineer in Product Edge Access Services Team @Netflix
Satyajit Thadeshwar
Secrets at Planet-Scale: Engineering the Internal Google KMS
![Qcon](https://archive.qconsf.com/sites/qcon_sf/files/styles/similar_talks/public/p1050851.jpg?itok=DWShAxQ7)
Software Developer @Google
Anvita Pandit
Architectures That Scale Deep - Regaining Control in Deep Systems
![Qcon](https://archive.qconsf.com/sites/qcon_sf/files/styles/similar_talks/public/bhs_headshot_2019.jpg?itok=J1YqqWfN)
CEO and co-founder @LightStepHQ, Co-creator @OpenTracing API standard