Presentation: Secrets at Planet-Scale: Engineering the Internal Google KMS

We propose to discuss Google’s internal key management system for cryptographic key material which is a critical part of Google's overall strategy for user data protection. The talk will cover the design choices and strategies that Google chose in order to build a highly reliable, highly scalable service. The talk will close with continued maintenance pain points and suggested practices for your own internal key management service.  

This internal KMS underlies most storage, authentication, cross-site scripting forgery, and other critical security systems at Google, and hence needs to have very high availability. Furthermore, Google’s internal KMS not only manages the generation, distribution and rotation of cryptographic keys, but it also manages other secret data. Google’s internal KMS serves a massive volume of queries, more per second than Gmail or any single Google service, and needs to be very reliable in order to do so, historically performing at more than 99.9999% availability.  

The design choices that favored high availability have caused a few pain points for our clients. An example is the delay introduced between clients updating their keys/configs and the changes being reflected in production. For many of the system’s clients this delay is too long. We’ll discuss this and other pain points, and how we’re improving the user experience.