AWS Secrets Manager Agent launched!

My take on this new agent that complements Secrets Manager and where it still falls a bit short...

Secret Manager

Finance sector in general, as you can imagine, is very paranoid about credentials storing and rightfully so. To mitigate this issue and only use encrypted store for credentials, using AWS Secrets Manager is one way to tackle the problem. It has been de facto standard for storing credentials for many companies, including ours.

Problems

But here come the problems. So far using secrets manager has resulted in few issues to say the least. Calling Secret Manager API directly from your code was the way it was suppose to be used and if you have few secrets, that might even float the boat. But as soon as you have 100s of credentials and a huge infrastructure, you run into several issues:

• Slow boot of an application - Calling API to load these secrets and so on, it just precious time to the boot up of application and we hate that.

• API rate limits - Even if high by AWS specification, you need to multiply amount of credentials by the amount of servers and this number all of the sudden gets pretty high.

• Costs - Imagine continuous deploys and autoscaled infrastructure where servers are all the time going up and down and being deployed and so on… these calls get costly after a while.

• Secrets Manager not being available - Low probability, but still… Ouch. You better not restart anything and immediately stop all autoscaling processes… :)

AWS did introduce bulk loading of secrets, which kinda helped with the situation, but still doesn’t solve all of the problems.

Secret Manager Agent

One way to deal with the situation is running some kind of an agent next to your application and here comes AWS Secrets Manager Agent. You can find it here: https://github.com/aws/aws-secretsmanager-agent

What it does is, it basically caches secrets for application to have them available immediately. This solves majority of the above listed problems.

To list at least few features that are significant improvement vs directly calling Secrets Manager API:

• Caches secrets (d0h!)

• In-memory storage of these secrets - thats good, cause it doesn’t store them on disk and they are immediatly available

• Ability to expire secrets after a while and re-retrieve them from Secrets Manager

• Interface to communicate with Agent instead of Secret Manager directly

For most of AWS Cloud infrastructures this is good enough. Unfortunately not for all. There are several issues where Agent still falls short:

• Doesn’t do cache invalidations - It’s a pull method upon TTL expiry. It doesn’t know that a secret changed and it should go re-fetch it and expose new value to the app.

• Secrets values in memory are not encrypted - Scrubbing memory and trying to protect credentials from memory dump attacks is hard.

• It fetches secret by secret, doesn’t do bulk load.

• Restart of Agent will wipe out all cached secrets and it will need to re-fetch them - Arguably this is good thing, since this means secrets are for sure refreshed and this way you can force cache invalidation.

Summary

Unfortunately this Agent came out a little late for us, to be using it, because we already implemented our own, called Secrets Caching Service or SCS in short. For now we will stick to our implementation, since SCS actually supports push method, grouping of secrets (pushing only a group of secrets in bulk to a SCS that is running on server) and utilizing Redis pub/sub functionality to notify appropriate servers that they should pull new secrets (because one of them changed in Secret Manager for example).

Disregarding the above, I still think majority of companies that are using AWS Secret Manager, should use Secret Manager Agent from now on. It’s just a no-brainer and a great improvement in secrets management in general. Great work AWS, I hope there will be additional features coming to it soon…