It's all the more pathetic that they responded to the first hack by "obfuscating" the client-side secret, as if that could even theoretically stop any attacker with a budget of more than about an hour.

Future service designers: if your client is talking directly to AWS, then your attacker will, too. Take the week to write a CRUD frontend server that enforces the policy you want.

Do you have a recommended security checklist for something like this? I remember seeing an old github repo with a bunch of good information but I cannot seem to find it and my search results are... unhelpful at best.

Checklists are cool and all. But, you shouldn't need a checklist to know that you shouldn't ship API keys.

Can't disagree with that.

Check out OWASP cheatsheets and guides:

https://www.owasp.org/

You wont like it, but... How would you feel being asked for a checklist for 'good software development' or something similar by a cooking chef? Hire skilled and dedicated DevOps/SRE/SysAdmin/hype-of-the-week person. ;)

100% this. I’ve been transitioning to doing more devops/admin work for the past several months and it’s amazing the stuff people are willing to put into production. Like no, maybe we shouldn’t be committing that secret to Github. Dedicated operations critters are essential for any kind of service.

A chef cooking something unfamiliar will look up the dish they are cooking. If your cookbook can't handle a checklist of common pitfalls and overlooked actions to take then it isn't much of a cookbook. (granted not putting your secrets in a public repo is definitely a given.)

This is a common issue, and many apps do this mistake.

Another common mistake is having /.git/ available on the domain itself, often with PHP sites or backend-less SPAs this is common, giving full access to the source, including those API keys. Even major sites do this – The Hill until recently had their git repo, including API tokens and access keys for everything, publicly available.

It should be mentioned that none of that should ever make its way into a Git repo in the first place. If a secret is committed to Git, it's compromised, period. Suck it up and generate a new secret.

Agreed ... use a pre-commit hook to scan your repository for high-entropy strings before they are forever enshrined in your history (https://github.com/dxa4481/truffleHog).

To be fair, though, history can be rewritten, albeit sometimes with some difficulty.

If a secret has ever been in Git then you probably can't know where it's been copied to and should treat it as likely to have been leaked.

Great idea! Thanks for linking to that tool.

No, you should never commit your secrets, not even to a private github repo, and not even to a privately hosted git server. Doing so increases your attack surface, sometimes in surprising ways.

Now, if your secrets are encrypted before being committed (using something like ansible vault) and the encryption key is not stored in the repo, that may be ok. However, you still need to be aware that any time you rotate that key, you also should to rotate every secret hidden behind that key.

So how do you do version the configuration management for the cluster that runs everything? It needs to be versioned, and yet with just the data in it you need to be able to recreate your entire environment from scratch.

If you want your secrets under version control, then they should be encrypted.

There are lots of ways of doing this. git-crypt is one that is configuration management agnostic. Most CM tools have their own way of dealing with secrets and there are CM agnostic tools like git-crypt. I'm not personally familiar with anything besides ansible-vault, but this article seems to provide a pretty good summary of several options: https://www.threatstack.com/blog/cloud-security-best-practic...

> Another common mistake is having /.git/ available on the domain itself, often with PHP sites or backend-less SPAs this is common

Yeah and same for .svn and .cvs (which may happen to include your VCS credentials in that directory, too).

I can publish my public key all day long for encryption into a back-end ... and later I can only read it with my private key. So they may not have used ROT13 but it certainly looks like they used something symmetrical.

This is not actually about an encryption key. Better to think about it as a password to access the server.