Hi all, I have integrated automated password rotation to secretsmanager which now RDS and ECS are pulling from. Unfortunately, after I do a new terrafrom apply and read in the changed secret from secretsmanager, the ECS services (metadata backend + UI) fail. I thought just rotating the password in secretsmanager and propagating the updated passwords to the resources that consume it would suffice, but is there an additional thing I need to do?

did you use the new RDS <> secrets manager integration? or the old style where you rotate secrets via lambda and update RDS in your lambda?

The old way - update RDS via lambda. Yes you're correct. I would have to do another

terraform apply

after the rotation to get the ECS task to re-read the secret. That's another thing too - passing the password via environment variables is not very secure. It's even visible from the console. Are there are other options?

yes! i was going to suggest that. Its not too bad in this case since RDS instance is not available outside from networking perspective, so we essentially rely on network perimeter security there to limit access, and to simplify open source terraform template. But to harden it further makes sense to use SecretsManager reference in ECS task.

Perfect. I'll give it a go. Thanks Oleg!