Hi all, I'm currently exploring the deployment of...
# ask-metaflow
e
Hi all, I'm currently exploring the deployment of Metaflow on an AWS EKS cluster using your Helm charts. Over the past few days, I've encountered some questions while trying to understand Metaflow's deployment on Kubernetes and configuring it for optimal functionality: 1. Securing RDS Credentials: 2. To make the deployment production-ready, I'm considering using RDS for a PostgreSQL database. However, I'm unsure how to securely inject credentials as a Kubernetes secret into the Helm chart. The values.yaml file in the Helm chart appears to have hardcoded environment variables. Since I plan to host the Helm chart on a GitHub repo for automatic deployment via ArgoCD, storing plain-text credentials for the RDS database is not an option, even if it's a private RDS cluster. 3. S3 Access for Metaflow: 4. Metaflow requires access to an S3 bucket. I assume this necessitates IAM permissions (perhaps injecting those via service account's annotations?). Is there documentation available on how to grant these permissions to Metaflow? 5. Deploying Metabase on EKS: 6. Additionally, I'm interested in deploying Metabase on an EKS cluster with an S3 bucket and RDS database to ensure it's production-ready in terms of security, autoscaling, etc. Is there existing documentation or recommended practices for achieving this? I'm more than willing to contribute by opening a PR for any necessary Helm changes. Thank you for your support in maintaining this project and delivering such an excellent product!
a
For RDS credentials you could store them in AWS Secrets Manager and maybe use something like External Secrets operator to expose them as a K8s secret, which in turn could be injected via
envFrom
. I think there is already support for envFrom in the helm chart but you'd need to disable env vars that are set directly on the pod. IIRC that'd need a change to helm chart
That helm chart is intended more as a starting point, it doesn't support all possible deployment configurations today
for S3 access yes, the best way is to use IRSA. We don't have a Metaflow specific doc for this yet, but the set up is pretty similar to what you'd do for any kubernetes workload using IRSA
e
Thanks Oleg for the explanation. You have pushed me in the right direction. The current Helm chart does not allow db secrets to be injected via envFrom, but it does give me a starting point.
👍 1