https://outerbounds.com/ logo
Join Slack
Powered by
We're migrating our metaflow install from one k8s ...
# ask-metaflow
p
We're migrating our metaflow install from one k8s cluster to another, and I had two related questions about if they could share some of the backing services (s3, postgres)
✅ 1
So more specifically, are there any known or anticipated problems with having two identical metaflow-services using the same s3 bucket for artifact storage? I see they're namespaced by flow then run_id, so my assumption was as long as we didn't run the same flows in two places, or made sure they didn't overlap run-ids we'd be ok
Then in a similar line of questioning, can two metaflow installs temporarily share the same postgres db/tables?
a
the metaflow service doesn't interface with s3
p
This is just to let us slowly ease over flow by flow during the migration, not as a desired steady state running both in parallel
a
so if you have N metaflow services and M buckets - all N x M combinations are valid
thankyou 1
p
Got it, but the client flows managed by a metaflow service would coordinate with that s3 bucket, still?
a
the service doesn't manage any flow execution. it's only responsible for cataloging executions.
p
I think I understand.
I think that answers my s3 sharing question. As for the other part, would having two metaflow-service setups pointing to the same postgres instance and table temporarily present any conflict?
a
that should be okay - by default we recommend running multiple replicas of the metaflow service
p
Got it, thank you. I'll take my unrelated question to a new thread.
👍🏼 1
6 Views
Open in Slack
PreviousNext
Powered by