https://outerbounds.com/ logo
Join Slack
Powered by
Hey outbounds folks :wave: At Attentive today we ...
# ask-metaflow
b
Hey outbounds folks 👋 At Attentive today we orchestrate our workflows with step functions and execute our workflows with AWS Batch. We’d like to move to orchestrating with Argo workflows, but we aren’t sure if we can still execute the compute on Batch. Would we need to move to executing the workflows on k8s if we move to Argo as our workflow engine?
✅ 1
v
yep, it'd be all K8s-native then
happy to show you a demo how the whole setup works on the OB platform 🙂
b
Cool cool that makes sense Question for you @straight-shampoo-11124: are there any examples of deploying and managing deployments with argo CD and argo workflows? I saw the below, but it seems like it only deploys the argo workflows chart https://github.com/outerbounds/terraform-aws-metaflow/tree/v0.10.0/examples/eks_argo
v
I haven’t heard of Argo CD examples but deploying from GitHub actions works well
m
What part of the tech stack are you looking to deploy via ArgoCD?
b
Hey @mammoth-rainbow-82717 - thx for responding! Argo workflows is what we would like to deploy w/ argo cd. Ideally we’d also manage Metaflow service versioning w/ Argo CD as well. I do see some helm charts available, but they’re in alpha https://github.com/outerbounds/metaflow-tools/tree/master/k8s/helm/metaflow
m
Argo Workflows you can deploy via ArgoCD, no probs. We do it through Kustomize atm. There is a Helm chart too, but from what I remember the Helm chart is not officially supported by the Argo team. That is the reason we use Kustomize for that install.
👀 1
✅ 1
The CNCF slack group is probably your best source of information on this front. ArgoCD/Argo Workflow channels are good places to start
b
CNCF?
m
Cloud native computing foundation - https://www.cncf.io/
By 
Metaflow service versioning
you mean the Metaflow service?
b
ah i mean like updates for the metaflow service image etc
as opposed to versioning datasets etc
m
We have the 
metadata_service
and 
ui_backend_service
as two separate services. We've installed them on ECS though and on EKS. Mainly because of our internal infrastructure. We use AWS API gateway to manage access to ECS from EKS.
Don't see a reason why you couldn't install them as Kubernetes services via ArgoCD though
👀 1
How you set up external access to the services, e.g., the UI would depend on your company infrastructure, I guess
b
Antoher question for you folks @mammoth-rainbow-82717 / @straight-shampoo-11124 I noticed in the metaflow architecture diagram there is a DDB table for SFN workflows If we’re migrating to argo workflows, do we need to backfill that DDB table into Argo / RDS somehow? Or would that be taken care of when we re deploy the flows? https://github.com/outerbounds/terraform-aws-metaflow/tree/v0.10.0
m
You should be able to use the same database, I believe
You should be able to have your Metaflow service wherever you want, it doesn't need to be in k8s just because you are going to run your workflows in k8s.
b
yeah totally, but if it is in k8s its infra policies and management are consistent with the rest of our infra
m
I see. You are using aroura for the RDS instance atm?
You can set up access to RDS instances from k8s too, if you need to do that for Metaflow, as the service sits outside k8s, but we do it for other things.
b
yeah we are - honestly was considering just keeping the db in rds as that’s how most our dbs are today
m
I see. I would do that too, personally.
b
There is some demand internally for us setting up our own manifests for the services tho
That way we can handle updates to the images with argo cd instead of terraform, which would be consistent w/ the rest of our update patterns
m
Yeah, I think you would need to do that anyway if you want to install the metaflow service in k8s as its own service.
At least, I've not seen any template/example for k8s deployments of the metaflow service.
b
Yeah, I was digging around for one yesterday & did not have much luck. I did see some charts that are in alpha/unsupported as you mentioned though. Shouldn’t be crazy hard to deploy though
m
nah, it should be pretty easy, I think
a
@broad-umbrella-22770 you wouldn’t need that DDB table in the kubernetes universe. The DDB table is an implementation detail for the SFN integration.
🙌 1
m
ah, my bad. I thought he was referring to the RDS for the metadata service.
b
Thanks! @square-wire-39606 / @straight-shampoo-11124 / @mammoth-rainbow-82717 - One consideration we have is around the state of Metaflow post migration (RDS DB & s3 in particular) We were thinking we will use the same RDS DB & s3 buckets when we transition workflow orchestration engines from SFN to Argo, but I know that could be hairy depending on how Metaflow stores past workflow state We were thinking about just ripping it in dev & seeing what happened but we’re blocked on the new cluster, so figured I’d ask here 1. Do you folks foresee problems using the same state (DB/S3) cross-orchestration engine? One thing I can imagine could be problematic would be viewing historical runs in the UI. Another could be workflow-to-workflow dependencies during the transition period a. Would it be OK if flows were running using both orchestration engines at the same time for some period? Or is a cold migration preferable / using downtime
✅ 1
m
My experience with Metaflow is restricted to Argo only, so probably not the best person to answer this question 🙂
a
@broad-umbrella-22770 you can use the same RDS DB and s3 buckets - without any issues across SFN & Argo. There is no orchestration specific state ever stored in RDS or S3.
You could run flows on SFN, Argo, Airflow, Kubernetes, Batch using the same RDS/S3 setup - we do the same for our OSS development
a
@broad-umbrella-22770 my team is in the process of migrating over to K8s from AWS Batch & SFN. We use both at the same time and they both point to the same data store and metadata store without any issues whatsoever.
🙌 1
Open in Slack
PreviousNext
Powered by