https://outerbounds.com/ logo
Join Slack
Powered by
was looking over this <https://admin-docs.metaflow...
# dev-metaflow
m
was looking over this https://admin-docs.metaflow.org/metaflow-on-aws/deployment-guide/aws-cloudformation-deployment and wondered, have you guys looked into using AWS CDK to define/deploy the metaflow stack to cloudformation?
⬆️ 1
✅ 1
b
also curious, as we use cdk internally
a
we had the same question a while back, ended up we translating the Cloudformation to CDK ourselves 🙂
a
tbh we haven't looked into CDK super deeply yet, but i know @flaky-plumber-70709 has been working on a CDK package for Metaflow https://github.com/bcgalvin/cdk-metaflow
👀 1
i believe the reason why "raw" Cloudformation was used in the first place was to have a human-readable/human-maintainable cloudformation config
though i'm not sure human readable cloudformation is physically possible at all 😄
m
@average-beach-28850 we've really enjoyed using CDK at my company this year. defining infra in a "regular" language like typescript/python makes life much easier
@adventurous-gigabyte-81428 i would love to pick your brain sometime if you have pointers you can share from your org's experiences with this :)
a
@mysterious-yacht-40013 👌
f
so far it was just a race to get all the pieces together but will expose the same params as the cfn - working on a project now that will use it so will dust it off and work on it some more
couldn't get the jsii to work w the go bindings yet but maybe that's been worked out in the last few months since I was fighting with it
I've been working with cdk the last few years and it's pretty amazing in terms of productivity vs cfn or terraform even. CDK pipelines especially is a treat to work with
👀 1
a
@mysterious-yacht-40013 @blue-daybreak-26318 @adventurous-gigabyte-81428 The rationale for using cloudformation over cdk at the time of release was purely driven by availability of skillsets on the team. @eager-jelly-89580 has contributed a Terraform template since and we would love to make sure people who want alternatives in cdk/pulumi etc. also have options 🙂
👍 2
m
@square-wire-39606 @flaky-plumber-70709 does it make sense to collaborate on getting https://github.com/bcgalvin/cdk-metaflow added to documentation?
s
Of course! @flaky-plumber-70709 - are the cdk modules ready for prime time?
f
i think it may need some work on my end to get it to a more generally usable state (right now is just hard coded for my use case) and stabilize the api before it gets the official blessing
👍 1
a
Let me know whenever you feel it's ready and we can add a pointer in our documentation to https://github.com/bcgalvin/cdk-metaflow
👍 1
f
can work on it over the weekend / early next week to expose the same functionality as the cloudformation template
also has some stuff in there like a cloud watch dashboard for s3/ecs that is kind of half implemented atm that I may strip out into its own construct repo
but would say it can be ready sometime next week
I can add a PR to the metaflow-tools repo too when it’s ready on how to use it from Python/typescript/go too if you think it belongs there @square-wire-39606
m
@flaky-plumber-70709 that's exciting to hear, but also I didn't mean to put you to work on your weekend! let me know if I can be of help either in development or test driving / feedback. I'll be taking a look around the repo soon as I'm evaluating metaflow for use at my company right now and we've been standardizing around cdk 🙂
👍 1
s
@flaky-plumber-70709 I was thinking of adding a pointer to your repository (+ a howto guide) in the official docs.
f
sure sounds good 👌🏾
also playing around with cdk8s quite a bit lately so hoping to add metaflow-on-kubernetes soon too (but that may need to be a separate repo initially)
excited 1
👀 1
but happy to have eyeballs on this, nice to have some positive pressure to work on it again. Thanks for the shout out @average-beach-28850 !
🙌 2
One of things that I’ve used cdk+metaflow for in the past that I think has a lot of potential was leveraging cdk pipelines to manage deployment of flows - so you can have codepipeline handle ECR custom images, add manual approval steps, integration tests and updating the step function
💯 2
a
Documenting this pattern would be amazing! Plenty of members in the community have asked for best practices in this area.
f
definitely - will add different configurations/scenarios to the demo directory
🙇 1
@mysterious-yacht-40013 I’ll reach out when I’ve done some more of the legwork here. Would love your feedback on the api
👍 1
Open in Slack
PreviousNext
Powered by