Is there any interest in creating an official AWS CDK stack / set of constructs for Metaflow? I think that would be super cool because it would make slight variations in architectures easy! For example: • you could optionally create a new RDS database or refer to an existing one • you could automate the creation of a Route53 subdomain like

<http://metaflow.my-company.com|metaflow.my-company.com>

and a ACM cert with

DNSValidatedCert

• Hooking up your own authentication solution over the metaflow UI would be easier; if you don't want to provide your own, you could use the default Cognito or API Gateway basic auth • We could create diagrams for the components and Stack--I think this would be helpful for new people wrapping their heads around how Metaflow is deployed. Note, I did see this project https://pypi.org/project/cdk-metaflow/. It could be worth exploring. Some concerns we'd probably want to address are: • It seems to be firstly written in TypeScript. I'm not sure if you how much you can edit it from Python. I'll ask the CDK community about this. • It's written using AWS CDK v1--v2 is out now. It could potentially be updated.

the author @flaky-plumber-70709 is on this slack :)

Oooh, @flaky-plumber-70709 I may end up needing to write some CDK code to set up metaflow for my team soon. Maybe I could turn it into a contribution to your project. It'd be awesome to build on something that already exists. Idk if this is a problem, but we use the Python CDK

haven't touched this in a while but some of those ideas (import rds or aurora postgres + handling acm) were items I wanted to implement. I actually started at a new job this week and plan to dust this off and add those in and escape hatches to access the underlying infrastructure a bit better

Nice! Yeah, if you want to collaborate on it, I may be able to get some time 😄

While I haven't looked closely at how the official Metaflow terraform config is implemented, it's worth noting perhaps that most of the goals mentioned in the original post should be possible with a proper submodule structure in terraform. E.g., consider the database: • a common pattern is to have the entrypoint terraform module take some optional config (db uri, etc) and configure things for existing db; when not provided, an RDS module is used to create a db for you according to defaults Done properly, you can have very customizable terraform modules that work out of the box but also let users mix and match components freely, while still staying firmly in the realm of a declarative infra DSL. CDK is also a cool option and could expose even more power/flexibility, of course, so not knocking it! Just wanted to let you know that you might not need to invest a lot of effort to achieve your usecase already w/ terraform.

great point - submodules would also make cdktf easier to work with if you wanted to use one of the supported languages over hcl too. In my case, I work w aws exclusively and wanted to use a few cdk-specific things like pipelines and projen (though think the support for cdktf is there now)

After a cursory look the official metaflow terraform module, looks like you'd just need to update the entrypoint module's vars to support optional args for an existing db config, and then update the entrypoint module's main config to make instantiating the

metaflow-datastore

submodule conditional on the vars you just defined not being set. You can see the other submodules (

metaflow-metadata-service

, etc) already interface w/ the datastore submodule by way of a few exported attributes, so it'd be easy to update this to instead use the values from the vars you added if they are defined. The only real problem I see in current official terraform is that it co-mingles RDS and S3 in the same datastore submodule. So, you'd probably want to either separate those two components, or else update the datastore submodule so that it also accepts some var to know whether it should create the RDS components or skip them.

^ great points, you could also create that S3 bucket directly without using

metaflow-datastore

, since Metaflow doesn't really require setting it up in a special way, its fairly easy. (that's what I do in deployments where I don't want to use RDS). Unlike some other things, S3 bucket is just another resource or two

I like that. What do you lose if you don't even include RDS? @narrow-lion-2703

I mean, you still need a Postgres database, its just in some scenarios I had an RDS instance created by some other means, or not even RDS. For example, for our internal testing we have some deployments where Postgres runs on K8S (i wouldn't recommend that for production but still)

Another example: maybe you want to write your own terraform config to provision a (recently GA'd) Aurora Serverless v2 postgres cluster instead of whatever the default RDS instance is, perhaps, and just pass in the results of that to the metaflow module. Or perhaps you've got some on-prem/standalone db you're managing on EC2.

The main takeaway is that you only need a couple of tweaks to official terraform modules to allow this kind of customization. We have our own internal terraform modules for most of these things that work similarly. It might be worth making a PR to the official terraform for some of these common customization points, and maybe including an example somewhere.

On the original subject: we could def have official CDK stack in the future too. Main things is that there is a combinatorial # of ways to deploy Metaflow infra, between {cloudformation,terraform,pulumi,CDK} plus all kinds of flavors of auth and compute environment configurations. Right now we are using Terraform internally (i have probably a dozen of Metaflow infra deployments using Terraform running right now) so it is easier for us to commit to support it as an "official" way, but def open to other options too as they mature

I see what you mean, @narrow-lion-2703. Also thanks, @User. For the time being I may just reference the terraform modules and write a purpose-built CDK package for my personal project and my team at work. By chance, do you know of a

docker-compose.yml

file that has metaflow and the UI and doesn't require you to build the images from source as in the

metaflow-service

repo? I'm struggling to understand the relationship of the UI service to the metaflow metadata service and also to the postgres instance. I feel like that's the main thing holding me back from being able to take off writing this myself

i don't think there's another

docker-compose.yml

. As for the relationship between those parts, take a look at the diagram in https://github.com/outerbounds/terraform-aws-metaflow

Okay! That makes so much more sense! I thought

metaflow-ui-backend

was just a duplicate of

metaflow-metadata

with different environment variables set. Thanks for that

btw if you look at the service repo it can be a bit confusing since there are all kinds of development-only docker files, that e.g. allow you to build separate images for UI backend and service. But the way I described above (single image for Metadata + UI backend, second for UI static files, running as 3 separate services) is the most paved road