I have a question on AWS cost for metaflow. I set up AWS batch with CloudFormation. I assume that we won't have any EC2 running after batch jobs finish, but I see I have a currently running EC2 of type c4.2xlarge. What's more, when I terminate this EC2 instance it automatically starts another instance of the same type. My question: does metaflow service itself requires an EC2 instance, outside of the batch jobs? Also, what is a normal monthly AWS cost for the situation where you don't run any batch jobs at all? (I am ML hobbyist and I am using Metaflow for my personal project, and I am worried that AWS cost gets out of control. It seems like I already have 80+ dollars on my AWS bill from the past 7 days since I set up CloudFormation, even though I didn't use Batch that much)

It runs postgres db and metaflow service on ec2 (in aws rds console - db.t2.small and the ec2 instance is c4.large - so 2 CPUs and 3.75g of ram). And I think somewhere in README for aws batch terraform code it says up to 400 USD a month. I write self. in json format to S3 for dev purposes, so that I can destroy infra after a test run. That lowers costs at idle by a little but the compute is the expensive part and one of the reasons for it - aws does not allow ebs drive attachment, so I keep on shuffling files to and from S3 between steps and that takes time on both step prep and download/upload. In theory it should be similar in speed to an ebs volume, but in practice it doesn't seem like it)

The cloudformation template has a default Batch parameter to keep a minimum of 8 CPUs running on standby so there's less job startup time (e.g. Batch having to provision a fresh instance). You can change that parameter to 0 and Batch will then only spin up EC2 instances as-needed.

MinVCPUBatch:
    Type: Number
    Default: 8
    MinValue: 0
    MaxValue: 16

The default CFT is using managed AWS services for pretty much everything, so there aren't any EC2 instances you'll see outside of those Batch spins up. To clarify @cold-ability-47914, the metadata service runs on Fargate which does not expose it's underlying compute via the EC2 console (same with RDS, you won't see that DB instance listed in the EC2 console) IIRC there's also a standby Sagemaker notebook configured in the default CFT that can be disabled to save on cost. Without the hot-standby batch cpus and the sagemaker notebook, I think the metaflow stacks works out to ~$1-2/day

Ok. This explains higher costs than one would expect from a small db running non stop. I don't mind startup time for now as the steps themselves take awhile to run.

Hey @fresh-laptop-72652, thanks for the insight here. That makes sense, and I don't need an always stand-by EC2 for batch since 1 it costs a lot and 2 I don't run batch that much. So to make CloudFormation not spin up stand-by Batch EC2 I will just change the

MinVCPUBatch

parameter in the CFT console on AWS right?

yep, just update that param to be 0! I'd also turn off the sagemaker notebook

A little update here: after making

MinVCPUBatch=0

and turning off the sagemaker notebook, daily cost goes from $17 to $2.44 (without running any batch jobs)