Hi, I'm trying to run a metaflow pipeline in AWS step-functions with each step as AWS Batch job. I have configured required memory for each step in my flow however it always seems to assign 4096 MB to the step resulting in OOM errors. Anyone have any insights on this issue? Thanks!

Did you create the deployment by using

batch

either with:

python flow.py --environment=conda --with batch --with retry step-functions create

Or by adding the

@batch

decorator to all the relevant steps?

Yes I have the

@batch

decorator

What are the values you set then?

@batch

has a default of

4GB

memory and

1VCPU

.

I set the values like this:

@batch(
        cpu=64,
        memory=512_000,
        gpu=0,
    )

Are you using AWS EC2 or AWS Fargate for AWS Batch?

EC2

What is the maxVCPU on the Compute Environment?

1000000

.

hey, just chiming in here. I am the OPs colleague, and when I deploy the flow the resources are set properly. So there shouldn't be anything wrong with the compute environment setup. Seems to be some difference in the local environments, but I have confirmed that we're both running the same version of metaflow

Perhaps you could share the commands you ran to deploy the Flow @fresh-wall-61998 and the code?

Facing an issue with memory specifications not working for a pipeline with steps running on AWS batch. I am defining my flow in a class inherited from FlowSpec with the step definitions like this:

@retry(times=0)
    @timeout(minutes=1080)
    @batch(
        cpu=1,
        memory=16_000,
        gpu=0,
    )
    @environment(vars={"DISABLE_PARALLELISM": "0", "AWS_ROLE_SESSION_TIMEOUT": "3600"})
    @step
    def start(self):
        self.is_train = False
        print(self.configs)
        self.config = merge_configs(self.configs)
        run_steps("start", self)
        self.next(self.load_train_data, self.load_test_data)

However, the flow is deployed to AWS step functions where it always ends up getting the default 4gb memory for the steps even though I've set it to be 16GB as shown above

I think I saw something similar, when CPU=1 means it can not find an instance with one CPU but with 16GB of ram in the compute pool of available instances in Batch. Does this issue happen on different combinations of cpu/memory?

Yeah it does. I think it is some sort of issue with my environment since it works when a colleague deploys the same code from their local environment

@fresh-wall-61998 what is the output of

python flow.py step-functions create --only-json

for this flow for you?

Untitled