I am trying to scale using aws batch and currently try to use:

from metaflow import FlowSpec, conda_base, step, resources, batch, conda

@conda_base(
    libraries={
        "numpy": "1.24.3"
    },
    python="3.10.6",
)

class BigSum(FlowSpec):

    # @resources(memory=98304, cpu=16) 
    @step
    def start(self):
        import numpy
        import time
        big_matrix = numpy.random.rand(50000, 50000)
        t = time.time()
        self.sum = numpy.sum(big_matrix)
        self.took = time.time() - t
        self.next(self.end)

    @step
    def end(self):
        print("The sum is %f." % self.sum)
        print("Computing it took %dms." % (self.took * 1000))

if __name__ == '__main__':
    BigSum()

Alas whatever I do AWS appears to default to 1cpu and 4GB. I tried:

python BigSum.py --with batch:cpu=16,memory=98304,queue=our-amazing-queue --environment=conda run

or:

python BigSum.py --with batch --environment=conda run

(uncomment @resources(memory=98304, cpu=16)

Any ideas? is this a metaflow config issue?

@happy-wolf-7852 how are you verifying that whatever I do AWS appears to default to 1cpu and 4GB. Can you share a screenshot from the AWS Batch UI?

well I think it turns out, that I was not patient enough (the flow stuck for a long time in status RUNNABLE - it appears to take a long time to provision especially larger resources in aws batch - to the point that it might take longer rather than running things sequentially/locally LOL. in my use case, I want to run model fitting and scoring for N (e.g. 10) folds in parallel using aws batch.