I am trying to scale using aws batch and currently try to use:

from metaflow import FlowSpec, conda_base, step, resources, batch, conda

@conda_base(
    libraries={
        "numpy": "1.24.3"
    },
    python="3.10.6",
)

class BigSum(FlowSpec):

    # @resources(memory=98304, cpu=16) 
    @step
    def start(self):
        import numpy
        import time
        big_matrix = numpy.random.rand(50000, 50000)
        t = time.time()
        self.sum = numpy.sum(big_matrix)
        self.took = time.time() - t
        self.next(self.end)

    @step
    def end(self):
        print("The sum is %f." % self.sum)
        print("Computing it took %dms." % (self.took * 1000))

if __name__ == '__main__':
    BigSum()

Alas whatever I do AWS appears to default to 1cpu and 4GB. I tried:

python BigSum.py --with batch:cpu=16,memory=98304,queue=our-amazing-queue --environment=conda run

or:

python BigSum.py --with batch --environment=conda run

(uncomment @resources(memory=98304, cpu=16)

Any ideas? is this a metaflow config issue?