Hi, again with a aws batch question.

class SampleFlow(FlowSpec):
    @step
    def start(self):
        pass
        self.next(self.a)

    
    @resources(cpu=16, memory=60000)
    @step
    def a(self):
        pass
        self.next(self.train_and_infer)
    
    @resources(gpu=1, cpu=32, memory=124000)
    @step
    def train_and_infer(self):
        pass
        self.next(self.c)
    
    @resources(cpu=32, memory=62000)
    @step
    def c(self):
        pass
        self.next(self.end)

    @step
    def end(self):
        pass

When we are running the flow. the whole flow is running in a gpu instance. Even tho we have specified to use gpu in only one step. We are using aws batch with step functions and we have deployed metaflow using terraform with stop instances.

@fancy-mouse-14245 do you have a single job queue with both gpu and non-gpu instances?

yeah single job queue with both gpu and non-gpu instances

one mechanism to work around this issue will be to have separate job queues - one with gpu instances and the other with non-gpu instances

for that I need to have two different compute environment, right?

since technically a workload that doesn't require non-gpu resources can be executed on a gpu node, you will get the current behavior if a single job queue has both gpu and non-gpu nodes

got it.