Hello, I’m running metaflow through AWS batch, do you know how I can get the vcpu count from within my step if I restrict it through the resource decorator? e.g. if I use os.cpu_count() within my step I get the machine cpu count instead of what was restricted by the resource decorator. also when I try to read

/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us

I get -1.

great question! It’s actually quite tricky since Batch allows bursting to all available cores on the instance

so does this mean there isn’t a reliable way to get the cpu_count at the container level if I use batch?

you can get the minimum cpu_count reliably, maximum is available indirectly, but the actual number is variable

a second questions would be, if I define the cpu count through the resource decorator, can I look up that value within the step and pass it to my function?

yes, let me give you an example..

how would the minimum cpu_count relate to what i’ve potentially defined in the resource decorator? would it be the same?

minimum equals to what's specified in

@resources

oh icic. i guess from my first question, how can I get the minimum cpu count as well

try this:

from metaflow import FlowSpec, current, resources, Parameter, step, Task

class CPUCountFlow(FlowSpec):

    @resources(cpu=8)
    @step
    def start(self):
        print('cpu count', self.cpu_count())
        self.next(self.end)

    @step
    def end(self):
        print("done!")

    def cpu_count(self):
        decos = self._graph_info['steps'][current.step_name]['decorators']
        for deco in decos:
            if deco['name'] == 'resources':
                return deco['attributes']['cpu'] 

if __name__ == '__main__':
    CPUCountFlow()

that makes sense. going back to the first question, this is more of a batch or ecs question then but do you know of if I can look something up within the container to get the constricted cpu count? Something like looking at

/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us

yep, that's the issue: Python's

os.cpu_count()

and

multiprocessing.cpu_count()

return all available cores on Batch, which is often too high

i’d be ok with setting the cpu_count to be the minimum

another solution is construct your compute environment so that

@resources

match instance sizes, so min==max, which effectively disables bursting

but how can i create a

cpu_count()

function to grab the minimum amount in batch?

this should do the job https://outerboundsco.slack.com/archives/C02116BBNTU/p1677257426570009?thread_ts=1677256450.120499&cid=C02116BBNTU

oh, i meant is there a solution to get the cpu count without using information from the flow. like in docker if I constrict the resources with

--cpu

I can look at

/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us

to figure out how much cpu I have. this would probably be more of a batch/ecs question but do you know of a way that I can determine the constrained cpu count from within the container if I’m using batch. when I look at

/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us

from batch, I get

-1

right, I don't know if there's a way to do it on Batch. Let me know if you find one 🙂

haha gotcha. thanks for the help!

sorry that we don't have a better answer today. Definitely follow up here if you find something