Hi guys getting this error `fatal error An error occurred 40 Outerbounds #ask-metaflow

Hi guys, getting this error `fatal error: An error...

fancy-mouse-14245

03/06/2023, 5:56 AM

Hi guys, getting this error

fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

while using

@batch

decorator. And while running the workflow am using

--with batch

and defining iam roles and resources that needed. My question is if we use

@batch

does it override

--with batch

arguments? And if so.. should one use

@resources

than

@batch

✅ 1

victorious-lawyer-58417

03/06/2023, 5:58 AM

do you see jobs submitted to Batch? If the error occurs in tasks running in Batch, it indicates that your Batch jobs don't have the right role attached to them

victorious-lawyer-58417

03/06/2023, 5:59 AM

here's why to use

@resources

instead of `@batch`: • it allows you to run code locally,

@batch

will send the step always to Batch (which may be desirable in some cases but usually local testable is a good idea) • it allows you to change the backend in the feature. With

@resources

, you can switch between

run --with kubernetes

and

run --with batch

without changing anything in the code

fancy-mouse-14245

03/06/2023, 6:03 AM

One more thing, if we are defining parameters like this

Copy code

python3 sampleflow.py run --with batch:cpu=1,memory=4000,image=ubuntu:latest,iam_role=arn:aws:iam::something:something

if will run all the code using batch with the specified iam role but if I am using

@batch

for a step.

@batch

will get the priority rather than cli arguments right?

victorious-lawyer-58417

03/06/2023, 6:05 AM

yep

fancy-mouse-14245

03/06/2023, 6:05 AM

cool thanks.

👍 1

Open in Slack

Previous Next