https://outerbounds.com/ logo
Join Slack
Powered by
Hello everyone, I did some testing for the `@pyto...
# ask-metaflow
a
Hello everyone, I did some testing for the 
@pytorch_parallel
decorator, but it didn't seem to work for me when combining it with 
@batch
. It did work locally, but when using 
@batch
, the ec2 instances immediately terminate and the multi-node bach job gets stuck in RUNNABLE status
✅ 1
I can confirm that my compute environment is also using private subnets. Although I don't believe the subnets have 
auto-assign public IP addresses
enabled. Not sure if that is mandatory.
a
can you try with the 
auto-assign public IP addresses
enabled?
a
Is that needed, Savin? Or only applicable to Fargate? I'm using Ec2. That's not quite trivial here because sadly don't have permissions to modify a subnet.
a
can you check that you are satisfying all the requirements listed under 
Compute Environment Considerations
here - https://docs.aws.amazon.com/batch/latest/userguide/multi-node-parallel-jobs.html
👌 1
a
From a networking aspect, looks good. Using a private subnet and NAT Gateway is being used. Using a managed compute environment too, and only 1 security group. The compute environment becomes INVALID though after running the job -
Copy code
Instance Launch failed. Client.InternalError: Client error on launch
Not sure what this means:
If you want to submit multi-node parallel jobs to a compute environment, create a cluster placement group in a single Availability Zone and associate it with your compute resources. This keeps your multi-node parallel jobs on a logical grouping of instances close with high network flow potential
Is that handled by Metaflow?
a
No - that's part of your compute environment configuration
a
OK multi-node is working now. Just a couple missing permissions from the batch execution role relating to 
kms
. Excited to see if I can get an LLM fine-tuning running on PyTorch Lightning working!
❤️ 1
4 Views
Open in Slack
PreviousNext
Powered by