hi ! wondering if anyone here had success with running distributed training jobs with Metaflow on AWS using EFA ? We ended up using a different AMI, created a new launch template, and a new compute environment. What we observe is that when an ECS task starts, there is always second ENI gets attached without EFA enabled, then the container uses it, instead of the default one (with EFA enabled) declared in launch template ? does this sound familiar to anyone ?

Hi! A hypothesis is this has to do with ENI trunking. In this case there is one more ENI than you declare in the launch template, called

ecs-eth0

and then the others called

eth0

,

eth1

, ... Maybe the container is confused about which ENIs to use for distributed training communications? Extending this hypothesis, I'd guess the container is trying to use the

ecs-eth0

and not the others (the "defaults") that have EFA enabled. @acoustic-van-30942 has been using EFA, any thoughts?

Did you set

efa=X

in the

@batch

decorator so that the container can access the EFA network devices?

yes, we do. @calm-sundown-51671

To add on here: When running with EFA , we do get the following message

ip-10-17-109-6:133:874 [1] NCCL INFO NET/OFI Selected Provider is efa (found 1 nics)

However, we next run into the following error

ip-10-17-109-6:134:894 [2] ofi_process_cq:188 NCCL WARN NET/OFI Request 0x7f675535bf78 completed with error. RC: 5. Error: Unresponsive receiver. This error is typically caused by a peer hardware failure or incorrect inbound/outbound rules in the security group - EFA requires "All traffic" type allowlisting. Please also verify the peer application has not terminated unexpectedly. My EFA addr: fi_addr_efa://[fe80::1431:1ff:fea8:e769]:2:575006717 My host id: i-032fff1277d957bc2 Peer EFA addr: fi_addr_efa://[fe80::144a:6bff:fedf:2747]:4:628728947 Peer host id: N/A. Completed length: 0, Request: { dev: 0, size: 0, state: CREATED, direction: SEND }
ip-10-17-109-6:134:894 [2] ofi_process_cq:188 NCCL WARN NET/OFI Request 0x7f675535af78 completed with error. RC: 5. Error: Unresponsive receiver. This error is typically caused by a peer hardware failure or incorrect inbound/outbound rules in the security group - EFA requires "All traffic" type allowlisting. Please also verify the peer application has not terminated unexpectedly. My EFA addr: fi_addr_efa://[fe80::1431:1ff:fea8:e769]:2:575006717 My host id: i-032fff1277d957bc2 Peer EFA addr: fi_addr_efa://[fe80::144a:6bff:fedf:2747]:4:628728947 Peer host id: N/A. Completed length: 0, Request: { dev: 0, size: 0, state: CREATED, direction: SEND }

We also checked that the security group allows "all traffic"

Have you talked to AWS Support on this? I'm stumped! Hmm...

yeah I did. The issue is that when batch creates the ECS task, it does so with AWSVPC as the option for networking. When the task is started on EC2. ECS attaches a new non-Fabric ENI. ECS supports arguments to set the ENI with Fabric. Batch does not have the ability to set Fabric on the connector when it configures ECS. The only way around this I can think of is to switch it to Unmanaged and either configure ECS to use Bridged networking or AWSVPC with Fabric enabled. I don't think it makes a lot of difference at that point, but I also don't know how much extra work this involves in terms of connecting batch to an unmanaged instance.

I may have cycles to look into this myself and try to reproduce. I'll keep you posted if I do.