I am getting this error when I am trying to error ...
# ask-metaflows
some-noon-7401
04/07/2023, 8:17 PMI am getting this error when I am trying to error in multi-gpu training on metaflow, using
python -m torch.distributed.launch --nproc_per_node=3 --node_rank=0 --master_port=8888 Copy code
2023-04-07 20:13:21.739 [206/train/707 (pid 32119)] [e50c4a8b-8e8a-44fc-bf67-5eb1dcc912fa] File "trainMF.py", line 1205, in <module> 2023-04-07 20:13:21.740 [206/train/707 (pid 32119)] [e50c4a8b-8e8a-44fc-bf67-5eb1dcc912fa] main() 2023-04-07 20:13:21.740 [206/train/707 (pid 32119)] [e50c4a8b-8e8a-44fc-bf67-5eb1dcc912fa] File "trainMF.py", line 607, in main 2023-04-07 20:13:21.740 [206/train/707 (pid 32119)] [e50c4a8b-8e8a-44fc-bf67-5eb1dcc912fa] transform=transform 2023-04-07 20:13:21.740 [206/train/707 (pid 32119)] [e50c4a8b-8e8a-44fc-bf67-5eb1dcc912fa] File "/root/metaflow/cottonDataExp.py", line 52, in __init__ 2023-04-07 20:13:21.740 [206/train/707 (pid 32119)] [e50c4a8b-8e8a-44fc-bf67-5eb1dcc912fa] self.img_blobs = [img.blob for img in raw_imgs] 2023-04-07 20:13:21.740 [206/train/707 (pid 32119)] [e50c4a8b-8e8a-44fc-bf67-5eb1dcc912fa] File "/root/metaflow/cottonDataExp.py", line 52, in <listcomp> 2023-04-07 20:13:21.740 [206/train/707 (pid 32119)] [e50c4a8b-8e8a-44fc-bf67-5eb1dcc912fa] self.img_blobs = [img.blob for img in raw_imgs] 2023-04-07 20:13:21.740 [206/train/707 (pid 32119)] [e50c4a8b-8e8a-44fc-bf67-5eb1dcc912fa] File "/root/metaflow/metaflow/plugins/datatools/s3/s3.py", line 274, in blob
2023-04-07 20:13:21.741 [206/train/707 (pid 32119)] [e50c4a8b-8e8a-44fc-bf67-5eb1dcc912fa] return f.read()
2023-04-07 20:13:21.741 [206/train/707 (pid 32119)] [e50c4a8b-8e8a-44fc-bf67-5eb1dcc912fa] OSError: [Errno 14] Bad addressv
victorious-lawyer-58417
04/08/2023, 12:31 AMpeculiar. Is there anything special about these instances, like a special filesystem?
victorious-lawyer-58417
04/08/2023, 12:42 AMif not, it might be an out of memory issue. You can try increasing
@resources(memory=)s
some-noon-7401
04/08/2023, 6:32 AMHi @victorious-lawyer-58417,
am still getting the same error.
Ps: I am running this on AWS batch
some-noon-7401
04/08/2023, 8:29 AMCan you point me to any documentation for an optimised dataloader in a multi-GPU run?
Looks like that for each process in each GPU the data is being downloaded again.
v
victorious-lawyer-58417
04/08/2023, 8:48 AMdid you increase memory for the step?
s
some-noon-7401
04/08/2023, 10:46 AMYes, it's throwing the same error.
2 Views