Hello - may I ask a question please? When executing a Metaflow step, we're seeing

No space left on device

error being raised by

s3op.py

(please find stack trace below). The execution is taking place on AWS Batch, and we've allocated

500GB

of diskspace. The data we're working on is estimated about

2GB

, and checking the code, this data is not assigned to

<http://self.my|self.my>_variable

. Has anyone encountered this error before? If so, would you be able to share any tips to troubleshoot this error please? Thank you for your help!

2023-10-31 16:39:49.565 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd] S3 non-transient error (attempt #7): Traceback (most recent call last):
2023-10-31 16:39:49.565 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/plugins/datatools/s3/s3op.py", line 1220, in <module>
2023-10-31 16:39:49.565 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]     cli(auto_envvar_prefix="S3OP")
2023-10-31 16:39:49.565 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/_vendor/click/core.py", line 829, in __call__
2023-10-31 16:39:49.565 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]     return self.main(*args, **kwargs)
2023-10-31 16:39:49.565 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/_vendor/click/core.py", line 782, in main
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]     rv = self.invoke(ctx)
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/_vendor/click/core.py", line 1259, in invoke
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]     return _process_result(sub_ctx.command.invoke(sub_ctx))
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/_vendor/click/core.py", line 1066, in invoke
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]     return ctx.invoke(self.callback, **ctx.params)
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/_vendor/click/core.py", line 610, in invoke
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]     return callback(*args, **kwargs)
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/plugins/datatools/s3/s3op.py", line 684, in wrapper
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]     return func(*args, **kwargs)
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/plugins/datatools/s3/s3op.py", line 712, in wrapper
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]     sz_results = start_workers(mode, urls, num_workers, inject_failure, s3config)
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]   File "/app/metaflow/metaflow/plugins/datatools/s3/s3op.py", line 342, in start_workers
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]     queue = Queue(len(urls) + num_workers)
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]   File "/build/python/lib/python3.9/multiprocessing/context.py", line 103, in Queue
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]     return Queue(maxsize, ctx=self.get_context())
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]   File "/build/python/lib/python3.9/multiprocessing/queues.py", line 43, in __init__
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]     self._rlock = ctx.Lock()
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]   File "/build/python/lib/python3.9/multiprocessing/context.py", line 68, in Lock
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]     return Lock(ctx=self.get_context())
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]   File "/build/python/lib/python3.9/multiprocessing/synchronize.py", line 162, in __init__
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]     SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]   File "/build/python/lib/python3.9/multiprocessing/synchronize.py", line 57, in __init__
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]     sl = self._semlock = _multiprocessing.SemLock(
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf] OSError: [Errno 28] No space left on device