Hello - may I ask a question please? When executing a Metaflow step, we're seeing

No space left on device

error being raised by

s3op.py

(please find stack trace below). The execution is taking place on AWS Batch, and we've allocated

500GB

of diskspace. The data we're working on is estimated about

2GB

, and checking the code, this data is not assigned to

<http://self.my|self.my>_variable

. Has anyone encountered this error before? If so, would you be able to share any tips to troubleshoot this error please? Thank you for your help!

2023-10-31 16:39:49.565 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd] S3 non-transient error (attempt #7): Traceback (most recent call last):
2023-10-31 16:39:49.565 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/plugins/datatools/s3/s3op.py", line 1220, in <module>
2023-10-31 16:39:49.565 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]     cli(auto_envvar_prefix="S3OP")
2023-10-31 16:39:49.565 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/_vendor/click/core.py", line 829, in __call__
2023-10-31 16:39:49.565 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]     return self.main(*args, **kwargs)
2023-10-31 16:39:49.565 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/_vendor/click/core.py", line 782, in main
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]     rv = self.invoke(ctx)
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/_vendor/click/core.py", line 1259, in invoke
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]     return _process_result(sub_ctx.command.invoke(sub_ctx))
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/_vendor/click/core.py", line 1066, in invoke
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]     return ctx.invoke(self.callback, **ctx.params)
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/_vendor/click/core.py", line 610, in invoke
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]     return callback(*args, **kwargs)
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/plugins/datatools/s3/s3op.py", line 684, in wrapper
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]     return func(*args, **kwargs)
2023-10-31 16:39:49.566 [1698763585031127/create_dataset_region_1/7 (pid 109147)] [b228f859-890b-4c6b-8fcc-2f9d0c4cfcfd]   File "/app/metaflow/metaflow/plugins/datatools/s3/s3op.py", line 712, in wrapper
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]     sz_results = start_workers(mode, urls, num_workers, inject_failure, s3config)
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]   File "/app/metaflow/metaflow/plugins/datatools/s3/s3op.py", line 342, in start_workers
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]     queue = Queue(len(urls) + num_workers)
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]   File "/build/python/lib/python3.9/multiprocessing/context.py", line 103, in Queue
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]     return Queue(maxsize, ctx=self.get_context())
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]   File "/build/python/lib/python3.9/multiprocessing/queues.py", line 43, in __init__
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]     self._rlock = ctx.Lock()
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]   File "/build/python/lib/python3.9/multiprocessing/context.py", line 68, in Lock
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]     return Lock(ctx=self.get_context())
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]   File "/build/python/lib/python3.9/multiprocessing/synchronize.py", line 162, in __init__
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]     SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]   File "/build/python/lib/python3.9/multiprocessing/synchronize.py", line 57, in __init__
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf]     sl = self._semlock = _multiprocessing.SemLock(
2023-10-31 16:39:01.884 [1698763585031127/create_dataset_region_1/5 (pid 109134)] [f3041734-7bcd-46b7-af14-d0eb470266bf] OSError: [Errno 28] No space left on device

Could you share more of the code? Could also be a noisy neighbour problem(although with 500GB would be highly unlikely).

It’s just one pipeline running so shouldn’t be. Also we increased it to 6Tb but getting the same error

Right either the code is doing something unexpected or the updates to the AWS Batch Compute Environment aren’t being applied.

Yep we are, it does show the compute environment being correct in the launch template as well

Are you using terraform to manage your infra?

For the context: we are using pandarallel to parallelize processing of our datasets

Did you make a change to the file with the batch compute environment? So that it will register a difference.