I'm having some trouble getting custom images to work with Metaflow and AWS Batch. When I run my flow and check the logs I get a pretty generic error (truncated):

File "/root/services/ui_backend_service/data/cache/get_log_file_action.py", line 150, in execute
    total_lines = count_total_lines(local_paths)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/services/ui_backend_service/data/cache/get_log_file_action.py", line 293, in count_total_lines
    for path in paths:

TypeError: 'NoneType' object is not iterable

I was having a hard time understanding what this error meant so I took a look at the logs for the container in Batch and found the following:

File "/usr/local/lib/python3.10/site-packages/metaflow/mflog/save_logs.py", line 20, in save_logs
    pathspec = os.environ["MF_PATHSPEC"]
File "/usr/local/lib/python3.10/os.py", line 680, in __getitem__
    raise KeyError(key) from None
KeyError: 'MF_PATHSPEC'

AFAIK this isn't an environment variable I am meant to set? Anyone have any ideas what might be going wrong?

Are you able to run workflows with environments like this?

@batch(image="python:3.10", ...)

I can't use base images like that because I have custom libraries I need to include that I can't pull in with the

pypi

or

conda

decorators

Ahh I only meant if you can test that, to isolate whether the issues are emerging in the container itself or if something else fishy is going on. The other route to help diagnose would be if you can point to the container image / Dockerfile that produces it.