Hi all. I have a custom docker image that cannot run flows as it fails with:

Setting up task environment.
/usr/bin/bash: line 1: [: -le: unary operator expected
/usr/bin/bash: line 1: [: -gt: unary operator expected
tar (child): job.tar: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now

My dockerfile is:

FROM pytorch/pytorch:2.10.0-cuda12.6-cudnn9-runtime

# Install required packages
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        python3-pip git tar bash libgl1 libglib2.0-0 && \
    rm -rf /var/lib/apt/lists/*

# Copy requirements
COPY requirements.txt ./

# Install Python packages
RUN pip install --no-cache-dir -r requirements.txt --break-system-packages

RUN mkdir /logs && chown 1000 /logs
RUN mkdir /metaflow && chown 1000 /metaflow
ENV HOME=/metaflow
WORKDIR /metaflow
USER 1000

ENV PYTHONPATH=/usr/local/lib/python3.12/site-packages

Any ideas or help ? Thanks in advance.

Hi Pablo, I think that if

Entrypoint

is non-empty and uses eval, that might be the issue.

From the error, it’s failing to extract the tarball which indicates a permission issue. Also from the bash errors looks like some env vars are missing.

I have added a default entrypoint and it still fails to run with the same error. @hundreds-rainbow-67050 what permission and env do you thing are missing?

probably the datastore ones (are you using s3?). is your metaflowconfig setup properly?

If I downgrade the base to pytorch/pytorch:2.8.0-cuda12.9-cudnn9-runtime it just works. Any ideas?

Which is the one that doesn't work?

pytorch/pytorch:2.10.0-cuda12.6-cudnn9-runtime

that didnt work

Hi incase the original authors or someone else needs this fixed. The breaking error is that

pip install boto3 requests

fails silently at runtime due to PEP 668 (externally-managed environment), which Ubuntu 24.04 enforces. Ubuntu 22.04 did not. I suspect your images fall into this difference. If you want you can build a custom docker image make sure to use the following.

Dockerfile 
 ....

# This is the key difference that makes it work because it allows metaflow bash 
# command to pip install awscli and requests
ENV PIP_BREAK_SYSTEM_PACKAGES=1 

....

Here is a complete dockerfile that fixed it for me.

FROM nvidia/cuda:13.0.3-cudnn-devel-ubuntu24.04
 
 
RUN apt update && apt install python3.12 python3-pip fd-find curl locales -y && \
    locale-gen en_US.UTF-8 && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.12 1 && \
    ln -sf /usr/bin/python3.12 /usr/bin/python && \
    ln -s /usr/bin/fdfind /usr/bin/fd
 
# Important to set this 
ENV PIP_BREAK_SYSTEM_PACKAGES=1
 
CMD ["/usr/bin/python3"]

Happy flowing