https://outerbounds.com/ logo
Join Slack
Powered by
Hey folks, I’m having trouble getting a multi-stag...
# ask-metaflow
m
Hey folks, I’m having trouble getting a multi-stage-built container running with 
@batch
. (The motivation in this case for the multi-stage build is mainly to save space/time.) I’ve tried a build like this:
Copy code
FROM python:3.11 as build-stage

RUN python -m venv /opt/venv

ENV PATH="/opt/venv/bin:$PATH"

#... install python dependencies to venv ...

FROM python:3.11-slim as deploy-stage

COPY --from=build-stage /opt/venv /opt/venv

ENV PATH="/opt/venv/bin:$PATH"
And a version of the above which adds the ENV/WORKDIR/USER lines mentioned in the docs (ref). Either version yields an error like:
Copy code
Data store error:
No completed attempts of the task was found for task '<my-task-path>'
The same flow runs successfully to completion if I use a one-stage build, ie one that just starts from 
python:3.11
. Any help appreciated.
✅ 1
a
is there any CMD or ENTRYPOINT that you specify in your build file?
m
Thank you. No, there is neither a CMD nor ENTRYPOINT line.
a
and what are the dependencies that are getting installed? I wonder if the ENV PATH or packages being installed are running into an issue
m
(Have been doing a minimal test case so far.) ATM the deps are exactly requests and llama-cpp-python (ref).
a
can you try once without any dependencies to rule them out as a possible cause?
oh - i think i know what's happening
the 
-slim
image wouldn't work because we need 
wget
m
Okay, that gives me a couple experiments to try, thank you kindly. I thought I’d tried running the flow with 
@batch(image="python:3.11-slim")
without making the calls to requests or llama-cpp, just an os.listdir() iirc, and it worked, but I might be misremembering. Would this experiment test the 
wget
hypothesis?
Okay, ran two experiments: 1. [`image="python:3.11-slim"`/`@batch()` step runs 
print(os.listdir())
as proof of life] This runs successfully e2e. 2. [`image="mmacpherson/llm-meta:latest"`/`@batch()` step runs 
print(os.listdir())
as proof of life] Where I removed the 
pip install
lines, so it is only creating the venv in the build stage, copying it over in the deploy stage, and setting the ENV/WORKDIR/USER per the docs above. (I confirm that I can run the python in that image at 
/opt/venv
as expected.) With no installed packages, this setup fails in the 
@batch()
step with that “Data store error:” noted above. Here is the complete Dockerfile used to produce the image in expt 2:
Copy code
FROM python:3.11 as compile-image

RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# RUN pip install --no-cache-dir --upgrade pip && \
#     pip install --no-cache-dir --upgrade \
#         llama-cpp-python \
#         requests

FROM python:3.11-slim AS run-image

COPY --from=compile-image /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# Per Metaflow's docs.
RUN mkdir /logs && chown 1000 /logs
RUN mkdir /metaflow && chown 1000 /metaflow
ENV HOME=/metaflow
WORKDIR /metaflow
USER 1000
Ran another experiment with a multi-stage build, with this Dockerfile, that differs from the above in that it doesn’t use the venv, but copies over the system python library directory. This yields a small image, about the size of the venv-based one above. This small image works, where the venv-based one does not. So the issue might be something to do with the path? Although the “Data store error:” seems unconnected to this on the surface…
Copy code
FROM python:3.11 as compile-image

RUN pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir --upgrade \
        llama-cpp-python \
        requests

FROM python:3.11-slim AS run-image

COPY --from=compile-image /usr/local/lib/python3.11 /usr/local/lib/python3.11

# Per Metaflow's docs.
RUN mkdir /logs && chown 1000 /logs
RUN mkdir /metaflow && chown 1000 /metaflow
ENV HOME=/metaflow
WORKDIR /metaflow
USER 1000
9 Views
Open in Slack
PreviousNext
Powered by