https://outerbounds.com/ logo
Join Slack
Powered by
Hi are there any examples of Dockerfiles that use ...
# ask-metaflow
b
Hi are there any examples of Dockerfiles that use poetry that get used in AWS Batch?
✅ 1
I can get poetry to install everything, but I don't know how to activate the poetry environment for when steps actually get run on the container
I see some stuff in other threads that look like this:
Copy code
ENV VIRTUAL_ENV=/opt/venv
RUN python -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
but a little out of my element here
I'm able to get to the point where I can build the container:
Copy code
docker build -t parsely-transformers-pytorch-gpu .
And then run it in interactive mode and import stuff
Copy code
wooyoung.moon docker % docker run -it parsely-transformers-pytorch-gpu
root@737ee207aeac:/app# python
Python 3.9.5 (default, Nov 23 2021, 15:27:38) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from metaflow import FlowSpec
>>>
But when I try to run things in batch I see this error now:
Copy code
2023-08-24 14:52:51.459 [36/train_model/146 (pid 66166)] [e1067133-989c-437f-9ed8-05f29bafde26] Task is starting (status STARTING)...
2023-08-24 14:53:18.350 [36/train_model/146 (pid 66166)] [e1067133-989c-437f-9ed8-05f29bafde26] Task is starting (status RUNNING)...
2023-08-24 14:53:17.530 [36/train_model/146 (pid 66166)] [e1067133-989c-437f-9ed8-05f29bafde26] Setting up task environment.
2023-08-24 14:53:31.701 [36/train_model/146 (pid 66166)] [e1067133-989c-437f-9ed8-05f29bafde26] Downloading code package...
2023-08-24 14:53:32.390 [36/train_model/146 (pid 66166)] [e1067133-989c-437f-9ed8-05f29bafde26] Code package downloaded.
2023-08-24 14:53:32.422 [36/train_model/146 (pid 66166)] [e1067133-989c-437f-9ed8-05f29bafde26] Task is starting.
2023-08-24 14:53:33.185 [36/train_model/146 (pid 66166)] [e1067133-989c-437f-9ed8-05f29bafde26] Traceback (most recent call last):
2023-08-24 14:53:35.691 [36/train_model/146 (pid 66166)] AWS Batch error:
2023-08-24 14:53:37.188 [36/train_model/146 (pid 66166)] Essential container in task exited This could be a transient error. Use @retry to retry.
2023-08-24 14:53:37.188 [36/train_model/146 (pid 66166)] 
2023-08-24 14:53:33.185 [36/train_model/146 (pid 66166)] [e1067133-989c-437f-9ed8-05f29bafde26]   File "/app/metaflow/train_section_model_flow_v2.py", line 15, in <module>
2023-08-24 14:53:33.185 [36/train_model/146 (pid 66166)] [e1067133-989c-437f-9ed8-05f29bafde26]     from metaflow import FlowSpec, step, batch
2023-08-24 14:53:33.185 [36/train_model/146 (pid 66166)] [e1067133-989c-437f-9ed8-05f29bafde26] ImportError: cannot import name 'FlowSpec' from 'metaflow' (/app/metaflow/__init__.py)
2023-08-24 14:53:37.423 [36/train_model/146 (pid 66166)] Task failed.
2023-08-24 14:53:37.634 Workflow failed.
2023-08-24 14:53:37.634 Terminating 0 active tasks...
2023-08-24 14:53:37.634 Flushing logs...
    Step failure:
    Step train_model (task-id 146) failed.
And here's the Dockerfile we're using:
Copy code
FROM nvidia/cuda:12.2.0-base-ubuntu20.04

ENV DEBIAN_FRONTEND noninteractive

# Install basic dependencies
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    build-essential \
    python3.9 \
    python3.9-dev \
    python3.9-distutils \
    python3.9-venv \
    curl \
    ca-certificates && \
    update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 1 && \
    update-alternatives --set python3 /usr/bin/python3.9 && \
    rm -rf /var/lib/apt/lists/*

# Install pip for Python 3.9
RUN curl <https://bootstrap.pypa.io/get-pip.py> -o get-pip.py && \
    python3.9 get-pip.py && \
    rm get-pip.py

# Set up a working directory
WORKDIR /app

RUN ln -s /usr/bin/python3 /usr/bin/python & \
    ln -s /usr/bin/pip3 /usr/bin/pip

ENV VIRTUAL_ENV=/opt/venv
RUN python -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

COPY pyproject.toml poetry.lock ./
RUN pip install poetry
RUN poetry install --without=dev
nevermind got it 🙂
For posterity, the Dockerfile copied above does actually work. There was just some other issue in our code that we had to fix
✅ 1
a
@bright-author-73055 curious are you using a custom docker image for your flows and/or how are you handling dependencies? I had my whole project set up with poetry but then tore it all out when I went to the docs and saw that they said metaflow was opinionated about using conda environments.
b
Still early days over here so there's no set process we've been using yet for any extended period of time. But yes, for now just using a custom docker image (made from the Dockerfile above) with poetry. I think we have a slight bias against conda on our team, so we're still trying to assess how much of a headache this will be. Were you running into any issues in particular before you switched to conda?
a
I actually have a strong preference for poetry. I'll check in on this thread after a while and ping you to see how it is working out for you. I'd love to say goodbye to conda.
👍 1
@bright-author-73055 how's the customer docker file + poetry working for you?
b
It's been working out just fine so far! But to be fair, I've only run a couple example flows for my team using poetry so far, so we haven't really stress tested it yet. What are the types of issues you are worried about with moving over to poetry? So I can keep an eye out for those things and report back
👀 1
h
I'm on Emily's team and can speak to this. Our only reason for migrating to conda was the ability to run flows seamlessly on both cloud and local, bringing dev envs much closer to full parity with deployed flows. The docs make 
@conda
seem like a hard requirement for this feature, and at least in theory it's an extremely smooth way to handle an otherwise painful process. Unfortunately we have had a lot of issues with both speed and package availability, even after taking the time to build out a micromamba based dev container with the (excellent) 
metaflow-netflixext
. My main question is whether poetry is compatible with deploying flow-specific envs to cloud resources, with the same ease found in the 
@conda
decorator.
b
Oh I see yeah our setup isn't that slick yet. Since we're currently mostly just using metaflow for a single project right now there aren't that many Dockerfiles and containers for us to manage. So I basically just created a Dockerfile that uses our 
pyproject.toml
to run on Batch. Actually took a decent amount of time because of a bunch of little issues that might be relatively specific to our container requirements (e.g. 
cuda
and 
libreoffice
) but it seems to be working reasonably well now. We also aren't doing anything slick with different steps requiring different sets of packages or anything. Not sure how different or specific your different flows are in terms of its requirements, but if it's as simple just differing 
pyproject.toml
and 
poetr.lock
files then I bet it's not a huge deal to use 
poetry
for this. Does that kinda help? Please let me know if I totally missed the mark!
👀 1
h
Hey, I'm curious how this ended up working out for you? We're about to embark on our metaflow journey but some of our projects are already using poetry. I'm wondering how much of a headache it will be to use poetry to manage dependencies for a metaflow project.
Open in Slack
PreviousNext
Powered by