https://outerbounds.com/ logo
Join Slack
Powered by
Hi team, I deployed Metaflow on GCP following <th...
# ask-metaflow
f
Hi team, I deployed Metaflow on GCP following this doc. And trying to run a simple pipeline using a custom Docker image I pushed to GCR made by following this doc. But my job failed at the very first 
start
step on GCP when I triggered it like
Copy code
PYTHONPATH=.python MY_FILE_NAME.py run --with kubernetes
locally it runs perfectly if I do
Copy code
PYTHONPATH=.python MY_FILE_NAME.py run
Here is the logs by 
kubectl logs MY-JOB-ID
Copy code
Setting up task environment.
Downloading code package...
bash: line 1: download-gcp-object: command not found
Downloading code package...
bash: line 1: download-gcp-object: command not found
Downloading code package...
bash: line 1: download-gcp-object: command not found
Downloading code package...
bash: line 1: download-gcp-object: command not found
Downloading code package...
bash: line 1: download-gcp-object: command not found
Downloading code package...
bash: line 1: download-gcp-object: command not found
Failed to download code package from <gs://MY_STORAGE_NAME/MY_PIPELINE_NAME/data/dc/dc584121a2e5317f62a240978df8bb9dc74e7bac> after 6 tries. Exiting...
What can I do? Thanks in advance
❤️ 1
✅ 1
s
@few-salesmen-35936 can you share a bit about your custom docker image? It seems that 
simple-gcp-object-downloader
wasn't downloaded at runtime for your job
f
Sure @square-wire-39606
Copy code
FROM --platform=linux/amd64 python:3.9

RUN mkdir /logs && chown 1000 /logs
RUN mkdir /metaflow && chown 1000 /metaflow
COPY . /metaflow
ENV HOME=/metaflow
WORKDIR /metaflow
USER 1000

RUN pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir -r requirements.txt -r requirements-dev.txt
also sharing my 
config.json
in case you see something weird here
Copy code
{
  "METAFLOW_DATASTORE_SYSROOT_GS": "<gs://STORAGE_NAME>",
  "METAFLOW_DEFAULT_DATASTORE": "gs",
  "METAFLOW_DEFAULT_METADATA": "service",
  "METAFLOW_KUBERNETES_NAMESPACE": "argo",
  "METAFLOW_KUBERNETES_SERVICE_ACCOUNT": "argo",
  "METAFLOW_SERVICE_INTERNAL_URL": "<http://metadata-service.default:8080/>",
  "METAFLOW_SERVICE_URL": "<http://127.0.0.1:8080/>",
  "METAFLOW_DEFAULT_CONTAINER_REGISTRY": "REGISTRY_NAME",
  "METAFLOW_DEFAULT_CONTAINER_IMAGE": "IMAGE_NAME"
}
@square-wire-39606 
simple-gcp-object-downloader
is installed in the image at runtime as I printed 
pip list
Copy code
simple-gcp-object-downloader 0.1.0
So 
download-gcp-object
executive can't be found. I built this image on M1 Mac with 
FROM _--platform_=linux/amd64 python:3.9
line in Dockerfile. Any ideas how to make it found by Metaflow?
OK, I found a solution. Simply adding a folder for 
pip
installed packaged worked. So your snippet on this custom docker snippet doc needs an update as follows
Copy code
FROM python:3.10-slim

RUN mkdir /logs && chown 1000 /logs
RUN mkdir /metaflow && chown 1000 /metaflow
ENV HOME=/metaflow

### NEW LINE
ENV PATH "$PATH:$HOME/.local/bin"

WORKDIR /metaflow
USER 1000

COPY requirements.txt requirements.txt
RUN pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir -r requirements.txt
Thanks for your awesome work😁
🙌 2
❤️ 1
10 Views
Open in Slack
PreviousNext
Powered by