Hello team,
One of our users is reporting an issu...
# ask-metaflowa
acoustic-van-30942
10/16/2023, 8:53 PMHello team,
One of our users is reporting an issue with their training job when using a custom docker image. They said it was working fine a few days ago, and now they're getting a new error. I suspect it might be incompatibility with newer versions of Metaflow?
acoustic-van-30942
10/16/2023, 8:54 PMHere is the error:
Copy code
No older events at this moment.
Retry
==========
== CUDA ==
==========
CUDA Version 11.8.0
Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
<https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license>
A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
Setting up task environment.
bash: line 1: 32 Killed python -m pip install requests -qqq
bash: line 0: [: -le: unary operator expected
bash: line 0: [: -gt: unary operator expected
tar: job.tar: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
/home/ray/anaconda3/bin/python: Error while finding module specification for 'metaflow.mflog.save_logs' (ModuleNotFoundError: No module named 'metaflow')a
ancient-application-36103
10/16/2023, 10:55 PMit seems that the new image doesn't have
tara
acoustic-van-30942
10/16/2023, 10:55 PMI think it does because they're using
tara
ancient-application-36103
10/16/2023, 10:55 PMwas the custom docker image pinned?
a
acoustic-van-30942
10/16/2023, 10:56 PMI think it might be that they didn't set
HOME👀 1
a
ancient-application-36103
10/16/2023, 10:56 PMcan you help me with the entry point for the job?
a
acoustic-van-30942
10/16/2023, 10:57 PMDon't think it was pinned. It's just an image that they built and then pushed to ECR
acoustic-van-30942
10/16/2023, 10:57 PMTrying to reproduce the issue on my side. I see others have experienced the same error as this, so trying different things.
acoustic-van-30942
10/17/2023, 1:22 AMResolved - I had to add this part to their custom image:
Copy code
RUN mkdir /logs && chown 1000 /logs
RUN mkdir /metaflow && chown 1000 /metaflow
ENV HOME=/metaflow
WORKDIR /metaflow3 Views