https://outerbounds.com/ logo
Join Slack
Powered by
I'm trying out the `@conda` decorator and having s...
# ask-metaflow
c
I'm trying out the 
@conda
decorator and having some issues. I have created a very simple setup that installs numpy and a specific python version, and am encountering the error:
Copy code
Step: start, Error: PackagesNotFoundError: The following packages are not available from current channels:
    
      - numpy==1.23.5
      - python==3.9.15
    
    Current channels:
    
      - <https://conda.anaconda.org/conda-forge/osx-arm64>
      - <https://conda.anaconda.org/conda-forge/noarch>
I have confirmed that creating a conda env outside of Metaflow succeeds in finding/installing those packages. And I am also able to reproduce the error if I set the environment variable (as Metaflow does) 
CONDA_USE_ONLY_TAR_BZ2=True
. The only seemingly relevant result when Googling that environment variable is an issue posted by you@square-wire-39606 😅. Any idea how I should get the Metaflow conda env finding the correct package?
(I am using conda version 22.11.1)
I have another issue, which may or not be related. Using 
--environment=conda
and 
-with kubernetes
I get the following error:
Copy code
Bootstrapping conda environment...(this could take a few minutes)
2022-12-14 14:24:34.474 Workflow starting (run-id 891):
2022-12-14 14:24:39.292 [891/start/4456 (pid 35679)] Task is starting.
2022-12-14 14:24:42.817 [891/start/4456 (pid 35679)] [pod t-dnmcc-2tlck] Task is starting (Pod is running, Container is running)...
2022-12-14 14:24:42.418 [891/start/4456 (pid 35679)] [pod t-dnmcc-2tlck] Setting up task environment.
2022-12-14 14:25:04.530 [891/start/4456 (pid 35679)] [pod t-dnmcc-2tlck] Downloading code package...
2022-12-14 14:25:05.567 [891/start/4456 (pid 35679)] [pod t-dnmcc-2tlck] Code package downloaded.
2022-12-14 14:25:05.592 [891/start/4456 (pid 35679)] [pod t-dnmcc-2tlck] Task is starting.
2022-12-14 14:25:06.779 [891/start/4456 (pid 35679)] [pod t-dnmcc-2tlck] Bootstrapping environment...
2022-12-14 14:25:10.365 [891/start/4456 (pid 35679)] [pod t-dnmcc-2tlck] Environment bootstrapped.
2022-12-14 14:25:10.365 [891/start/4456 (pid 35679)] [pod t-dnmcc-2tlck] bash: line 1: metaflow_DependencyTestFlow_linux-64_7b6d74c7d55dd5d2b9a4874a8334691e4d156140/bin/python: No such file or directory
2022-12-14 14:25:14.628 [891/start/4456 (pid 35679)] Kubernetes error:
2022-12-14 14:25:14.716 [891/start/4456 (pid 35679)] Error (exit code 127). This could be a transient error. Use @retry to retry.
2022-12-14 14:25:14.716 [891/start/4456 (pid 35679)] 
2022-12-14 14:25:15.289 [891/start/4456 (pid 35679)] Task failed.
This is with a step that does nothing but call 
print
. The steps are decorated with 
@conda()
(i.e. just metaflow deps) to get around the issue in the original post.
a
Hi! We should have a fix out for this quite soon. We are testing a few possibilities.
c
Great, thanks 🙂. To clarify: a fix for the first or second issue?
a
The first issue. Sorry, just parsing the error message for the second one.
👍 1
c
Thanks. Is there a github issue or something I can reference for the former?
a
just created one - I should have a PR out very likely today.
🙏 1
regarding the second issue - what is the memory that you are assigning to the pod?
c
Default: i guess 4096MB
Just ran it again with 
resources(memory=4096),
same issue
a
and what are the instances that you are using for your kubernetes cluster?
c
m5a.4xlarge
, so 64GiB total
To be clear, we have no issues running many other flows. This is an issue I'm seeing specifically with the Conda env usage
I think I have narrowed down the issue. If I specify 
python:latest
image, the issue doesn't occur, so I think it must be some interaction with the image we are using and the conda env setup.
s
I was think if you were using ARM based instances
I am rather curious - what does the docker image that is causing this issue look like?
c
It's based on 
nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04
, and we add python & python deps in there.
Are the requirements for getting the conda env working listed anywhere?
s
The only items needed are python, pip and tar. We install everything else (awscli, boto3, requests, mamba) dynamically if they are not already available.
🤔 1
I will try with this docker image and see if the error is reproducible on our end.
c
Can confirm our image has python, tar, and pip. The first two are in the 
PATH
, but pip is not (though still invokable via 
python -m pip install ...
).
a
let me look into this after addressing the 
.conda
issue
👍 1
q
Hi! Any news on .conda issue? I'm having it too
a
which version of metaflow are you on? the 
.conda
files are now supported
3 Views
Open in Slack
PreviousNext
Powered by