When using --environment=conda, if I specify a conda_base for the flow but also specify a specific image for a step using @batch(image=xyz), I get the following error

bash: line 1: metaflow_FlowName_linux-64_sha/bin/python: No such file or directory

The docker image I'm using is a custom python:3.10-slim image where I have installed some python packages with pip into a venv with then added that venv to the path. So, conda isn't installed. I'm assuming that's the issue. Am I right, or is there something else going on?

The image needs wget. If you don’t use the slim image, and just the vanilla python image - that should do the trick

Understood, I'll install wget into the image. However, can you explain what will happen. It sound like it's just downloading the conda env into the image and then using that to run. Will it still work, if I want it to use a venv instead of conda?

in terms of steps that happen when the job starts 1. the code package gets downloaded from s3 2. individual conda packages are downloaded in parallel from s3 3. a conda executable is downloaded from the internet if conda isn't available in the image 4. conda blesses the downloaded set of packages as a conda environment 5. metaflow switches the python interpreter to the one within the conda environment 6. user code executes

Understood, so there's no way to have that one step just use the venv or conda env that's present within the dockerfile?

you can skip the

@conda

decorator and ensure that the python interpreter for your dockerfile points to that conda environment

I'm assuming that means I need to not set @conda_base at the beginning of the file, but per step

correct and set

@conda(disabled=True)

for this step

Thanks!