Hello, Can a flow run another flow that uses kubernetes steps with the Runner API? When I run the parent flow on a macOS it run in the environment

d0a3e13ca27571df018ab89ed0b6dd649ffca3a8 (4f335769efbc1faf587e200c9eedb0926730dd43)

Then the sub flow (using Runner) the kubernetes steps also try and use

d0a3e13ca27571df018ab89ed0b6dd649ffca3a8 (4f335769efbc1faf587e200c9eedb0926730dd43)

The parent flow uses `pypi_base`and the subflow uses

conda_base

If I don't use

pypi_base

on the parent flow it works as expected Thanks for any help 🙂 I've attached a more detailed log.

theoretically it should. do you have a non-sensitive reproducible example?

I don't at the moment. However am I reading the logs correctly? It is trying to use the wrong environment right?

hey @creamy-stone-99746, let me take a look at that. Could you rerun with: • METAFLOW_DEBUG_CONDA=1 • and, also add that environment variable in

@environment

. Could you also let me know what the command line for that step (the one that runs on kube) is. I suspect that somehow the value is getting passed badly there. To make sure I parsed the log properly, you are starting a new flow

SlideToTIFF

in the

slide_conversion

step of the previous flow. For that new flow, the

start

step runs locally using

bac099b8ef09ba99f32812140d85680ca66ecc56 (a2271166301c8d4332566033673762770b78f643)

as well as

isyntax_formats

and

general_formats

(also using the same environment). Then, the issue is

process_isyntax

which should run on kubernetes and tries to use the original environment. Can you also let me know what the

bac…

environment is?

@dry-beach-38304 I agree with your parsing of the log. The kubernetes step appears to be trying to load the environment from the parent flow. Here is the log with the

METAFLOW_DEBUG_CONDA=1

set. I also had the sub flow output the debug. I've attached the log The

bac

environment is

{
    "pandas": "2.2.0",  
    "pydantic": "2.6.1",
    "pyjson5": "1.6.6",
    "kubernetes": "29.0.0",
    "metaflow": "2.12.8",
    "metaflow-netflixext": "1.2.1",
}

I’m going to try to repro internally with our own batch system. Could you confirm what the command line for the pod is?

Could you elaborate on "command line for the pod" I don't understand.

I mean the container is executing a command line. It downloads the package, extracts the Conda environment, etc. I just want to know what that command line is.

@dry-beach-38304 is this what you where looking for?

bash                                                                                                                           ││       -c                                                                                                                             │
│       ${METAFLOW_INIT_SCRIPT:+eval "${METAFLOW_INIT_SCRIPT}"} && true && mkdir -p $PWD/.logs && export PYTHONUNBUFFERED=x MF_PATHSPE ││ C=SlideToTIFF/1907/process_isyntax/24344 MF_DATASTORE=s3 MF_ATTEMPT=0 MFLOG_STDOUT=$PWD/.logs/mflog_stdout MFLOG_STDERR=$PWD/.logs/m │
│ flog_stderr && mflog(){ T=$(date -u -Ins|tr , .); echo "[MFLOG|0|${T:0:26}Z|task|$T]$1" >> $MFLOG_STDOUT; echo $1;  } && mflog 'Sett ││ ing up task environment.' && python -m pip install requests -qqq && python -m pip install awscli boto3 -qqq && mkdir metaflow && cd  │
│ metaflow && mkdir .metaflow && i=0; while [ $i -le 5 ]; do mflog 'Downloading code package...'; python -m awscli ${METAFLOW_S3_ENDPO ││ INT_URL:+--endpoint-url="${METAFLOW_S3_ENDPOINT_URL}"} s3 cp <s3://metaflows3qjr8flmt/metaflow/SlideToTIFF/data/2c/2cea2b077df8913c79> │
│ 0d8d9c6329e479ff055b47 job.tar >/dev/null && mflog 'Code package downloaded.' && break; sleep 10; i=$((i+1)); done && if [ $i -gt 5  ││ ]; then mflog 'Failed to download code package from <s3://metaflows3qjr8flmt/metaflow/SlideToTIFF/data/2c/2cea2b077df8913c790d8d9c632> │
│ 9e479ff055b47 after 6 tries. Exiting...' && exit 1; fi && TAR_OPTIONS='--warning=no-timestamp' tar xf job.tar && mflog 'Task is star ││ ting.' && (export CONDA_START=$(date +%s) && echo 'Bootstrapping environment ...' && python -m metaflow_extensions.netflix_ext.plugi │
│ ns.conda.remote_bootstrap SlideToTIFF process_isyntax d0a3e13ca27571df018ab89ed0b6dd649ffca3a8 4f335769efbc1faf587e200c9eedb0926730d ││ d43 s3 && export _METAFLOW_CONDA_ENV=$(cat _env_id) && export PYTHONPATH=$(pwd)/_escape_trampolines:$(printenv PYTHONPATH) && if [[  │
│ -n $(printenv LD_LIBRARY_PATH) ]]; then export MF_ORIG_LD_LIBRARY_PATH=$(printenv LD_LIBRARY_PATH); export LD_LIBRARY_PATH=$(cat _en ││ v_path)/lib:$(printenv LD_LIBRARY_PATH); fi && echo 'Environment bootstrapped.' && export CONDA_END=$(date +%s) && ./__conda_python  │
│ -u slide_to_tiff.py --quiet --metadata service --environment conda --datastore s3 --event-logger nullSidecarLogger --monitor nullSid ││ ecarMonitor --datastore-root <s3://metaflows3qjr8flmt/metaflow> --with conda_env_internal --package-suffixes .py,.R,.RDS --pylint step │
│  process_isyntax --run-id 1907 --task-id 24344 --input-paths ${METAFLOW_INPUT_PATHS_0} --split-index 0 --retry-count 0 --max-user-co ││ de-retries 0 --tag ProductionPipelineFlow/1906 --namespace user:mikebentleymills) 1>> >(python -m metaflow.mflog.tee task $MFLOG_STD │
│ OUT) 2>> >(python -m metaflow.mflog.tee task $MFLOG_STDERR >&2); c=$?; python -m metaflow.mflog.save_logs; exit $c

I see. This is very helpful. Let me dig a bit more in this. To make sure I understand: if you remove the disable=True, it uses the proper Conda environment.

Yes if I remove the

@conda(disable=True)

it does appear that it's loading the expected environment. I need it not to do that and just use the docker images environment.

ok — this is helpful. I’ll look into it in more detail.

Thanks I really appreciate the help.

ok — I think I figured it out. I’ll try it out and check if this solves it but ya, I think you nailed the corner case that I hadn’t handled for this 🙂

Let me know if I can help with anything 🙂

Ya. I was a bit busy this weekend but should have something to test soon.

No rush,

PR is here: https://github.com/Netflix/metaflow-nflx-extensions/pull/45. Could you try out this version when you get a chance and let me know. I haven’t played around with it too much but wanted to get something out. I believe it should “just work” 🙂

@dry-beach-38304 That fixed the issue thank you. 🙂

awesome. I’ll merge and release this and a bunch of other fixes (including that other one you had)