Hi folks. I have a use case for deploying flows fr...
# ask-metaflow
f
Hi folks. I have a use case for deploying flows from deployed flows. I thought this would be an application of using the
Deployer
API within a flow, but it seems to be a bit more complicated. Locally this approach works fine, but when I try to use it either in a flow running on kubernetes or deployed as an argo-workflow, it fails during the deployment process with the error "No mamba binary found". I'm using
Metaflow 2.19.12+netflix-ext(1.3.1)
. Is this pattern supposed to work or is the Deployer API only meant to be used from outside of a workflow orchestrator?
a
Does it work if you don't use the Netflix-ext? You might have to uninstall the extension to try it out.
d
I’ll answer in a bit. Short answer is it works but you may need some addl configuration. But yes this is fully supported.
f
That's great! Would be interested to understand what's missing in my setup to make it work.
d
Ah yes, sorry, I totally forgot 😞.
OK, so basically the extension needs some conda distribution to work. What we do internally is we maintain a conda distribution and download that.
let me look at the code one sec
so if you can run with a docker image that has a conda/mamba environment, that would be the best.
Else, what we do internally is have a distribution of conda/mamba that is uploaded to S3 and it is installed when needed.
Let me find the internal script we use to build those distributions
Copy code
#!/usr/bin/env bash

# Exit immediately if any command fails.
set -e
set -o pipefail


if [ $# -ne 2 ]; then
    echo "Usage: $0 <CONDA_VERSION> linux-64|osx-64|osx-arm64"
    exit 1
fi

# --- CONFIGURATIONS ---
# Make sure to update these values as appropriate
CONDA_VERSION="$1"

if [ "$2" == "linux-64" ]; then
    BUILD_DIRS="/tmp /data /mnt"
    INSTALLER="Miniforge3-Linux-x86_64.sh"
elif [ "$2" == "osx-64" ]; then
    BUILD_DIRS="/tmp "
    INSTALLER="Miniforge3-MacOSX-x86_64.sh"
elif [ "$2" == "osx-arm64" ]; then
    BUILD_DIRS="/tmp "
    INSTALLER="Miniforge3-MacOSX-arm64.sh"
else
    echo "Second argument must be one of: osx-arm64, osx-64, linux-64"
    exit 1
fi


S3_UPLOAD_PATH="<s3://XXXXXX/metaflow/condav2/conda-local/>"
S3_REMOTE_UPLOAD_PATH="<s3://XXXXXX/metaflow/condav2/conda-remote/>"


# Download the installer if it doesn't exist
if [ ! -f "$INSTALLER" ]; then
    wget "<https://github.com/conda-forge/miniforge/releases/latest/download/${INSTALLER}>"
fi

for dir in $BUILD_DIRS; do
    BUILD_DIR="$dir"

    BUILD_SUFFIX=$(basename "${BUILD_DIR}") # Extracts 'tmp' from '/tmp'
    CONDA_LOCAL_PATH="${BUILD_DIR}/metaflow-condav2-${CONDA_VERSION}"
    CONDA_LOCAL_DIST="conda-${2}-${BUILD_SUFFIX}-${CONDA_VERSION}.tgz" # Changed extension to .tgz

    echo "Starting Conda Distribution Build for version ${CONDA_VERSION}..."
    echo "Build directory set to: ${BUILD_DIR}"
    echo "Final artifact will be named: ${CONDA_LOCAL_DIST}"

    # Create the target directory and ensure we have permissions
    # mkdir -p "$(dirname "$CONDA_LOCAL_PATH")"
    # chown -R "$(whoami)" "$(dirname "$CONDA_LOCAL_PATH")"

    # Run the installer and clean up
    bash "$INSTALLER" -b -p "$CONDA_LOCAL_PATH"

    # Create the .condarc configuration file
    cat <<EOF > "${CONDA_LOCAL_PATH}/.condarc"
channels:
- conda-forge
pip_interop_enabled: true
remote_max_retries: 10
channel_priority: strict
repodata_use_zst: true
EOF

    # Copy the configuration for Mamba
    cp "${CONDA_LOCAL_PATH}/.condarc" "${CONDA_LOCAL_PATH}/.mambarc"

    # Install and update core packages
    MAMBA_EXE="${CONDA_LOCAL_PATH}/bin/mamba"

    "$MAMBA_EXE" -r "$CONDA_LOCAL_PATH" update -n base mamba conda pip -y
    "$MAMBA_EXE" -r "$CONDA_LOCAL_PATH" install -n base \
    "conda-lock==3.0.4" \
    "conda-package-handling" \
    "micromamba>=2.3.2" \
    "boto3" \
    "uv" \
    -y

    PYTHON_VERSION=$($CONDA_LOCAL_PATH/bin/python -c "import sys; print(f'python{sys.version_info.major}.{sys.version_info.minor}')")

    # Apply conda-lock patch (remove when no longer needed) and update conda-lock
    # version to be unpinned.    
    SITE_PACKAGES_DIR="${CONDA_LOCAL_PATH}/lib/${PYTHON_VERSION}/site-packages"
    PATCH_FILE="$(cd "$(dirname "${BASH_SOURCE[0]}")/../operations" && pwd)/conda-lock_v3.0.4.diff"
    ORIG_DIR="$(pwd)"
    cd "${SITE_PACKAGES_DIR}" || { echo "Failed to cd to ${SITE_PACKAGES_DIR}"; exit 1; }
    patch -p1 < "${PATCH_FILE}"
    cd "${ORIG_DIR}"
    
    # Isolate the environment by modifying Conda's search path
    CONSTANTS_PY_PATH="${CONDA_LOCAL_PATH}/lib/${PYTHON_VERSION}/site-packages/conda/base/constants.py"

    sed -i.bak "/DEFAULT_CHANNEL_ALIAS/i\\
    SEARCH_PATH = ('\$CONDA_ROOT/.condarc',)
    " "$CONSTANTS_PY_PATH"

    # Clean up caches to reduce size
    "$MAMBA_EXE" -r "$CONDA_LOCAL_PATH" clean --all -y

    # Create the final tarball
    cd "$CONDA_LOCAL_PATH"

    if [[ "$(uname)" == "Darwin" ]]; then
        tar --disable-copyfile -czf "${BUILD_DIR}/${CONDA_LOCAL_DIST}" .
    else
        tar -czf "${BUILD_DIR}/${CONDA_LOCAL_DIST}" .
    fi

    cd -

    echo "✅ Build complete!"
    echo "Distribution created at: ${BUILD_DIR}/${CONDA_LOCAL_DIST}"

    # Upload the final artifact to S3
    echo "Uploading ${CONDA_LOCAL_DIST} to ${S3_UPLOAD_PATH}..."
    aws s3 cp "${BUILD_DIR}/${CONDA_LOCAL_DIST}" "${S3_UPLOAD_PATH}${CONDA_LOCAL_DIST}"

    # Upload the micromamba binary as the remote installer
    echo "Uploading micromamba binary to ${S3_REMOTE_UPLOAD_PATH}..."
    aws s3 cp "${BUILD_DIR}/metaflow-condav2-${CONDA_VERSION}/bin/micromamba" "${S3_REMOTE_UPLOAD_PATH}conda-${2}-${CONDA_VERSION}"

    echo "🚀 Upload complete!"
done

rm "$INSTALLER"
And then, you can configure it using stuff like this:
Copy code
CONDA_REMOTE_INSTALLER_DIRNAME = from_conf(
    "CONDA_REMOTE_INSTALLER_DIRNAME", "conda-remote"
)

# Override when using a consistent naming scheme for installed conda distribution.
# Note: set with METAFLOW__CONDA_VERSION_ID (double __)
_CONDA_VERSION_ID = from_conf("_CONDA_VERSION_ID", "20251022")

# Ditto for local installation of conda. If not specified,
# a conda executable needs to be installed and will be used (if locatable in PATH)
CONDA_LOCAL_DIST_DIRNAME = from_conf("CONDA_LOCAL_DIST_DIRNAME", "conda-local")

# Binary within CONDA_REMOTE_INSTALLER_DIRNAME to use as the binary on remote instances.
# Use {arch} to specify the architecture. This should be fully functional binary
# To regenerate this binary, download micromamba for the proper platform and upload
# it here
CONDA_REMOTE_INSTALLER = from_conf(
    "CONDA_REMOTE_INSTALLER", f"conda-{{arch}}-{_CONDA_VERSION_ID}"
)

CONDA_LOCAL_DIST = from_conf(
    "CONDA_LOCAL_DIST", f"conda-{{arch}}-{_conda_dir_name()}-{_CONDA_VERSION_ID}.tgz"
)

CONDA_LOCAL_PATH = from_conf(
    "CONDA_LOCAL_PATH",
    os.path.join("/", _conda_dir_name(), f"metaflow-condav2-{_CONDA_VERSION_ID}"),
)
so basically, your choices are: • have an image that already has a conda environment that works for you (and you can point the extension to it) • pre-build these environments and store them in S3 and allow Metaflow to fetch and use those. • We could also do the same as the mainline pypi decorator and download one if none exists. I can look into that one. It’s a bit more complicated because we do a bit more than just resolving but we should be able to do a minimal setup of it.
f
Thank you very much. I will look into this 🙂
d
let me know if you have questions. wrote this very quickly
f
Thank you @dry-beach-38304. Since I'm on Mac it looks like I will have to do this cross-platform and I'm running into a bit of trouble there. However if I understand it correctly, the only thing I need to actually make this work from the deployed flow is the micromamba binary in the correct directory according to CONDA_REMOTE_INSTALLER, right? Can I just directly download micromamba binary and put it there instead of going through the whole install procedure or is there something that breaks in that case?
Ah I see now. Since I'm running Deployer inside a pod, the pod is basically the local client for the Deployer and it wouldn't be enough to just have micromamba available at CONDA_REMOTE_INSTALLER. It would still look in the location indicated by CONDA_LOCAL_DIST and CONDA_LOCAL_PATH and would use what it finds there, right?
d
correct — when you are running the Deployer on a remote node, it acts like the “local” conda version. So on that remote node you can actually have both the remote distribution (just micromamba) and the local one (to deploy new flows). Btw, this does get around the cross compile issue — you are now no longer doing a cross arch deploy since you are deploying from the same type of the node as the one you would deploy to.
f
Great, I could make it work now. Thank you very much! When set up correctly, this would mean that also people on their local workstations would use those specific conda versions in the conda-local folder, right? I'm assuming metaflow checks somewhere whether a particular version was already downloaded before and proceeds to use that one instead of always downloading those tarballs again and again, right?
d
Yep to both.