Thanks for this package. How do I enforce metaflow to only use

pip

and not mamba to resolve package dependencies? For instance, when I'm not able to add hydra-core to

@pypi_base

due to this error: "does not currently support source distributions". Also the extensions package has the following error:

The pypi_base decorator requires --environment=conda

, which seems strange

It always uses Conda/mamba/micromamba. You’ll need one of those to create the base environment. However, in both cases, if you only use the pypi decorator, it will then just use pip.

Hmm... if that's the case, I should be able to install

hydra-core

, since pip can resolve the dependency?

I can try it when I get to work. What is the content of the decorator you are trying to do. And also are you doing it on a Mac for batch/kube/etc (ie: cross platform).

Sure here is the decorator at hand. I'm operating on a Mac, running the following command:

python main.py --environment=pypi run

. Eventually the goal is to deploy to step functions so the pypi decorator needs to work. I've tried running this command with and without the extension package, and I'm running into the same issue

@pypi_base(python="3.11.9", packages=deps)
class MyFlow(FlowSpec):

deps =

{
  "absl-py": "2.1.0",
  "adbc-driver-manager": "1.1.0",
  "adbc-driver-postgresql": "1.1.0",
  "aiobotocore": "2.13.1",
  "aiohttp": "3.9.5",
  "aioitertools": "0.11.0",
  "aiosignal": "1.3.1",
  "alembic": "1.13.2",
  "annotated-types": "0.7.0",
  "antlr4-python3-runtime": "4.9.3",
  "astunparse": "1.6.3",
  "attrs": "23.2.0",
  "boto3": "1.34.131",
  "botocore": "1.34.131",
  "botocore-stubs": "1.34.145",
  "catboost": "1.2.5",
  "cattrs": "23.2.3",
  "certifi": "2024.7.4",
  "cffi": "1.16.0",
  "cfgv": "3.4.0",
  "charset-normalizer": "3.3.2",
  "clearml": "1.16.2",
  "click": "8.1.7",
  "cloudpickle": "3.0.0",
  "colorama": "0.4.6",
  "colorlog": "6.8.2",
  "contourpy": "1.2.1",
  "coreforecast": "0.0.12",
  "cryptography": "42.0.8",
  "cycler": "0.12.1",
  "decorator": "5.1.1",
  "distlib": "0.3.8",
  "dm-tree": "0.1.8",
  "et-xmlfile": "1.1.0",
  "fastexcel": "0.10.4",
  "filelock": "3.15.4",
  "flatbuffers": "24.3.25",
  "fonttools": "4.53.1",
  "frozenlist": "1.4.1",
  "fsspec": "2024.6.1",
  "furl": "2.1.3",
  "gast": "0.6.0",
  "google-pasta": "0.2.0",
  "graphviz": "0.20.3",
  "greenlet": "3.0.3",
  "grpcio": "1.65.1",
  "gurobipy": "11.0.3",
  "h5py": "3.11.0",
  "hydra-core": "1.3.2",
  "hydra-zen": "0.13.0",
  "identify": "2.6.0",
  "idna": "3.7",
  "importlib-resources": "6.4.0",
  "jinja2": "3.1.4",
  "jmespath": "1.0.1",
  "joblib": "1.4.2",
  "jsonschema": "4.23.0",
  "jsonschema-specifications": "2023.12.1",
  "keras": "3.4.1",
  "kiwisolver": "1.4.5",
  "libclang": "18.1.1",
  "llvmlite": "0.43.0",
  "lsprotocol": "2023.0.1",
  "mako": "1.3.5",
  "markdown": "3.6",
  "markdown-it-py": "3.0.0",
  "markupsafe": "2.1.5",
  "matplotlib": "3.9.1",
  "mdurl": "0.1.2",
  "metaflow": "2.12.8",
  "metaflow-stubs": "2.12.8",
  "ml-dtypes": "0.4.0",
  "mlforecast": "0.12.1",
  "multidict": "6.0.5",
  "multimethod": "1.10",
  "mypy-boto3-cloudformation": "1.34.111",
  "mypy-boto3-dynamodb": "1.34.131",
  "mypy-boto3-ec2": "1.34.145",
  "mypy-boto3-lambda": "1.34.77",
  "mypy-boto3-rds": "1.34.145",
  "mypy-boto3-s3": "1.34.138",
  "mypy-boto3-sqs": "1.34.121",
  "mypy-extensions": "1.0.0",
  "namex": "0.0.8",
  "ndg-httpsclient": "0.5.1",
  "networkx": "3.3",
  "nodeenv": "1.9.1",
  "numba": "0.60.0",
  "numpy": "1.26.4",
  "omegaconf": "2.3.0",
  "openpyxl": "3.1.5",
  "opt-einsum": "3.3.0",
  "optree": "0.12.1",
  "optuna": "3.6.1",
  "orderedmultidict": "1.0.1",
  "packaging": "24.1",
  "pandas": "2.2.0",
  "pandas-stubs": "2.2.2.240603",
  "pathlib2": "2.3.7.",
  "patsy": "0.5.6",
  "pillow": "10.4.0",
  "platformdirs": "4.2.2",
  "plotly": "5.22.0",
  "polars": "1.2.1",
  "polars-lts-cpu": "1.1.0",
  "pre-commit": "3.7.1",
  "protobuf": "4.25.3",
  "psutil": "6.0.0",
  "psycopg2-binary": "2.9.9",
  "pyaml": "24.7.0",
  "pyarrow": "17.0.0",
  "pyasn1": "0.6.0",
  "pycparser": "2.22",
  "pydantic": "2.8.2",
  "pydantic-core": "2.20.1",
  "pygls": "1.3.1",
  "pygments": "2.18.0",
  "pyjwt": "2.8.0",
  "pyopenssl": "24.1.0",
  "pyparsing": "3.1.2",
  "python-dateutil": "2.9.0.",
  "python-dotenv": "1.0.1",
  "pytz": "2024.1",
  "pyyaml": "6.0.1",
  "referencing": "0.35.1",
  "requests": "2.32.3",
  "rich": "13.7.1",
  "rpds-py": "0.19.0",
  "s3fs": "2024.6.1",
  "s3transfer": "0.10.2",
  "scikit-learn": "1.5.1",
  "scipy": "1.14.0",
  "setuptools": "69.5.1",
  "sf-hamilton-lsp": "0.1.0",
  "shellingham": "1.5.4",
  "six": "1.16.0",
  "sqlalchemy": "2.0.31",
  "sqlmodel": "0.0.18",
  "statsmodels": "0.14.2",
  "tenacity": "8.5.0",
  "tensorboard": "2.17.0",
  "tensorboard-data-server": "0.7.2",
  "tensorflow": "2.17.0",
  "tensorflow-io-gcs-filesystem": "0.37.1",
  "termcolor": "2.4.0",
  "tf-keras": "2.17.0",
  "tfcausalimpact": "0.0.16",
  "threadpoolctl": "3.5.0",
  "tqdm": "4.66.4",
  "typeguard": "4.3.0",
  "typer": "0.12.3",
  "types-awscrt": "0.21.2",
  "types-pytz": "2024.1.0.20240417",
  "types-s3transfer": "0.10.1",
  "typing-extensions": "4.12.2",
  "typing-inspect": "0.9.0",
  "tzdata": "2024.1",
  "urllib3": "2.2.2",
  "utilsforecast": "0.1.12",
  "virtualenv": "20.26.3",
  "werkzeug": "3.0.3",
  "wheel": "0.43.0",
  "window-ops": "0.0.15",
  "wrapt": "1.16.0",
  "xlsx2csv": "0.8.2",
  "xlsxwriter": "3.2.0",
  "yarl": "1.9.4"
}

Error message:

Pip ran into an error while setting up environment:
    Could not find a binary distribution for antlr4-python3-runtime==4.9.3 
    for the platform osx-arm64

sorry — busy meeting day for some reason. OK, so I tired: • resolving on mac for mac • resolving on mac for linux • resolving on linux for linux • resolving on linux for mac The only one that doesnt’ work is the last one but then I realized it’s because we build wheels internally at Netflix (at least I think). It should work for sure in a non-cross-platform setting but in a cross platform one, there may be a bit of extra stuff to do. If you are using the extension, could you rerun with METAFLOW_DEBUG_CONDA=1 and send me the output and I can see why this error is happening.

I've resolved it now. Thanks. Just resorted to using a docker image instead