https://outerbounds-community.slack.com/archives/C02116BBNTU/p1745352122226819?thread_ts=1731291743.653309&cid=C02116BBNTU Question regarding local packages:

package_a/
  src/
    package_a
  tests/
  pyproject.toml
flows/
  flow_a.py # references import package_a
  pyproject.toml # lists package_a as dependency via { path = "../package_a", editable=true }

Is using symlinks still the only way to include local packages that are not inside of the flow folder? I would like to avoid that since that would include all the tests + test_data

There is now another way. If your package is installed, if it has METAFLOW_PACKAGE_POLICY=« include » at the top level, it will get included. Note that I may have found a bug in some cases for editable packages which I am tracking down. It doesn’t directly affect this but may be a similar problem.

ah cool! so where exactly do I set

METAFLOW_PACKAGE_POLICY="include"

? And by installed that means it is installed in my local environment right?

Yes for installed. In the init file for the package. Basically it needs to be a member of the module.

Running into issues:

├── flows
│   ├── flow.py
│   ├── pyproject.toml
└── geoimage
    ├── pyproject.toml
    └── src
        ├── geoimage
        │   ├── __init__.py
        │   └── process.py

init.py of geoimage

from .process import hello
METAFLOW_PACKAGE_POLICY = "include"
__all__ = ["hello"]

pyproject.toml in flows

[project]
name = "flows"
version = "0.1.0"
dependencies = [
    "metaflow",
    "geoimage"
]

[tool.uv.sources]
geoimage = { path="../geoimage", editable=true }

flow.py

from metaflow import FlowSpec, step
# from geoimage.process import hello as geoimage_hello # error


class MyFlow(FlowSpec):
    @step
    def start(self):
        from geoimage.process import hello as geoimage_hello # works local but does not exist on batch

Run via

uv run flow.py run

If I import geoimage on the top level of the file I get (if it has

METAFLOW_PACKAGE_POLICY

):

Metaflow 2.18.6 executing MyFlow for user:kokottc
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
    Internal error
Traceback (most recent call last):
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/cli.py", line 658, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/cli_components/utils.py", line 69, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/tracing/__init__.py", line 27, in wrapper_func
    return func(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/cli_components/run_cmds.py", line 156, in wrapper
    return func(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/cli_components/run_cmds.py", line 355, in run
    before_run(obj, tags, decospecs)
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/cli_components/run_cmds.py", line 67, in before_run
    obj.package = MetaflowPackage(
                  ^^^^^^^^^^^^^^^^
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/package/__init__.py", line 96, in __init__
    self._mfcontent = MetaflowCodeContentV1(criteria=_module_selector)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/packaging_sys/v1.py", line 100, in __init__
    self._files_from_modules.update(self._module_files(k, v.root_paths))
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/packaging_sys/v1.py", line 423, in _module_files
    raise RuntimeError(
RuntimeError: Distribution 'geoimage' is not contributing to module 'geoimage' as expected (got '/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/geoimage' when expected one of {'/Users/kokottc/formexp_v2/packages/geoimage/src/geoimage'})

If I do the import only within the start step it works running locally. But if I launch it on batch I get:

ModuleNotFoundError: No module named 'geoimage'

when it gets to the import in the step.

ok, so there seems to be a bug/unhandled case in including the package. You are supposed to include it at the top level (it won’t work otherwise as you noted). It’s probably related to how we find the directories. It may be related to this issue with editable packages. I am working on something to fix editable packages in a different part of the code so let me add this to it as well.

image.png

Y’a. Something with the editable package install.

Yes in non-editable mode it works fine!

cool — at least it’s not totally borked 🙂

Unfortunately the same error:

File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/package/__init__.py", line 96, in __init__
    self._mfcontent = MetaflowCodeContentV1(criteria=_module_selector)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/packaging_sys/v1.py", line 100, in __init__
    self._files_from_modules.update(self._module_files(k, v.root_paths))
  File "/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/metaflow/packaging_sys/v1.py", line 461, in _module_files
    raise RuntimeError(
RuntimeError: Distribution 'geoimage' is not contributing to module 'geoimage' as expected (got '/Users/kokottc/formexp_v2/packages/flows/.venv/lib/python3.11/site-packages/geoimage' when expected one of {'/Users/kokottc/formexp_v2/packages/geoimage/src/geoimage'})

Installed metaflow via:

uv add git+<https://github.com/Netflix/metaflow/> --branch fix/support_editable_pkgs_more

dependencies = [
    "metaflow",
    "geoimage",
]

[tool.uv.sources]
geoimage = { path="../geoimage", editable = true }
metaflow = { git = "<https://github.com/Netflix/metaflow/>", branch = "fix/support_editable_pkgs_more" }

ok — sorry for the delay. I will make a few more modifications to it for you to try again. I missed a case it seems

Okay finally got to try it! Looks like it was merged already and it seems to be working now. Thanks! 🙂

cool — glad it’s fixed 🙂