hello folks, I'm trying to create a custom

StepDecorator

and followed the guides from

metaflow-extensions-template

, also looked at examples in

metaflow-nflx-extensions

, but something is still missing as the registration fails with the following error message:

metaflow_extensions: Found packages present at {'/home/daroczig/.local/share/virtualenvs/sc-resource-tracker-WkxmM6V1/lib/python3.13/site-packages/metaflow_extensions'}
metaflow_extensions: Found extension package 'resource-tracker' at '/home/daroczig/.local/share/virtualenvs/sc-resource-tracker-WkxmM6V1/lib/python3.13/site-packages/metaflow_extensions'...
metaflow_extensions:     Top-level 'resource_tracker' extends 'plugins' with config 'None'
metaflow_extensions:     Top-level 'resource_tracker' found config file 'metaflow_extensions.resource_tracker.plugins.mfextinit_resource_tracker'
Traceback (most recent call last):
  File "/home/daroczig/projects/sc-resource-tracker/examples/metaflow/1-single-step.py", line 4, in <module>
    from metaflow import Flow, FlowSpec, step
  File "/home/daroczig/.local/share/virtualenvs/sc-resource-tracker-WkxmM6V1/lib/python3.13/site-packages/metaflow/__init__.py", line 48, in <module>
    from metaflow.extension_support import (
    ...<7 lines>...
    )
  File "/home/daroczig/.local/share/virtualenvs/sc-resource-tracker-WkxmM6V1/lib/python3.13/site-packages/metaflow/extension_support/__init__.py", line 890, in <module>
    _all_packages, _pkgs_per_extension_point = _get_extension_packages()
                                               ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/daroczig/.local/share/virtualenvs/sc-resource-tracker-WkxmM6V1/lib/python3.13/site-packages/metaflow/extension_support/__init__.py", line 666, in _get_extension_packages
    raise RuntimeError(
    ...<2 lines>...
    )
RuntimeError: Unresolved dependencies in 'metaflow_extensions': 'resource-tracker'

could someone please take a look at suggest what am I doing wrong? https://github.com/SpareCores/resource-tracker/tree/main/src/metaflow_extensions/resource_tracker/plugins

Nothing comes to my mind immediately - can you run your flow with the flag

METAFLOW_DEBUG_EXT=1

and then show the outputs here?

thanks @rhythmic-controller-77489 🙇 the above was already run with

METAFLOW_DEBUG_EXT=1

Okay, taking a look

thanks a ton! there's an example flow in the

examples/metaflow

folder if you may have time to reproduce the problem

So, I removed the symlinks and moved things around a bit to have the following dir structure:

mfextinit_track_resources.py
track_resources
├── __init__.py
├── resource_tracker
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-310.pyc
│   │   └── tracker.cpython-310.pyc
│   └── tracker.py
└── track_resources_decorator.py

mfextinit_resource_tracker.py

STEP_DECORATORS_DESC = [
    ("track_resources", ".track_resources.track_resources_decorator.ResourceTrackerDecorator"),
]

And finally moved this import to inside the function, and it seems to have fixed the issue for me.

thank you very much for taking a look! I highly appreciate it. I moved things around as you suggested, and commited at https://github.com/SpareCores/resource-tracker/commit/beaa54651ce2505a3a547d00e5a1c588809a5e90 the current folder structure (after building to filter out side effects from pyproject.toml config issues):

metaflow_extensions
└── resource_tracker
    └── plugins
        ├── mfextinit_resource_tracker.py
        ├── __pycache__
        │   └── mfextinit_resource_tracker.cpython-313.pyc
        └── track_resources
            ├── __init__.py
            ├── __pycache__
            │   ├── __init__.cpython-313.pyc
            │   └── track_resources_decorator.cpython-313.pyc
            ├── resource_tracker
            │   ├── __init__.py
            │   ├── __pycache__
            │   │   ├── __init__.cpython-313.pyc
            │   │   └── tracker.cpython-313.pyc
            │   └── tracker.py
            └── track_resources_decorator.py

also got rid of the symlink (as per above), and updated

mfextinit_resource_tracker.py

but still getting the same error when running

METAFLOW_DEBUG_EXT=1 python 1-single-step.py run

in the

examples/metaflow

folder 😞

I also modified the sample script to import the decorator as follows:

from metaflow import track_resources

unfortunately, I'm still getting the same error

Tagging @hundreds-rainbow-67050 who might be able to help as well.

oh, wow, I think I know facepalm these are the offending lines in my `pyproject.toml`:

[project.optional-dependencies]
metaflow = ["metaflow"]
all = ["resource-tracker[metaflow]"]

sounds like a metaflow extension pkg should not have optional dependencies?

Ah great

any ideas, though, why is that causing a problem? 🤐