Hello, We would like, as part of real-time monitoring of our Metaflow workflows, to log through calls to an external API every time a step finishes, or when a workflow concludes, or when an error occurs in a specific task. This logging is done through a library we've developed that sends messages to a PostgreSQL database. We've taken inspiration from what can be done with other scheduling tools like Airflow and Prefect, namely using callback functions. The equivalent in Metaflow would be a decorator. So, we've tried to make custom decorators on our end, but the only implementation that worked involves putting the following code in an

__init__.py

file:

from metaflow.decorators import step, _import_plugin_decorators 
from metaflow.plugins import STEP_DECORATORS 
from .custom_step_decorator import TestDecorator 

STEP_DECORATORS.append(TestDecorator) 

_import_plugin_decorators(globals())

This allows us to import the decorator into the workflow code itself and apply it to the steps. However, we find this approach complex and are not sure how it fits into deployment on AWS with Step Functions. What would be the best way to proceed? On a related note, the package we use for logging is on GCP Artifact Registry. Is there a straightforward way to import it with GCP credentials, perhaps via the PyPI tag? How can we import this package for the entire workflow without having to do it for each step? Thank you for your response and have a great day!

Hello — you can check out extensions: https://github.com/Netflix/metaflow-extensions-template. As the documentation notes, this is advanced usage and things may break as we evolve Metaflow but it will work one way or another (we are heavy users of extensions and other ppl are using it too). It’s definitely more supported than the snippet you have above 🙂

Hi, thanks for your answer ! I looked at previous messages and saw this discussion : https://outerbounds-community.slack.com/archives/C02116BBNTU/p1697501486790269 I tried to go through with it but am stuck close to a solution I think. When I try to bootstrap the environment, it doesn't find my custom package, even though the mamba command includes the right --index-url (when I try to install with the pip install --index-url, this works) I get the following error message :

ERROR: Could not find a version that satisfies the requirement job-tracker-ers==0.0.9 (from versions: none)

ERROR: No matching distribution found for job-tracker-ers==0.0.9

It's as if it didn't look in the index-url. Do you have any ideas on the matter ? Thanks again !

is this with the default

@pypi

or the bleeding edge one?

Default one

yes — the bleeding edge one uses mamba by default but you can have it use micromamba if you want too (see here: https://netflix.slack.com/archives/C02116BBNTU/p1697644492961639?thread_ts=1697501486.790269&cid=C02116BBNTU).

Oh you might be right on the pip used ! Great idea, I'm going to dig into that

so some details on how the decorator handles indices: • everything is read from the host systems pip.conf via

pip config list

• if there's an index-url or extra-index-url, these are passed as part of the pip commands inside the underlying conda environment ◦ this is also why there is a limitation regarding authentication, where currently the credentials need to be a part of the url. • if there is a custom index-url configured, this will override pip defaults, leading to no lookup in pypi index ◦ custom indices like AWS codeartifact can be configured with an upstream to make public packages available ◦ alternatively the default pypi repo can be added as an extra index in the config

<https://pypi.org/simple>

Hi, first thank you for all the details on the matter ! I configured the upstream link to pypi from cloud artifact. In my pip.conf, I only have the URL with the password to CodeArtifact. When I try your command with my private package, it only works when I remove the --only-binary=all

excellent 🙂