https://outerbounds.com/ logo
Join Slack
Powered by
Hi, another `metaflow.Runner` thing to note: there...
# dev-metaflow
b
Hi, another 
metaflow.Runner
thing to note: there is some global state set when running a flow that isn't cleaned, in my case again with flow decorators. I have two flows with different class names in 
flow1.py
and 
flow2.py
that each use 
@project(name="myproject")
. This crashes with `metaflow.decorators.DuplicateFlowDecoratorException: Flow already has a decorator 'project'. You can specify each decorator only once."`:
Copy code
with metaflow.Runner("flow1.py").run() as running:
    assert running.run.successful

with metaflow.Runner("flow2.py").run() as running:
    assert running.run.successful
Any idea how to work around this?
full traceback:
Traceback (most recent call last):
File "/Users/max.friedrich/c/metaflow-212-test/test.py", line 6, in <module>
with metaflow.Runner("flow2.py").run() as running:
File "/Users/max.friedrich/Library/Caches/pypoetry/virtualenvs/metaflow-212-test-CfFDw3qu-py3.10/lib/python3.10/site-packages/metaflow/runner/metaflow_runner.py", line 268, in init
self.api = MetaflowAPI.from_cli(self.flow_file, start)
File "/Users/max.friedrich/Library/Caches/pypoetry/virtualenvs/metaflow-212-test-CfFDw3qu-py3.10/lib/python3.10/site-packages/metaflow/runner/click_api.py", line 183, in from_cli
flow_parameters = extract_flowspec_params(flow_file)
File "/Users/max.friedrich/Library/Caches/pypoetry/virtualenvs/metaflow-212-test-CfFDw3qu-py3.10/lib/python3.10/site-packages/metaflow/runner/click_api.py", line 151, in extract_flowspec_params
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/Users/max.friedrich/c/metaflow-212-test/flow2.py", line 5, in <module>
class TestFlow2(FlowSpec):
File "/Users/max.friedrich/Library/Caches/pypoetry/virtualenvs/metaflow-212-test-CfFDw3qu-py3.10/lib/python3.10/site-packages/metaflow/decorators.py", line 416, in wrap
return _base_flow_decorator(decofunc, f, **kwargs)
File "/Users/max.friedrich/Library/Caches/pypoetry/virtualenvs/metaflow-212-test-CfFDw3qu-py3.10/lib/python3.10/site-packages/metaflow/decorators.py", line 404, in _base_flow_decorator
raise DuplicateFlowDecoratorException(decofunc.name)
metaflow.decorators.DuplicateFlowDecoratorException: Flow already has a decorator 'project'. You can specify each decorator only once.
Seems like it happens when the current interpreter executes the flow file in 
extract_flowspec_params
In my previous tests I was just running 
python flow1.py ...
in a subprocess without checking which parameters the flow has, so I never had to run the flow file in the main test process
a
@billowy-exabyte-36648 thanks for the bug report! We are actively looking into this
🙌 1
b
@billowy-exabyte-36648 I think the fix is in https://github.com/Netflix/metaflow/pull/1878 Maybe you can try that branch and let me know? Thanks!
b
thank you! the fix works for me, no need to 
FlowSpec._flow_decorators = {}
between tests anymore 😄
9 Views
Open in Slack
PreviousNext
Powered by