Hi team, I am trying to play around

@tensorflow_parallel

. I installed it by command

pip install git+<https://github.com/netflix/metaflow.git@refs/pull/872/head>

But when I try to run a flow I get an error:

(tf_parallel) linz1@ADSKTR2CQ24X50 metaflow % python flow.py run                                                        
Traceback (most recent call last):
  File "/Users/linz1/Team/onboarding/metaflow/flow.py", line 1, in <module>
    from metaflow import (
  File "/Users/linz1/miniconda3/envs/tf_parallel/lib/python3.9/site-packages/metaflow/__init__.py", line 214, in <module>
    _import_plugin_decorators(globals())
  File "/Users/linz1/miniconda3/envs/tf_parallel/lib/python3.9/site-packages/metaflow/decorators.py", line 504, in _import_plugin_decorators
    from .plugins import STEP_DECORATORS, FLOW_DECORATORS
  File "/Users/linz1/miniconda3/envs/tf_parallel/lib/python3.9/site-packages/metaflow/plugins/__init__.py", line 139, in <module>
    from .frameworks.pytorch import PytorchParallelDecorator
ModuleNotFoundError: No module named 'metaflow.plugins.frameworks'

I seems I still need some other dependency packages? Does anyone have some ideas about how to solve this? Thanks!

@fast-dog-88545 - is it okay we will follow up on this with you later in the afternoon?

Oh sure that's good for me! Thanks a lot for your time!

followed up on this earlier thread on the same topic

Hi @crooked-jordan-29960, do you know from which channel should I install

tensorflow

and

cudatoolkit

with

@conda_base

?

looks like conda-forge latest cudatoolkit version is 11.8.0. any luck if you downgrade from version 11.9.0 in the @conda_base spec?

Hi @crooked-jordan-29960, I am still trying to use @tensorflow_parallel, this will be important to our data scientists. Thanks for all your help! I now have a problem that the step failed without any traceback error. Does this indicate anything in metaflow? The training codes are here, and I have checked the codes works without

@tensorflow_parallel

by this file, and here is a notebook of the training codes without metaflow.

Hi Ziqi! Sorry on delay here, will dive in later tonight and/or tomorrow am and verify I can reproduce.

Thank you so much @crooked-jordan-29960! I really appreciate your time!

awesome! so to be clear, you were able to run the task to completion? I imagine you still aren't seeing the tqdm progress bar though?

Yes the task finished successfully

if you remember, did that progress bar only appear at the very end of task lifecycle, or was it going throughout the model training like in your notebook?

Yes this is an issue, it's not like running without metaflow, which will give a dynamic progress bar. It will only show the bar after the training is done

ok cool. ya this is a general MF x TQDM issue that I'll put some thought into and suggest workarounds for

That sounds so good, the optimization will make a difference. Thanks for the amazing work! Please let me know when it's ready, my team would love to try

will do. glad the core of the tensorflow_parallel piece is working, please keep us posted as you test that out.

Oh and one more thing that in the metaflow UI it will stdouts the progress bar for each step but in a messy way

Hey @fast-dog-88545 I'm not sure if there is a great Metaflow-native solution for the tqdm business, but after thinking about this for a bit I think the best way to help you avoid headaches around this in the future is an idea about structuring Metaflow codebases to play nice with notebooks for iteration and then copy the results of that into the FlowSpec when the notebook code feels good to you. Check out the notebook in this PR on your repo for details.

Thanks for the wonderful work @crooked-jordan-29960. Another folk from my team find the same issue for @pytorch_parallel, your suggestion will definitely help us a lot

Glad to hear. It still isn't the most satisfying resolution the progress bar thing, but I think the best approach for that is to create some templates, one for tf and one for torch, that write custom messages to stdout in callbacks. I think doing that in the FlowSpec instead of relying on how these frameworks use tqdm is the piece that will actually solve the core issue.

cc: @acoustic-van-30942