Hello, is it normal that when running the metaflow service and and metaflow ui I have this error regarding the dag display ?

Hey Lamia - yes, the UI runs against data that is stored in the DB and S3. If you are not using them, then you will see partial information at best.

Hi thanks @square-wire-39606, actually I have more knowledge on ML and DL, and I'm not familiar with AWS. If I want to have the ui with all features, I understood that I'll have to follow the instructions from this link right ?https://outerbounds.com/engineering/deployment/aws-managed/cloudformation/

@flaky-plumber-48220 can you copy and share that whole stack trace from your screenshot?

Traceback (most recent call last): File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 307, in <module> cli(auto_envvar_prefix='MFCACHE') File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in call return self.main(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 301, in cli Scheduler(store, max_actions).loop() File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 199, in init maxtasksperchild=512, # Recycle each worker once 512 tasks have been completed File "/usr/local/lib/python3.7/multiprocessing/context.py", line 119, in Pool context=self.get_context()) File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 176, in init self._repopulate_pool() File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 241, in _repopulate_pool w.start() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 112, in start self._popen = self._Popen(self) File "/usr/local/lib/python3.7/multiprocessing/context.py", line 277, in _Popen return Popen(process_obj) File "/usr/local/lib/python3.7/multiprocessing/popen_fork.py", line 20, in init self._launch(process_obj) File "/usr/local/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch code = process_obj._bootstrap() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/root/services/ui_backend_service/data/cache/client/cache_worker.py", line 29, in execute_action execute(tempdir, action_cls, request) File "/root/services/ui_backend_service/data/cache/client/cache_worker.py", line 56, in execute invalidate_cache=req.get('invalidate_cache', False)) File "/root/services/ui_backend_service/data/cache/generate_dag_action.py", line 97, in execute results[result_key] = json.dumps(dag) File "/usr/local/lib/python3.7/contextlib.py", line 130, in exit self.gen.throw(type, value, traceback) File "/root/services/ui_backend_service/data/cache/utils.py", line 130, in streamed_errors get_traceback_str() File "/root/services/ui_backend_service/data/cache/utils.py", line 124, in streamed_errors yield File "/root/services/ui_backend_service/data/cache/generate_dag_action.py", line 93, in execute dag = DataArtifact("{}/_graph_info".format(param_step.task.pathspec)).data File "/usr/local/lib/python3.7/site-packages/metaflow/client/core.py", line 835, in data obj = filecache.get_artifact(ds_type, location[6:], meta, *components) File "/usr/local/lib/python3.7/site-packages/metaflow/client/filecache.py", line 216, in get_artifact [name], File "/usr/local/lib/python3.7/site-packages/metaflow/datastore/task_datastore.py", line 364, in load_artifacts for (key, blob) in self._ca_store.load_blobs(to_load.keys()): File "/usr/local/lib/python3.7/site-packages/metaflow/datastore/content_addressed_store.py", line 140, in load_blobs with open(file_path, "rb") as f: TypeError: expected str, bytes or os.PathLike object, not NoneType

Thanks

hi @brave-lion-15961 any news on my issue ?

Can you confirm your

metaflow

version and your

metaflow-service

version?

metaflow-service

version can be found under the

Quick Links

menu, and one way you can see the

metaflow

version is to open the

Details

menu of a run. (see attached)

Hi @brave-lion-15961 I have metaflow_version:2.8.2 with python_version:3.8.10

Can you click on The Quick Links button?

image.png

If you just cloned a couple of weeks ago, it’s likely you have a new version. Just to check, take a look inside the setup.py file for the version (around line 25)

@brave-lion-15961 my version is version="2.3.8", from the setup.py

Your versions are close to the latest, so the issue with the DAG's is not version incompatibility. We will have to dig in deeper.

ok ! thanks a lot !

Do you get the same error for all flows? Or just one in particular?

all of them

Can you share a flow? Perhaps just with the structure intact if you don't want to share proprietary code?

well for instance when I try running the flow hello world flow of the tutorial I have this issue. How should I share it

OK - I wanted to rule out problems with your flow - so that's not it

@brave-lion-15961 no not yet, I wanted first to resolve this issue before setting up the s3

The DAG's are only stored in S3, not locally, so you will have to setup S3 to see them for now.

ok I'll try that ! thanks a lot !