https://outerbounds.com/ logo
Join Slack
Powered by
Hello! First time question here! We're evaluating ...
# ask-metaflow
o
Hello! First time question here! We're evaluating Metaflow as a potential tool for our organization. We are using GCP. I managed to deploy Metaflow service and UI using a (slightly modified version of) the Helm charts provided by 
metaflow-tools
and configured a GCS bucket as my data store. I can run some of the tutorial flows from my local machine + view the runs in the UI, so far so good! However, when I try to view the DAG structure or the task logs, I get an error message indicating that the 
google-cloud-storage
dependency is missing (error message below in thread). How I fix this? Happy to provide more details if necessary!
Copy code
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 307, in <module>
    cli(auto_envvar_prefix='MFCACHE')
  File "/opt/latest/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/opt/latest/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/opt/latest/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/latest/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 301, in cli
    Scheduler(store, max_actions).loop()
  File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 199, in __init__
    maxtasksperchild=512,  # Recycle each worker once 512 tasks have been completed
  File "/usr/lib/python3.7/multiprocessing/context.py", line 119, in Pool
    context=self.get_context())
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 176, in __init__
    self._repopulate_pool()
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 241, in _repopulate_pool
    w.start()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
    code = process_obj._bootstrap()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/root/services/ui_backend_service/data/cache/client/cache_worker.py", line 29, in execute_action
    execute(tempdir, action_cls, request)
  File "/root/services/ui_backend_service/data/cache/client/cache_worker.py", line 56, in execute
    invalidate_cache=req.get('invalidate_cache', False))
  File "/root/services/ui_backend_service/data/cache/get_log_file_action.py", line 143, in execute
    results = {**existing_keys}
  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/root/services/ui_backend_service/data/cache/utils.py", line 130, in streamed_errors
    get_traceback_str()
  File "/root/services/ui_backend_service/data/cache/utils.py", line 124, in streamed_errors
    yield
  File "/root/services/ui_backend_service/data/cache/get_log_file_action.py", line 136, in execute
    current_hash = log_provider.get_log_hash(task, logtype)
  File "/root/services/ui_backend_service/data/cache/get_log_file_action.py", line 270, in get_log_hash
    return get_log_size(task, logtype)
  File "/root/services/ui_backend_service/data/cache/get_log_file_action.py", line 177, in get_log_size
    return task.stderr_size if logtype == STDERR else task.stdout_size
  File "/opt/latest/lib/python3.7/site-packages/metaflow/client/core.py", line 1317, in stdout_size
    return self._get_logsize("stdout")
  File "/opt/latest/lib/python3.7/site-packages/metaflow/client/core.py", line 1438, in _get_logsize
    return self._log_size(stream, meta_dict)
  File "/opt/latest/lib/python3.7/site-packages/metaflow/client/core.py", line 1525, in _log_size
    ds_type, ds_root, stream, attempt, *self.path_components
  File "/opt/latest/lib/python3.7/site-packages/metaflow/client/filecache.py", line 148, in get_log_size
    return task_ds.get_log_size(LOG_SOURCES, logtype)
  File "/opt/latest/lib/python3.7/site-packages/metaflow/datastore/task_datastore.py", line 45, in method
    return f(self, *args, **kwargs)
  File "/opt/latest/lib/python3.7/site-packages/metaflow/datastore/task_datastore.py", line 413, in get_log_size
    sizes = [self._storage_impl.size_file(p) for p in paths]
  File "/opt/latest/lib/python3.7/site-packages/metaflow/datastore/task_datastore.py", line 413, in <listcomp>
    sizes = [self._storage_impl.size_file(p) for p in paths]
  File "/opt/latest/lib/python3.7/site-packages/metaflow/plugins/datastores/gs_storage.py", line 195, in size_file
    import google.api_core.exceptions

ModuleNotFoundError: No module named 'google'
v
did you use the Docker images provided by us or your own images? The former should work out of the box. Otherwise you should 
pip install google-cloud-storage
o
I used the following images:
Copy code
# For UI backend + metadata service
  image:
    repository: public.ecr.aws/outerbounds/metaflow_metadata_service
    pullPolicy: IfNotPresent
    tag: "v2.3.11"

  # For static UI
  uiImage:
    repository: public.ecr.aws/outerbounds/metaflow_ui
    pullPolicy: IfNotPresent
    tag: "v1.3.3"
v
o
Redeployed using the default image versions, same error EDIT: new error, see below FTR, defaults listed in TF template are: • 
public.ecr.aws/outerbounds/metaflow_metadata_service:v2.3.3
• 
public.ecr.aws/outerbounds/metaflow_ui:v1.1.4
Copy code
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 307, in <module>
    cli(auto_envvar_prefix='MFCACHE')
  File "/opt/latest/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/opt/latest/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/opt/latest/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/latest/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 301, in cli
    Scheduler(store, max_actions).loop()
  File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 199, in __init__
    maxtasksperchild=512,  # Recycle each worker once 512 tasks have been completed
  File "/usr/lib/python3.7/multiprocessing/context.py", line 119, in Pool
    context=self.get_context())
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 176, in __init__
    self._repopulate_pool()
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 241, in _repopulate_pool
    w.start()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
    code = process_obj._bootstrap()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/root/services/ui_backend_service/data/cache/client/cache_worker.py", line 29, in execute_action
    execute(tempdir, action_cls, request)
  File "/root/services/ui_backend_service/data/cache/client/cache_worker.py", line 56, in execute
    invalidate_cache=req.get('invalidate_cache', False))
  File "/root/services/ui_backend_service/data/cache/generate_dag_action.py", line 97, in execute
    results[result_key] = json.dumps(dag)
  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/root/services/ui_backend_service/data/cache/utils.py", line 130, in streamed_errors
    get_traceback_str()
  File "/root/services/ui_backend_service/data/cache/utils.py", line 124, in streamed_errors
    yield
  File "/root/services/ui_backend_service/data/cache/generate_dag_action.py", line 93, in execute
    dag = DataArtifact("{}/_graph_info".format(param_step.task.pathspec)).data
  File "/opt/latest/lib/python3.7/site-packages/metaflow/client/core.py", line 904, in data
    return filecache.get_artifact(ds_type, location[6:], meta, *components)
  File "/opt/latest/lib/python3.7/site-packages/metaflow/client/filecache.py", line 214, in get_artifact
    [name],
  File "/opt/latest/lib/python3.7/site-packages/metaflow/client/filecache.py", line 243, in get_artifacts
    ds = self._get_flow_datastore(ds_type, ds_root, flow_name)
  File "/opt/latest/lib/python3.7/site-packages/metaflow/client/filecache.py", line 360, in _get_flow_datastore
    storage_impl = self._get_datastore_storage_impl(ds_type)
  File "/opt/latest/lib/python3.7/site-packages/metaflow/client/filecache.py", line 349, in _get_datastore_storage_impl
    raise FileCacheException("Datastore %s was not found" % ds_type)

metaflow.client.filecache.FileCacheException: Datastore gs was not found
If I understand the above errors correctly, the default (old) image versions from the Helm chart do not yet support the 
gs
datastore, so those will never work. The newer images do have support for 
gs
but to me it seems as if the required dependencies for Google were simply forgotten... • 
local
datastore does not require extra deps • 
s3
datastore is natively supported by 
metaflow
since it pulls in 
boto3
• 
azure
datastore deps are listed in ui_backend_service/requirements.txt • 
gs
datastore deps are nowhere! Should I open an issue for this? The only other solution I see is to build my own image, but I'd like to avoid that.
I just tested adding 
google-cloud-storage
to 
services/metadata_service/requirements.txt
and built my own version of the metadata service image, this fixed the issue.
a
@orange-secretary-61629 would you like to create a PR with your fix? we can expedite a release so that you don't have to bother with a custom image
o
🙌 1
s
I will merge as soon as the tests pass
o
Great thanks!
@square-wire-39606 looks like tests have passed! 🎂
s
done!
5 Views
Open in Slack
PreviousNext
Powered by