https://outerbounds.com/ logo
Join Slack
Powered by
Hi all, Wanted to play around with the experiment...
# ask-metaflow
a
Hi all, Wanted to play around with the experimental 
@pyspark
decorator. The documentation is a bit lacking (understandably so since it's experimental), but looks like it's a Metaflow extension. Was also looking into 
awswrangler
as well, which works quite well too. I tried doing 
python setup.py install
, but I get the following error:
Copy code
RuntimeError: Conflicts in 'metaflow_extensions' files:
    Packages 'metaflow-pyspark[pyspark]', and '/opt/conda/envs/pyspark/lib/python3.10/site-packages/metaflow_pyspark-0.0.1-py3.10.egg/metaflow_extensions[pyspark]' define the same configuration module 'metaflow_extensions.pyspark.plugins.__init__'
✅ 1
d
This error typically means that metaflow is finding the extension twice. Try uninstalling it and the installing it from pypi. You shouldn’t need to run the setup.py. If that doesn’t work, re-run with 
METAFLOW_DEBUG_EXT
and I may be able to glean additional information about why it is finding the extension twice.
a
Okay got it - thank you. Will give it a try.
Here's the additional information:
Copy code
metaflow_extensions: Found packages present at {'/home/sagemaker-user/spark-flow/metaflow_extensions', '/opt/conda/envs/pyspark/lib/python3.10/site-packages/metaflow_extensions'}
metaflow_extensions: Found extension package 'metaflow-pyspark'...
metaflow_extensions:     Top-level 'pyspark' extends 'plugins' with config 'metaflow_extensions.pyspark.plugins.__init__'
metaflow_extensions: Ignoring duplicate package 'metaflow-pyspark' (duplicate paths in sys.path? (['/home/sagemaker-user/spark-flow', '/opt/conda/envs/pyspark/lib/python310.zip', '/opt/conda/envs/pyspark/lib/python3.10', '/opt/conda/envs/pyspark/lib/python3.10/lib-dynload', '/opt/conda/envs/pyspark/lib/python3.10/site-packages', '/opt/conda/envs/pyspark/lib/python3.10/site-packages/certifi-2023.5.7-py3.10.egg']))
metaflow_extensions: 'metaflow_extensions' distributions order is ['metaflow-pyspark']
metaflow_extensions: Non installed packages present at {'/opt/conda/envs/pyspark/lib/python3.10/site-packages/metaflow_extensions'}
metaflow_extensions: Walking path /opt/conda/envs/pyspark/lib/python3.10/site-packages/metaflow_extensions (package name _pythonpath_0)
metaflow_extensions:     Extends 'plugins' with config 'metaflow_extensions.pyspark.plugins.__init__'
Traceback (most recent call last):
  File "/home/sagemaker-user/spark-flow/sparkflow.py", line 1, in <module>
    from metaflow import FlowSpec, step, IncludeFile, Parameter
  File "/opt/conda/envs/pyspark/lib/python3.10/site-packages/metaflow/__init__.py", line 54, in <module>
    from metaflow.extension_support import (
  File "/opt/conda/envs/pyspark/lib/python3.10/site-packages/metaflow/extension_support/__init__.py", line 794, in <module>
    _all_packages, _pkgs_per_extension_point = _get_extension_packages()
  File "/opt/conda/envs/pyspark/lib/python3.10/site-packages/metaflow/extension_support/__init__.py", line 756, in _get_extension_packages
    raise RuntimeError(
RuntimeError: Conflicts in 'metaflow_extensions' files:
    Packages 'metaflow-pyspark[pyspark]', and '/opt/conda/envs/pyspark/lib/python3.10/site-packages/metaflow_extensions[pyspark]' define the same configuration module 'metaflow_extensions.pyspark.plugins.__init__'
d
ok, so it seems you have the metaflow-pyspark package in both 
home/sagemaker-user/spark-flow/metaflow_extensions
and 
/opt/conda/envs/pyspark/lib/python3.10/site-packages/metaflow_extensions
so metaflow gets confused. The messages are basically telling me: • you have the package in those two places • both places are in sys.path (that’s the duplicate message — it’s telling you that it is ignoring the second package it found) I can actually make a small fix that will actually ignore the second package. Here it is ignoring it once when it does processing of sys.path but then still goes and looks for it. I can have a small patch for you to do this. You can alternatively remove one of the two packages and it should work.
a
Ah OK thanks. I can just remove one of the packages then and that should be fine. Thanks very much!
d
here is the patch that should fix it as well (even if you have both packages): https://github.com/Netflix/metaflow/pull/1437
It’ll go out with the next release.
a
Awesome, thanks so much!
d
sure — lmk if that doesn’t fix it for you.
16 Views
Open in Slack
PreviousNext
Powered by