https://outerbounds.com/ logo
Join Slack
Powered by
Hello Metaflow friends (and hi Ville and Savin, lo...
# ask-metaflow
c
Hello Metaflow friends (and hi Ville and Savin, long time no speak!)! We're finally ramping up on Metaflow (we've got a K8s cluster running on GCP, we're testing out features one at a time, etc.). I've run into an issue with using the 
@conda
decorator, in which I get a 
json.decoder.JSONDecodeError
when I try to run the flow. Will paste the full error, as well as the Flow definition (it's a dummy flow) in the thread. INTERIM SOLVE: downgrade conda to 
22.11.1
. In later versions of conda, 
conda info
(or 
self._info()
) no longer outputs JSON. Hat tip to @faint-rocket-4408 for pointing this out.
✅ 1
👀 1
Here's the error:
Copy code
(mflow2) davidkale@ringil metaflow-tools % python test_flow.py --environment=conda run
Metaflow 2.9.11 executing TestFlow for user:davidkale
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Bootstrapping conda environment...(this could take a few minutes)
    Internal error
Traceback (most recent call last):
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/cli.py", line 1172, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/cli.py", line 691, in wrapper
    return func(args, kwargs)
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/cli.py", line 828, in run
    before_run(obj, tags, decospecs + obj.environment.decospecs())
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/cli.py", line 886, in before_run
    obj.package = MetaflowPackage(
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/package.py", line 73, in __init__
    deco.package_init(flow, step.__name__, environment)
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/plugins/conda/conda_step_decorator.py", line 355, in package_init
    self._prepare_step_environment(step, self.local_root)
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/plugins/conda/conda_step_decorator.py", line 216, in _prepare_step_environment
    env_id = self._resolve_step_environment(ds_root)
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/plugins/conda/conda_step_decorator.py", line 124, in _resolve_step_environment
    CondaStepDecorator.conda = Conda()
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/plugins/conda/conda.py", line 43, in __init__
    if LooseVersion(self._info()["conda_version"]) < LooseVersion("4.6.0"):
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/site-packages/metaflow/plugins/conda/conda.py", line 111, in _info
    return json.loads(self._call_conda(["info"]))
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/davidkale/miniconda3/envs/mflow2/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
a
meow wave for sure! happy to debug
c
Here's the flow:
Copy code
from metaflow import FlowSpec, step, schedule, conda
from metaflow import conda_base


@conda_base(python='3.10.6')
class TestFlow(FlowSpec):
    """
    A flow where Metaflow prints 'Hi'.

    Run this flow to validate that Metaflow is installed correctly.

    """

    # @conda(disabled=False)
    @step
    def start(self):
        """
        This is the 'start' step. All flows must have a step named 'start' that
        is the first step in the flow.

        """
        print("Step (start)")
        self.next(self.a)

    @conda(libraries={'pandas': '1.5.1'})
    @step
    def a(self):
        """
        A step for metaflow to introduce itself.

        """
        print("Step (a)")
        self.next(self.end)

    # @conda(disabled=False)
    @step
    def end(self):
        """
        This is the 'end' step. All flows must have an 'end' step, which is the
        last step in the flow.

        """
        print("Step (end)")


if __name__ == "__main__":
    TestFlow()
I ran it with the command 
python test_flow.py --environment=conda run
a
can you help me with how did you install 
conda
?
c
And finally, I'm running this locally on my 2020 M1 Macbook Pro with Ventura (in case that matters).
I installed it with the Miniconda package installer. In case it's also relevant, it's worth pointing out that I'm also using pyenv on the same machine for other projects.
a
thanks that helps. let me try to reproduce
c
Finally, in case it's not already super complicated: last night, I swear I was able to execute this once successfully. Then I tried to run it on our cluster with 
--with kubernetes
and got a DIFFERENT error, related to the ncurses package and file system case insensitivity. After that, I started getting this json decode error. I unfortunately didn't record that error and so far haven't been able to reproduce it.
Thanks @square-wire-39606! I'm going to run the kids to camp and then will be back shortly. Will keep an eye on Slack (when not driving, of course!).
among us party 1
f
I also encountered this. I think it's because "conda info" doesn't output json in the new version anymore (self._info() method as the error reflects). Downgrading to conda to version 22.11.1 solves the issue until it's fixed.
👀 1
c
Thanks, @faint-rocket-4408! Lemme give that a shot!
Hey that appears to have done it @faint-rocket-4408 @square-wire-39606. I wound up doing a clean install of miniconda 22.11.1, and subsequently running with 
--environment=conda
worked great (locally...gonna try 
--with kubernetes
next). Savin, should I file an issue on github?
a
Yes please - I will take a look at what changed with the recent conda release
👍 1
c
11 Views
Open in Slack
PreviousNext
Powered by