Hi All, I'm having a problem running the NYCVizFlow() example from the Outerbounds Sandbox, with the difference being I'm trying to run it on my own stack that I opened via CloudFormation now. I'm getting an insane amount of conda conflicts when I was able to install this environment fine locally. Not sure what the issue is. I'll attach the logs in this thread.

sure! Try this:

@conda_base(python='3.9.10', libraries={"datashader": "0.14.0", "pandas": "1.4.2", "pyarrow": "5.0.0"})

I'll give it a try! I also had an issue where using

@conda("test-env.yaml)

returned this:

(base) honson@Aisan:~/metaflow$ CONDA_CHANNEL=conda-forge python flow2.py --environment=conda run
Traceback (most recent call last):
  File "/home/honson/metaflow/flow2.py", line 14, in <module>
    @conda("./test-env.yaml")
  File "/home/honson/.local/lib/python3.10/site-packages/metaflow/decorators.py", line 433, in _base_step_decorator
    raise BadStepDecoratorException(decotype.name, func)
  File "/home/honson/.local/lib/python3.10/site-packages/metaflow/decorators.py", line 33, in __init__
    deco=deco, func=func.__name__
AttributeError: 'str' object has no attribute '__name__'. Did you mean: '__ne__'?

Any ideas on this?

@conda

doesn't support

.yaml

definitions natively. You have to specify a

libraries=

attribute

oh? Does this mean the sandbox example is outdated?

correct,

@conda_base

is applied to all steps of a flow. You can use it to define a base set of libraries that all steps need and then you can optionally add

@conda

if some steps need some extra libraries

See screenshot of the sandbox!

the commented out line above

@conda_base(libraries={"datashader": "0.14.0", "pandas": "1.4.2", "pyarrow": "5.0.0"})
class NYCVizFlow(FlowSpec):

is what you get out of the box. Could it be that

test-env

is related to your testing?

so, the "out of the box" sample had

conda_base

commented out in replacement for the yaml that was not something I added in there

great to hear that you are writing a guide! We are happy to get you over any hurdles

loading... ⏳

note that if you are curious, you can install

mamba

as an alternative to Conda which helps make it a bit faster

Noted. Why does the conda environment take so long? You mentioned network speed. Is this yaml or dict not being send over through to the network to be setup "locally" on aws?

there’s a lot going on behind the scenes in `@conda`: it resolves the whole dependency graph and caches all packages in your S3 bucket so all tasks are guaranteed to get the same stable environment This is why it takes a why initially. Subsequent runs use the cached environment so it’s faster