Hello Metaflow team, thanks for sharing this project with the community. We are currently developing a R package that relies on several Python dependencies. I believe our packages are taking a similar approach to managing the Python deps. To be specific, my package is setting up a conda or virtual environment with a specific name, installing the deps within it, then using

reticulate::use_*env

or

reticulate::use_python

to activate the environment for use with the R package. When I try to test a flow that loads our package, it seems to fail at the step where my package is trying to find the Python environment it needs. I have attached a script and the error message, however, please know that the package I'm loading is not available to the public yet. I can share any code that might be helpful for understanding the problem. Do you have any general advice on how to use

reticulate

and Python within R packages so that they operate smoothly with other R packages that have their own Python dependencies? In addition, are there any Metaflow-specific adjustments I could make to my package to enable interoperability? Best, Justin

in general either of these approaches should work: 1. You let Metaflow/Python handle the environments and your R packages just tag along - they shouldn't set of create any virtual environments of their own. 2. You create your own environments (can be through R, or can be in a Docker image) and make sure that all Python/R dependencies are available there. Better to avoid using

@conda

in this case to make

reticulate

work nicer. Mixing the two approaches can lead to difficulties since the two systems are competing and overlapping. It sounds like you are taking the second approach, which is cool. Note that if you want to execute the code remotely, you need to make sure that the same environment is available in a Docker image used for remote executions. I'd suggest that you first make it work without Metaflow and once it works, then you can include Metaflow in the mix. Metaflow by itself shouldn't require anything special

Thanks for the advice @straight-shampoo-11124. You are correct in that I'm taking the second approach -- my R package creates its own virtual environment and installs the packages it needs. My R package and Python deps are working fine outside of the Metaflow ecosystem. The problem seems to be that I'm letting both packages take the second approach of creating their own Python env. After looking at the Metaflow code more, I see that I might be able to create a single Python environment with my deps for both packages and set the

METAFLOW_PYTHON

environment variable before loading the Metaflow package. I'll see if this solves the problem. Thanks again @straight-shampoo-11124.

let me know how it goes - happy to help more if needed