https://outerbounds.com/ logo
Join Slack
Powered by
I realize this question has likely been asked 100'...
# ask-metaflow
l
I realize this question has likely been asked 100's of times, but interestingly, it seems to be one of the top reservations my team has for adopting Metaflow: Is there a way we can not use 
conda
and use other mechanisms to • document dependencies • lock dependencies I'll give some opinions and I fully understand if others do not agree with them:
✅ 1
• Our non-Data Scientists do not use 
conda
for their development work. Our team standardized on 
pip
, 
docker
and 
pip-compile
for documenting/locking dependencies. This includes documenting the Python language version and any non-python dependencies, e.g. things installed via 
apt-get
. • Our data scientists prefer to use the same tools as our non data scientists since this has made collaboration/support easier. Maybe we're unique, but our data scientists feel as strongly as our "engineers" about not using conda. • Hosting private conda channels is a pain (in our experience), but we have a company wide AWS CodeArtifact instance which acts as our private PyPI repository.
Ultimately, the biggest draws for us is that • Metaflow allows us to avoid kubernetes, while deploying on our own infratructure • Metaflow integrates with AWS Event Bridge and Step Functions, which makes it really nice to trigger from our other tools. Step Functions is also super reliable. • It runs using native AWS resources. Since we are heavily invested in AWS, that has many benefits when it comes to integration, e.g. we can use IAM auth. • Metaflow is nicer to "monitor" than vanilla Step Functions / AWS Batch because of it's UI
d
You can definitely use your own docker image with batch, sfn, etc if that is something you want to do. You can bake your own image with thr dependencies you need and use that to run your steps. There is also work going on to support a pure pip decorator which should be coming at some point. If you want to try something now, there is an extension based on what we have internal at netflix that you can find here https://github.com/Netflix/metaflow-nflx-extensions. I think @wooden-dusk-90720 has used it with CodeArtifact successfully. It’s not as supported as the “official” metaflow but if you have issues with it, I can usually resolve it fairly quickly. It has a pip and pip_base decorator.
🎉 1
w
Ya we're using pip_base with our internal CodeArtifact repo and it's been working fine so far
🎉 1
d
I should add I guess that the extension provides a sort of environment management that allows you to document your dependencies. You can give it a requirements.txt and it will resolve it for you and show you what was locked. The locked environment can be saved and also named (like docker tags) and then you have commands like get and show to “document” what the environment has. It also has support somewhat for what you are saying with non Python dependencies but those are still via conda (not sure if that is ok with you). Internally for example, people depend on ffmpeg which is a conda package (also via apt-get). You can specify those with an extension in the requirements.txt syntax. The underlying infra still does use conda to create the base environment.
🎉 1
e
FWIW we’re using it with poetry+Docker and also all seems to work just fine
🎉 1
l
Sorry for taking ages to respond. This is so great to hear! I'm so excited that Metaflow is growing in this way!
6 Views
Open in Slack
PreviousNext
Powered by