https://outerbounds.com/ logo
Join Slack
Powered by
Hi all. I’m trying to use metaflow with AWS batch ...
# ask-metaflow
m
Hi all. I’m trying to use metaflow with AWS batch to download artifacts from another flow. The problem is that the artifacts are too large and I don’t have enough disk space. Can I utilize the built in artifact system with a mounted EFS volume? So essentially I’d have metaflow download artifacts directly to EFS rather than the local batch disk
✅ 1
a
m
yeah I saw that, but we still wouldn’t be able to use the abstracted artifact methods and would have to manually download flow data from s3 with metaflow.s3 right?
a
you would be able to use the artifact methods - they use metaflow.s3 behind the scenes
m
Okay got it. So I can use the memory datastore option with the artifact methods but not an EFS volume
a
you could use an EFS volume too - but perf might be a bit suss
m
Could you help guide me into how to use the efs volume with the native artifact methods?
If I attach an efs volume will it automatically handle it under the hood?
I can’t find any documentation on it
a
you can point this config variable to the attached EFS mount
is there a specific reason you would want to use EFS over tmpfs?
the artifacts are not technically downloaded to 
disk
m
Not quite yet, I just want to know so I can do some pro / cons. It’ll come down to cost considerations and if we need the speed that tmpfs provides or if efs is enough
👍🏼 1
a
tmpfs
is the easiest to play with
m
“the artifacts are not technically downloaded to 
disk
” - so they’re loaded straight into memory? My engineers are reporting that they’re getting disk space issues though when trying to load stuff that’s too big
I’ll need to have them specify what that “stuff” is they’re trying to load
a
ideally even that wouldn't be needed but AWS Batch makes it really difficult to change the disk specification for the nodes
m
yep exactly
a
this was one of the major reasons for us to offer a native integration to kubernetes - where many of these issues can be handled much better
m
Yeah I mean one day we’ll get to k8s but we’re a super small team and would rather not have the operational load of an eks cluster right now
We’ll do some investigation into the artifacts with tempfs vs artifacts with efs and make a choice
thanks for your help
a
sounds good! also happy to put you in touch with folks on our side if you are ever curious about the managed offering that runs k8s behind the scenes
m
Thanks, okay for right now as we haven’t done the investigation but I’ll mention it to my manager
👍🏼 1
@ancient-application-36103 sorry for the direct ping, but you’re saying if we add use_tempfs=True then any artifacts we pull down should go to tempfs rather than the local (non memory) disk?
because artifacts use metaflow.s3 on the backend
nvm that worked.
12 Views
Open in Slack
PreviousNext
Powered by