https://outerbounds.com/ logo
Join Slack
Powered by
Hi everyone - bit new here! Just a quick question ...
# ask-metaflow
s
Hi everyone - bit new here! Just a quick question - it seems like AWS provide support for configurable ephemeral_storage for Batch on Fargate - is this exposed anywhere in the Metaflow API at all? Edit - seems to be only exposed in the job definition so I suppose not exposed in the 
@batch
decorator for example. Does anyone have a workflow for varying the size of the ephemeral store - multiple job definitions and the flows which need higher ephemeral storage get routed to that particular job definition?
a
interesting - it's not enabled yet but we can look into introducing it in 
@batch
. at the moment, you can use 
tmpfs
to mount memory as disk https://docs.metaflow.org/scaling/data#using-metaflows3-for-in-memory-processing
s
Thanks @ancient-application-36103. The use case is for large GIS raster files (which ultimately come in as numpy arrays). At the moment the serialisation to pickle format to be able to reuse the objects in subsequent steps is taking up a lot of both memory and tmp space. We are already at 72GB memory for the batch step to avoid running out of memory (but /tmp fills up), so we are not too far from Fargate limits once you add in the tmpfs space. So I think a way to configure ephemeral_storage would be good for our use case, but there may be other potential optimisations. It would be a lot more memory efficient to do everything in a single step but to me that negates some of the benefits of Metaflow.
👍🏼 1
Just tried the tmpfs route - is it me or is there a code path where you're not guaranteed to have the key 
["linuxParameters"]
in 
["containerProperties"]
in the job definition - it seems to be added if the platform is EC2 or spot, but not Fargate. I'm certainly getting an error: 
File ".../metaflow/plugins/aws/batch/batch_client.py", line 302, in _register_job_definition
job_definition["containerProperties"]["linuxParameters"]["tmpfs"] = [
KeyError: 'linuxParameters'
On version 2.9.7 but couldn't see anything in later versions that would address it
👀 1
FYI this got rid of the error for me (on master) 
diff --git a/metaflow/plugins/aws/batch/batch_client.py b/metaflow/plugins/aws/batch/batch_client.py
index 8ef2607..fda1e4e 100644
--- a/metaflow/plugins/aws/batch/batch_client.py
+++ b/metaflow/plugins/aws/batch/batch_client.py
@@ -298,7 +298,8 @@ class BatchJob(object):
else:
+++ b/metaflow/plugins/aws/batch/batch_client.py
@@ -327,6 +327,9 @@ class BatchJob(object):
# default tmpfs behavior - <https://man7.org/linux/man-pages/man5/tmpfs.5.html>
tmpfs_size = int(float(memory)) / 2
+            if "linuxParameters" not in job_definition["containerProperties"]:
+                job_definition["containerProperties"]["linuxParameters"] = {}
+
job_definition["containerProperties"]["linuxParameters"]["tmpfs"] = [
{
"containerPath": tmpfs_path,
a
do you want to create a PR? 🙏🏼
s
Should be able to yes, just need to confirm I can.
👍🏼 1
Put in a PR
Actually, think it makes sense as tmpfs isn't applicable to Fargate, might repurpose the PR to add docs. detailing this.
So I think this makes the case for adding an 
ephmeral_storage
parameter stronger. I've put in a new issue detailing this detailing this
@ancient-application-36103 PR for the above - I've only been able to check it produces the right parameters so far due to perms but looks good so far. I'm not here next week but it's there if you want to take a look
a
thanks! we will review this early next week
s
Hi savin, thanks for that. I just made a minor update to the PR to comply with formatting standards, hopefully will pass tests now.
👍🏼 1
Now tested on real infra and writes the expected job definition
3 Views
Open in Slack
PreviousNext
Powered by