https://outerbounds.com/ logo
Join Slack
Powered by
Hello team, as you guys might remember, time to ti...
# ask-metaflow
q
Hello team, as you guys might remember, time to time there have been discussions around ECS’s 8192 character limit on 
ContainerOverrides
object. I have noticed that earlier you guys attempted to patch it up by compressing the amount of information metaflow sends compulsorily, but another case where this limit is breached is if the user input itself is large enough. In our case we stumbled upon this issue when we started use a json payload which can a couple of KBs large sometimes. This combined with information that metaflow already attaches makes the 
ContainerOverride
object again exceed the hard 8192 limit 😕 As a short term workaround we are exploring passing input params through a persistent storage like s3/dynamo but I am just wondering if this is something metaflow team would consider supporting natively? So a blob-like or json-like input param type that is passed through a dynamodb/s3 layer implementaiton-wise. Perhaps then, in fact, all payloads can be passed as such uplifting this restriction forever?
The reason a native support would be more meaningful is that it can be more easily interpreted by things such as our UI dashboard service and metadata service while responding to queries. So from a end-user perspective the fact that the parameter is stored in dynamo would be completely transparent. Doing it at a userspace means that our datascientists inspecting flow runs through the api would see the raw s3 path/dynamodb paths instead of the actual value.
In case I was not super clear in what I mean, by native support I meant (taking step functions as example) metaflow generating a SFN definition which has an additional step at the start to write the parameters directly to metaflow’s CAS / 
_parameters
object and then actual python steps can just read this file (which they already do via 
input_paths
). By an additional “step” I don’t mean the step in the metaflow sense, but a step function “state” which directly calls s3 apis (i.e. not an ECS task otherwise there’s the same problem).
This is not foolproof as step functions themselves have a payload size restriction of 256KB I believe, but much better than the absurd 8KB restriction from ECS.
a
Totally! A workaround for now would be to rely on 
IncludeFile
for your parameters - https://docs.metaflow.org/scaling/data#data-in-local-files
Also, is my understanding correct that the issue is not between Metaflow steps passing the parameters between themselves (that already happens via s3) but passing parameters from the user console to step functions?
we don't have much control available to us in the later scenario - the first opportunity to execute any arbitrary code is within the 
_parameters
task that you have correctly identified, but using 
IncludeFile
will signal to Metaflow that the parameter object needs to be read from s3.
q
but passing parameters from the user console to step functions?
Yes. More specifically, translating step functions input to ECS task definition that is used to run the step. The resulting task definition embeds the entire input in the environment variables (for at least the 
start
step) which can result in breaching the 8192 character limit. The workaround could be that we have a non-metaflow step even before the 
start
step that saves the step functions input to a well-known s3 file that could be then read by the 
start
step. I wasn’t aware about 
IncludeFile
primitive, that looks neat for my short term needs, I”ll try that out, thanks!
Oh actually it appears that include files can not point to a s3 url? https://github.com/Netflix/metaflow/blob/c030d1ddf8d7dc1148020e3dea15309040f90256/metaflow/includefile.py#L167 If we are deploying a flow as a step function, then whenever executing it in production with a different input we would want to point it to a s3 file where we have saved the json input. Is it possible?
2 Views
Open in Slack
PreviousNext
Powered by