Hey All,
For our workflow orchestrations, we use ...
# ask-metaflowq
quaint-sandwich-34433
01/30/2025, 6:02 PMHey All,
For our workflow orchestrations, we use Metaflow with AWS Batch. When triggering workflows with large input parameters, we receive the following error on AWS Batch jobs:
"Container Overrides length must be at most 8192."
In AWS, there is no option to increase the "Container Overrides length". Is there an alternative way to pass large input parameters while using Metaflow with AWS Batch?
Best regards,
Raaj
✅ 1
a
ancient-application-36103
01/30/2025, 6:21 PMHi! What version of metaflow are you on?
q
quaint-sandwich-34433
01/31/2025, 9:19 AMHello,
Thanks for the response. Please find the details below.
Metaflow Service - netflixoss/metaflow_metadata_service:v2.4.12
Metaflow SDK - metaflow-2.11.15
s
square-wire-39606
01/31/2025, 3:55 PMcan you try again with a newer version of metaflow - 2.11.15 is quite old
q
quaint-sandwich-34433
01/31/2025, 5:18 PMSure, will try and post you. Thank you!
👍🏼 1
quaint-sandwich-34433
02/06/2025, 2:15 PMHi there, we are facing the same issue with "Metaflow 2.13.9" as well. We use AWS Batch for compute and AWS Step Functions for orchestration.
Here when we run following
"python3 paramerflow.py step-functions trigger --alpha 'very-large-input'"
we encounter error on step function execution that states ""Container Overrides length must be at most 8192."
a
ancient-application-36103
02/06/2025, 3:33 PMdo you have a lot of parameters etc.? step functions has a limit of 8192 characters for container overrides that is hard to work around - this was one of the reasons why we started supporting argo-workflows.
👍 1
h
hundreds-rainbow-67050
02/06/2025, 3:46 PMWe used to get around this by having all our parameters in a single file using IncludeFile, and a second parameter for all the overrides.
this 1
👍 1
q
quaint-sandwich-34433
02/07/2025, 4:29 AMThank you all for the suggestions! We will use the IncludeFile option here.
4 Views