Hitting the dreaded 8192 character limit with step functions again (ref https://github.com/Netflix/metaflow/issues/1482) One thing which would help is to remove the

"note": "Internal representation of IncludeFile(…)"

from the parameters Is there any reason not to do that? (I can’t see how it could be used anyway), or at least allow it to be set rather than always defaulting The main problem is the length of

"ContainerOverrides": {
                                            "Command": [
                                                "bash",
                                                "-c",
                                                "true && mkdir -p $PWD/.logs && ....

Do you have a lot of parameters? We used to get around that by having all our parameters in a single file using IncludeFile, and a second parameter for all the overrides.

Not many - 2 IncludeFiles both of which end up taking up quite a bit of the allocation it’s sufficiently close that removing the note makes it work

hey, we removed the url in 2.15.11 -- can you give it a try and see if it solves your issue?

Thanks - that helps a bit It’s a rather annoying arbitrary 8192 limit on AWS as can be seen by the number of times there has been an issue raised for it The only obvious thing I can see to make a decent sized difference would be to put the batch command string into a file (sent to S3) and invoke that instead of the full thing Not difficult but would need a little bit of refactoring - could be done for just step or all batch