Hello Outerbounds, we are running a flow in Argo, which has a foreach with

2191

pods, all is good, it runs nicely, but in the join step, we get an error in

init

exec /bin/argoexec: argument list too long

The same flow runs with no issues in K8s (without argo). It seems to be an issue related to the max size of env variable

ARGO_TEMPLATE

in the

init container

that has a limit of 128kb (source), have any of you encountered this issue before?

What’s the version of argo-workflows you’re running? There this recent comment on the same thread that says the issue has been fixed in some of the later versions

We are using v3.4.8.

are you able to reproduce if you upgrade to v3.4.11?

We got the same error, in the

init

container of our

Join

step. I am not really sure how to address this issue. Is there a way to make the

Join

step

init

container not to have the list of steps to join repeated 3 times? the variable ARGO_TEMPLATE in the container looks like this (attached, too large to paste it here). Our join step looks like this:

@kubernetes(disk=KUBERNETES_DISK)
@step
def join(self, inputs):
    from lab.projects.sport_basic.detection_model.data_generation.helpers import create_subset_json create_dataset_metadata,

    inputs = [input for input in inputs if not input.process_failure]
    self.merge_artifacts(inputs, include=["config"])
    self.combined_data = [(input.data_splits, input.data) for input in inputs]

    create_subset_json(config=self.config, combined_data=self.combined_data)
    create_dataset_metadata(self.config, flow=current.flow_name)

    self.next(self.end)

Where the step before is a foreach of +2000... Thank you in advance for the help 🙏

this seems like a regression - we will address it promptly

Github issue for tracking: https://github.com/Netflix/metaflow/issues/1538