Dear Metaflow Community, We’ve encountered a weird behavior when upgrading from metaflow 2.12.12 to 2.12.18. In one of the steps, that runs in a custom docker container with aws cli installed, we use aws cli to download large files in parallel into the docker container. Then line that does this is provided below. Everything runs fine in 2.12.12 but fails in 2.12.18 with the following error message:

FileNotFoundError: [Errno 2] No such file or directory: 'aws'

Line that downloads files:

aws_cli_result = subprocess.run(
  [
    "aws",
    "s3",
    "cp",
    source,
    destination,
    *extra_args,
    "--recursive",
    "--only-show-errors",
  ],
  capture_output=True,
)

I was wondering if it is a bug or might be a container issues that 2.12.18 is exposing.

@fast-school-58057 - in

2.12.18

- we removed metaflow's dependency on awscli. if you are reading/writing to s3, then metaflow's s3 client might be a good fit

Thanks for clarification, I’ll try to look why the aws cli installed in the docker container is not accessible. Probably it was never accessible and we always relied on the metaflow one without realizing. 😅

yep - that's very likely 🙂