Hi, We have an old Metaflow setup (2.5.0) on AWS, deployed using CloudFormation. Trying to upgrade this, but locally while giving a job using an upgraded Metaflow (2.10), inside Batch we get a s3 access denied error

2023-11-15 09:02:46.980 [10656/run_on_batch/265393 (pid 13400)] [065eb4d7-9332-4629-be20-5a60aeca9a9e]     stdout_lines, stderr = self._s3op_with_retries(
2023-11-15 09:02:46.980 [10656/run_on_batch/265393 (pid 13400)] [065eb4d7-9332-4629-be20-5a60aeca9a9e]   File "/metaflow/metaflow/plugins/datatools/s3/s3.py", line 1697, in _s3op_with_retries
2023-11-15 09:02:46.981 [10656/run_on_batch/265393 (pid 13400)] [065eb4d7-9332-4629-be20-5a60aeca9a9e]     ) = try_s3_op(last_ok_count, pending_retries, out_lines, inject_failures)
2023-11-15 09:02:46.981 [10656/run_on_batch/265393 (pid 13400)] [065eb4d7-9332-4629-be20-5a60aeca9a9e]   File "/metaflow/metaflow/plugins/datatools/s3/s3.py", line 1681, in try_s3_op
2023-11-15 09:02:46.981 [10656/run_on_batch/265393 (pid 13400)] [065eb4d7-9332-4629-be20-5a60aeca9a9e]     raise MetaflowS3AccessDenied(err_out)
2023-11-15 09:02:46.981 [10656/run_on_batch/265393 (pid 13400)] [065eb4d7-9332-4629-be20-5a60aeca9a9e] metaflow.plugins.datatools.s3.s3.MetaflowS3AccessDenied: s3op failed:
2023-11-15 09:02:46.981 [10656/run_on_batch/265393 (pid 13400)] [065eb4d7-9332-4629-be20-5a60aeca9a9e] Access denied to URL: <s3://metaflow-stack-metaflows3bucket>-<pvt>/metaflow/conda/conda.anaconda.org/conda-forge/linux-64/tk-8.6.13-noxft_h4845f30_101.conda
2023-11-15 09:02:46.981 [10656/run_on_batch/265393 (pid 13400)] [065eb4d7-9332-4629-be20-5a60aeca9a9e]
2023-11-15 09:02:49.863 [10656/run_on_batch/265393 (pid 13400)] Task failed.
2023-11-15 09:02:49.975 Workflow failed.
2023-11-15 09:02:49.975 Terminating 0 active tasks...
2023-11-15 09:02:49.976 Flushing logs...
    Step failure:
    Step run_on_batch (task-id 265393) failed.

It works on a fresh setup of upgraded Metaflow though. I can compare different Cloudformation templates which was used to deploy, but just wondering if there was a migration guide somewhere, of what changed between different templates

I don't think much has changed permission wise 🤔 That error may mean a few things 1. GetObject permissions missing 2. ListBucket permission missing. If you don't have that and request an object that doesn't exist. They return 403 in this case to make it harder to list bucket by bruteforcing object names I guess 3. KMS permissions missing for the key used for the bucket I'd check on those. There is also slight possibility that new metaflow triggers a different code path and it doesn't try to read that object and it doesn't trigger (2) above.