[conda dependecies] Hi, we’re dealing with the following scenario while deploying flow on AWS. Conda and dependencies are stored on S3, but we have obligatory lifecycle rule on bucket, which means after couple of days, flow crashes as cannot find cached conda on S3. Any easy way to always tell metaflow to download dependencies instead of using the cached ones from S3?

not 100% certain but I think the closest you can get is to use the metaflow-nflx-extensions - it exposes quite a bit of functionality around managing conda that could get you there. It's pretty janky but think you can get there using the example at the bottom of the conda docs in the repo: https://github.com/Netflix/metaflow-nflx-extensions/blob/main/docs/conda.md#variable-substitution

ooh, I don’t have a great solution for this. even the extension assumes that things are persistent. @flaky-plumber-70709’s suggestion of building an environment as part of your first step could work but you would lose a bunch of advantages for the system. It could also still fail in case the process that deletes files happens between the time you resolve your environment in the first step and you use it later. 2.10 also has a new conda implementation but it makes the same assumption around caching (in the sense that if you have a datastore defined, it will cache stuff). We could add an override to say “even if you have a datastore, don’t cache anything”. It would work probably but you may end up having other issues (the one we have typically seen is an overload of the pypi mirror you are using or some rate limiting). But ya, the best solution ideally is no LC management. If it’s a cost thing, that is hard to argue against. If it’s a data protection thing, the data stored in the conda cache is pretty innocuous.