Hi, I've a project structure like this -

primary-module
..primary_module
....model.py
..secondary-module
....secondary_module
......util.py
....pyproject.toml
..metaflow_run.py
..setup.py

And I'm running

metaflow_run.py

- this depends on the

secondary_module

which is a separate python project cloned locally here (path is being resolved as

sys.path.insert(0, secondary-module)

This flow has been running for a few months However, currently, it seems to be taking a very old code of

secondary_module

(old version of

util.py

) The code was,

secondary_module

used to be a project with

setup.py

, which got changed into

poetry

with

pyproject.toml

But I'm not sure how that should affect, since the whole source is anyway cloned here Is there a way to debug why this is happening?

when you refer to a very old code being used - are you running locally or remotely on AWS Batch/Kubernetes?

remotely from Batch and sure, will check it with the package list is there a way to check the full source code files of what was pushed to Batch? or does package list give this itself?

it gives the path to all the files that are included. you can peek inside them to check content

it lists my local path, and the file is updated there but when it runs on batch, it seems to be taking an older version of the same file is it uploading the files to s3? caching it somewhere? is there a way to see that?

you can check the code that was executed using the metaflow client -

Run("flow_name/run_id").code.tarball

For me

Run("flow_name/run_id").code

is coming as

None

I'm on Metaflow 2.7.11 Is there some other issue going on here?

did that run have any steps that ran on AWS Batch or Kubernetes? We only store the code at the moment for those runs where at least one step is executing remotely

yeah, the step which is failing is running on Batch, so ideally it should have been logged

but did the step for the

run_id

run on

@batch

?

yeah, it does show in the logs, starting

run_id

, then in the same logs, running on remote machine, then that particular step fails

the step that failed executed on AWS Batch?

yeah

it is guaranteed to be the same code tarball that you downloaded using

Run.code

- did the docker image also contain any prebaked remnants of the code?

how do I check the docker image? I'm using

@conda

to define the dependencies, not

@image