Hi everyone, I have this folder structure for one of my projects with metaflow. └── folder/ ├── subfolder_1/ │ └── flow.py ├── subfolder_2/ │ ├── flow.py │ └── config_2.json └── global_config.json The flows are deployed on AWS and I would like to understand the reason why the flow in subfolder_2 has access to the global_config.json file but not the config_2.json file. I do something like this at the beginning of flow.py in subfolder_2:

parent_folder = Path(__file__).parent.parent

global_configuration_path = parent_folder / "global_config.json"

current_folder = Path(__file__).parent

local_configuration_path = current_folder / "config_2.json"

Since I forgot to add ".json" to the package suffixes when creating the step function, I was thinking that it would be failing while reading the global_config.json file but it does not seem to be the case, and it fails while reading config_2.json. By the way, I use a custom docker image where needed dependencies are installed, but all the files in folder are copied as well. I don't know if that could explain the behaviour. Thanks for your support 🙏

you need to add

python flow.py --package-suffixes=.json run

to include all

.json

files in the package. See these docs for guidance how to structure projects in general. Alternatively, you can use a JSON-typed

Parameter

or

IncludeFile

to include configuration in your runs

Thanks, I currently use a json typed parameter, but I would prefer to read content of json files as default value. Would you know why the json file located one level above in the folder structure is accessible without adding the .json extension to package suffixes?

that's mysterious. If you run

python flow.py package list

, do you see the JSON files in the listing?

I see none of them, and when I add the .json to the package suffixes it adds only the local one config_2.json

ok, that sounds more like the expected behavior. I wonder if it's possible that

global_config.json

is already included in your custom Docker image

Yes it is already included! I was wondering where and when was the mechanism that uses the package suffixes to filter uploaded file

Metaflow includes all

.py

files and files with suffixes listed in

--package-suffixes

that are in the same directory as your

flow.py

, including all subdirectories Hence in your case

config_2.json

should be included when you do

--package-suffixes=.json

but

global_config.json

won't. If you want to include it, you can add a symlink to it in

subfolder_2

Yes adding the .json to the list of suffixes gave me access to the

config_2.json

. I still have access to

global_config.json

so the symlink is not needed, but what you described is what I would have expected. I will try to test in the coming days if I remove the json from the docker image if it still works or not. Thanks for your support and have a nice day !

great, you too