The human-centric platform for production ML & AI

Outerbounds

Hey guys,
During a `foreach` flow, where are the class variables stored? I’m running a pipeline with many parallel tasks that keeps failing with this error (after some time) `bash: line 1:    34 Killed`
 I’m assuming it fails due to lack of memory

the artifacts (variables attached to `self`) are persisted at the end of each `@step` , so if a step is failing you will not be able to access that artifact itself, but you can access it's parent steps artifacts (e.g. to reproduce locally while debugging)

<https://docs.metaflow.org/metaflow/client>

Yes, but where are they stored? In an external storage like S3 or in memory?

That makes total difference when I have 600+ foreach instances, each one with persisted data to then be used in the join step

there's various backend datastores that are supported, but typically it's S3 -- it depends on how you deployed the metaflow stack

if you're using a local-only setup, then the files are local on your machine

you don't need to reference the storage locations directly when accessing the artifacts through metaflow -- it handles that for you