Hi, there. I have question regarding flow artifacts. I also understand that one wants to avoid storing data artifacts (like ingested data (raw), train and test data).....but if your steps in a flow depend on the parent step. How would it be possible to exclude actual data from being persisted as an artifact? Lets say your flow consists of the following steps: ingest data, transform data, split data, train model....now since train model step requires the train data from the parent step I would have to define self.X_train, self.X_test, self.Y_train, self.Y_test on the split data step (hence this data is now going to become an artifact), similarly when I want to transform data it depends on the parent step ingest data, now in order to pass the data to the transform step I have to define self.data in the ingest data step (which now becomes another data artifact)......I could collapse all these steps into one step and just make use of method local scope variables (omit the self key word) in which case it will now no longer persist these data artifacts, but then my step no longer pertains to only doing one unit of work....Please can you shed some light for me on this, is there something I'm missing?

Hey Byron - for larger data sets - you can choose to write them to s3 directly and only persist the s3 path as the artifact. That way you can manage the lifecycle of s3 objects independently. https://docs.metaflow.org/scaling/data#store-and-load-objects-in-a-metaflow-flow

Hi Savin, thanks for the reply I saw this section of the documentation, and I think it will still require me to persist the data. In this case the data would come from s3 in the ingest_data step, but this has to be passed into the transform_data step and this step can only reference the data from ingest_data if it has self.data (example). Alternatively I could read from S3 and transform in one step and persist the transformed data to S3, then in the next step read the transformed and split it into train and test, and write this back to S3, then in fit model step read from S3 again....and so on...It does not seem to me like I would be able to get away from not having to persist data at all..

@clever-pencil-68951 the data would need to live somewhere if you are passing it from one step to another. With

metaflow.s3

the performance implications of shuttling data between ec2 and s3 are addressed considerably from a throughput point of view

Thanks Savin, what I wanted to achieve with the framework was to: read data in from a source layer (S3/RDS/DynamoDB) at the start of a flow in an ingestion step, process it via the various flow steps in memory (not persisting bits of data as it moves through the flow), persist only relevant variables & parameters as artifacts of interest. At the end of the flow write the data results to a data layer S3 (external to the flow, so no need for the flow artifacts to include any reference to data)