I see that

artifacts

are available of all the variables. What if I just want to know the paths for the artifacts, such s3, azure or Ali cloud path from which the data was read in any of the steps in the

end

step. Is that possible?

artifacts are Metaflow's internal constructs, which it serializes, compresses, and stores in a content-addressed data store using its internal mechanism. If you want to share data with external systems, it is better to use

metaflow.S3

so you get to control the serialization and location (optionally). It's pretty straightforward:

with metaflow.S3(run=self) as s3:
   url = s3.put('myobject', pickle.dumps(obj))
print('data available at', url)

We are multi cloud. What about Azure and Ali. Also, what if I want to read data with Spark?

we don't have a special client for Azure and GCP but some of them provide S3-compatible APIs which might work. Otherwise you can use their native Python libraries

you can save the URLs as Metaflow artifacts so data lineage get persisted in runs

We want users of the library to not worry about storing urls in the artifacts but rather collect them automatically (like

autolog

does in

MetaFlow

(but it's buggy) or what

openlineage

listener to

spark

does) when their code is running. This is to avoid them missing out on any external sources they may have accessed but forgot to enter them in url artifact. Also, there might large number of paths they may read. It would not be possible to ask them to enter to all of them.

Please correct me if I'm mistaken on understanding that you are referring the developer of the flow to make sure that store the urls in artifacts when developing the code.

I'm still reading the documentation on MetaFlow so I might have knowledge gap on some of the concepts.

let me know if you have any specific questions about it

Good morning. Is it possible to use @pyspark with emr serverless and a custom ecr docker image? https://aws.amazon.com/blogs/big-data/add-your-own-libraries-and-application-dependencies-to-spark-and-hive-on-amazon-emr-serverless-with-custom-images/ Or should I create a standby emr cluster? Thank you very much 🙏

it should be possible but I haven’t tried. I’ll get back to you