Hi, I'm trying to use the Metaflow client API to access data from previous runs. Is it possible via the API to get the s3 location where the underlying artifacts corresponding to https://docs.metaflow.org/api/client#MetaflowData are stored?

you can access the artifacts directly from the client API via

from metaflow import Run, namespace

# set to global namespace so you aren't limited to your user namespace
namespace(None)
some_df = Run("FlowName/123").data.some_df

another option is to pass in the run id to the s3 client

from metaflow import S3

with S3(run=Run("FlowName/123") as s3:
    data = [obj.blob for obj in s3.get_many(urls)]
    ...

lastly, I believe you could also derive the s3 paths yourself using something like this https://outerbounds-community.slack.com/archives/C02116BBNTU/p1622485446115600?thread_ts=1622448797.113500&cid=C02116BBNTU

Thanks, last option you mentioned is what I was looking for. I don't actually want to download the artifact to my local machine but just need the s3 url

all the artifacts are pickled, gzip compressed, hashed, and saved in content-addressed storage (e.g. if the same artifact has already been stored from another execution, it won't be duplicated) -- because of that, each run stores the metadata needed to retrieve the associated data artifacts IIRC the actual pickled/compressed data is stored under a

data/

dir within the associated flow, e.g.

<s3://your-s3-bucket/metaflow/FlowName/data/><hashed value>/<hashed values>

I'm not entirely sure how that content-addressed storage should be queried, would be a better question for the metaflow team –– that said, it's probably not a good design pattern to piggy back on top of. If you need to access raw files in S3 directly, you probably should be storing them yourself and using a format you can design around (e.g. serializing a dataframe to parquet). You can still use metaflow's S3 client to do those writes for you, and then just store the pointer to the file in S3 alongside the flow for metadata tracking. That same pointer can be used directly elsewhere as you see fit