https://outerbounds.com/ logo
Join Slack
Powered by
Hi everyone. I'm new to metaflow and I think it is...
# ask-metaflow
c
Hi everyone. I'm new to metaflow and I think it is an amazing tool. My use case would be related to start using it as a data transformation tool from the beggining. So I have buit one simple workflow to transform data and generate a pandas dataframe by the end. The next step would be saving it somehow. So I have two simple questions: 1. Is there anyway we can store on metaflow directory some structured data, like .parquet files? It seems metaflow already deals with S3 in a decent fashion, but I'm not sure about what would be the best practices to serialize new files incrementally to disk; 2. I have seen from the discussions that there is currently no way to have composite workflows with dependencies. However, can I use one workflow to provide data to the other? Like, the ETL workflow would save some .parquet files that the training workflow will use. Thank you in advance. Really loving this tool.
✅ 1
v
hi Eduardo! 👋 great questions
you can store files are on disk like you would do in any Python script. However, using a datastore like S3 makes it very convenient to make sure that 1. data is available in all environments for processing and analysis 2. it gets versioned properly (you can leverage Metaflow artifacts for this) 3. it is stored durably counterintuitively, the S3 performance can be higher than performance on local disk, so it can make processing faster too when managing large amounts of data
for small amounts amount data (a few gigabytes), check IncludeFile, which allows you to include local files as artifacts in Metaflow runs
re: your second question - absolutely! You can use the Client API to fetch results of the first flow in a second flow, like here
if you have large amounts (tens of gigabytes or more) of Parquet files, you can store a pointer to an S3 location in a Metaflow artifact instead of storing the data itself. This way you can handle arbitrary amounts of data - we have workflows handling terabytes in this fashion
for more information, see how-tos in the data section. Also the whole chapter 7 in the Metaflow book is dedicated to data topics (if you happen to want to buy the book, you can get 35% off with the code `au35tuu`_*)*_
hope these pointes get you started! We are happy to help you with further questions too 🤗
c
Wow, that's a lot of information. Thank you for the help. Let me go through all this material, but I believe there is a good path to follow already.
22 Views
Open in Slack
PreviousNext
Powered by