https://outerbounds.com/ logo
Join Slack
Powered by
Hi all, What would be the correct way to parallel...
# ask-metaflow
a
Hi all, What would be the correct way to parallelize downloads of some image files in s3 using Metaflow's s3 client? They would need to be saved to a local folder. So far, I did something like this:
Copy code
from metaflow import S3
import os
with S3(s3root='<s3://metaflow-s3-ampsdemo/test_data/sample_training/>') as s3:
    if not os.path.exists('sample_training'):
       os.makedirs('sample_training')
    res = s3.get_all()
    for obj in res:    
        with open(f'sample_training/{obj.key}', 'w') as f:
            f.write(obj.text)
✅ 1
Follow-up question - I'm working with a model that relies on local paths for the 3 different separate data sources. I was going to use Metaflow to download these different data sources to 3 local paths that I would then pass the directories into PyTorch model? Looks like I can perhaps also leverage 
mounted_volume
within the 
@batch
decorator. Do I need to load in the data within the same task as the task used to load in the model or can it be in a separate task?
v
you don't need the last step that writes it to a file, you can simply refer to 
obj.path
as long as you are inside the 
S3
scope
a
Should Metaflow users typically load in data and write to local directory within the same task as the loading in the model that relies on these local paths? It would seem that you would have to do that in order to stay in scope
v
if you need to download data from 3 separate sources and then pass the resulting 3 directories, you can do something like this:
Copy code
SOURCES = ['s3://.../path1', 's3://.../path2', 's3://.../path3']
sources = []
dirs = []
for src in SOURCES:
    s3 = S3(s3root=src)
    sources.append(s3)
    objs = s3.get_all()
    if objs:
        dirs.append(os.path.dirname(objs[0].path))

print('do something with data in directories', dirs)

for src in sources:
    src.close()
here 
dirs
contain the data downloads from 
SOURCES
. Local copies get deleted in the end when you do 
src.close()
you want to do all this inside one step, since when you execute tasks remotely e.g. with 
@batch
, you can't rely on local directories across steps/tasks
note that counterintuitively the above pattern can be faster than loading data from local disk, if you run it on large enough instances
a
This is perfect! Thanks so much Ville! Very helpful.
👍 1
9 Views
Open in Slack
PreviousNext
Powered by