https://outerbounds.com/ logo
Join Slack
Powered by
Hi, I am trying to read images from a S3 bucket, t...
# ask-metaflow
s
Hi, I am trying to read images from a S3 bucket, the URI to the image is read from the CSV file, which is dumped to a pandas csv. I am noticing a weird pattern in the reading time, after reading 4 images (with approx 0.07ms per image) the 5th image is taking approx 200 secs, there are 64 images in a batch and total 119 batches. This is slowing down the entire training pipeline. This is how my dataloader's getitem() looks like:
Copy code
def __getitem__(self, index):
        with S3() as s3:
            byte_data = s3.get(self.data.iloc[index]['url']).blob
        if byte_data is None:
            return self.__getitem__((index + 1) % len(self.data))
        img = Image.open(BytesIO(byte_data)).convert('RGB')
        if self.transform is not None:
            img = self.transform(img)
        if self.target_transform is not None:
            label = self.target_transform(self.data.iloc[index]['label'])
        else:
            label = self.data.iloc[index]['label']
        return img, label
Is there anything which I might have missed out in the code which is slowing the loading process?
Extras: This is the init constructor for the loader:
Copy code
def __init__(self,
                 root,
                 split='train',
                 transform=None,
                 target_transform=None,
                 batch_size=64):
        self.root = root
        self.transform = transform
        self.target_transform = target_transform
        self.split = split
        self.batch_size = batch_size
        self.data = pd.read_csv(self.root)
        self.target_transform = target_transform
        self.data = pd.read_csv(self.root).drop(columns=['geometry'])
        self.data['count'] = self.data.groupby('url')['url'].transform('count')
        self.data = self.data.drop_duplicates(subset=['url'])
        self.data['label'] = self.data['count'].apply(
            lambda x: 0 if x == 0 else 1 if x >= 1 and x < 5 else 2)
        self.data = self.data.drop(columns=['count'])
        if split == 'train':
            self.data = self.data[self.data['split'] == 'train']
        elif split == 'val':
            self.data = self.data[self.data['split'] == 'val']
v
hey Ashish! Firstly, retrieving many images in parallel with 
s3.get_many
is likely to be faster than retrieving them sequentially with 
s3.get
. It's possible that a batch of 64 images is too small in the IO point of view, although it might make sense for the model. If you really want to optimize your data loader, you can use larger IO batches, e.g. 256 images, to make 
get_many
go faster, and then feed them as 4*64 batches to your model
a single S3 operation takes about 50-100ms at the minimum, so 0.07ms/image seems incorrect if it includes the S3 operation too You can try
Copy code
from metaflow import profile

with profile('load image'):
    ...your code here...
to get more accurate timings
s
Hello @victorious-lawyer-58417!, How should I use 
get_many
inside my getitem? Like how can I specify multiple keys? I didn't understand your second answer regarding profile, could you please redirect me to any related documentation?
I tried using 
get_many
to read all keys inside the folder of S3 bucket, I am getting the following error when I am trying to access the item using its index.
Copy code
with S3() as s3:
   self.images = s3.get_many(self.data['url'].tolist())
Modified get item:
Copy code
def __getitem__(self, index):
        byte_data = self.images[index].blob            
        if byte_data is None:
            return self.__getitem__((index + 1) % len(self.data))
        img = Image.open(BytesIO(byte_data)).convert('RGB')
        if self.transform is not None:
            img = self.transform(img)
        if self.target_transform is not None:
            label = self.target_transform(self.data.iloc[index]['label'])
        else:
            label = self.data.iloc[index]['label']
        return img, label
I was able to find the file, which the reader is trying to open, but it's at a path (./metaflow_x.y.z) different from the path it is trying to read from.
I would like to highlight that high reading time from S3 bucket using metaflow S3 is only while running the flow on AWS batch, it runs at a decent speed locally (without metaflow orchestration + batc)
3 Views
Open in Slack
PreviousNext
Powered by