Hi guys. I am running metaflow in minikube. When uploading a dataset into minio using the S3 client, but i am having the flow paused in the transfer of the

.arrow

files. Metaflow only transfer the lighter files, but the .arrow files are not transfered, and the flow is stuck. Do you know what can it be?

{"local": "/tmp/open-perfectblend/dataset_dict.json", "url": "<s3://metaflow-test/metaflow/data/perfect-blend/dataset_dict.json>", "key": "dataset_dict.json", "content_type": null}
{"local": "/tmp/open-perfectblend/train/data-00004-of-00006.arrow", "url": "<s3://metaflow-test/metaflow/data/perfect-blend/train/data-00004-of-00006.arrow>", "key": "train/data-00004-of-00006.arrow", "content_type": null}
{"local": "/tmp/open-perfectblend/train/data-00005-of-00006.arrow", "url": "<s3://metaflow-test/metaflow/data/perfect-blend/train/data-00005-of-00006.arrow>", "key": "train/data-00005-of-00006.arrow", "content_type": null}
{"local": "/tmp/open-perfectblend/train/state.json", "url": "<s3://metaflow-test/metaflow/data/perfect-blend/train/state.json>", "key": "train/state.json", "content_type": null}
{"local": "/tmp/open-perfectblend/train/dataset_info.json", "url": "<s3://metaflow-test/metaflow/data/perfect-blend/train/dataset_info.json>", "key": "train/dataset_info.json", "content_type": null}
{"local": "/tmp/open-perfectblend/train/data-00001-of-00006.arrow", "url": "<s3://metaflow-test/metaflow/data/perfect-blend/train/data-00001-of-00006.arrow>", "key": "train/data-00001-of-00006.arrow", "content_type": null}
{"local": "/tmp/open-perfectblend/train/data-00002-of-00006.arrow", "url": "<s3://metaflow-test/metaflow/data/perfect-blend/train/data-00002-of-00006.arrow>", "key": "train/data-00002-of-00006.arrow", "content_type": null}
{"local": "/tmp/open-perfectblend/train/data-00000-of-00006.arrow", "url": "<s3://metaflow-test/metaflow/data/perfect-blend/train/data-00000-of-00006.arrow>", "key": "train/data-00000-of-00006.arrow", "content_type": null}
{"local": "/tmp/open-perfectblend/train/data-00003-of-00006.arrow", "url": "<s3://metaflow-test/metaflow/data/perfect-blend/train/data-00003-of-00006.arrow>", "key": "train/data-00003-of-00006.arrow", "content_type": null}

Can you share the code snippet?

I am using

put_files

in the s3 client

def _upload_directory(self, local_path: str, store_key: str = ""):
        final_path = os.path.join(self._store_root, store_key)
        with S3(s3root=final_path) as s3:
            s3.put_files(self._walk_directory(local_path))

We haven't stress tested against minio - the s3 client, as the name suggests, works against S3. @hundreds-rainbow-67050 do you have any ideas here?

can you try uploading one of the failing files manually using the aws cli? that will help narrow down the issue

aws --endpoint-url http://<minio-url> \
    s3 cp data-00000-of-00006.arrow \
    <s3://metaflow-test/test.arrow>

Hi guys. Thanks for the idea. Debugging i found that the minio server was crashing when uploading any large file. The thing i did was to increase the deployment resources using

kubectl edit deployment minio

. Now its working fine!!

resources:
          limits:
            cpu: "1"
            memory: 2Gi
          requests:
            cpu: "1"
            memory: 2Gi

you can edit the tiltfile: https://github.com/Netflix/metaflow/blob/master/devtools/Tiltfile#L126

is it the one located in

/home/user/venv/share/metaflow/devtools/Tiltfile

oh yeah if you installed it via pip

Its not taking the changes. Maybe because is a package file 🤔

did you restart tilt? maybe clean all and make again?

still nothing. Deleted containers, images and did

metaflow-dev up

again. I am gonna put this detail on my README. thanks Nissan!!