Hi everybody! I am trying to run a flow. I am getting an error at the step;

self.data_processing = DataProcessing(self.config, self.raw_dfs)

self.raw_dfs is a dictionnary of dataframes. See below for a more details:

@step
    def load_data(self):
        '''Loads the data from the database'''
        import pandas as pd
        import numpy as np
        from data_manager import DataManager

        self.data_manager = DataManager(self.config)
        self.data_manager.load_all_input_data()
        self.raw_dfs = self.data_manager.raw_dfs
        self.next(self.processing)

    @step
    def processing(self):
        '''Processes the data'''
        from data_processing import DataProcessing
        self.data_processing = DataProcessing(self.config, self.raw_dfs)

any advice ?

what is the error?

the error I got:

2023-01-12 11:39:00.660 [87/processing/382 (pid 8573)] <flow ResolvabilityTrainingFlow step processing> failed:
2023-01-12 11:39:04.607 [87/processing/382 (pid 8573)] Data store error:
2023-01-12 11:39:04.696 [87/processing/382 (pid 8573)] Could not unpack artifact 'ResolvabilityTrainingFlow/data/99/99ff93f506c20fe05a7bb84632e46cc86f789192': CRC check failed 0xa3013bdc != 0x7dc7f674

interesting - is the issue only present with this flow or is it affecting all flows that access a self attribute?

for now, I am just running one flow, so I can't answer.

can you try this flow - https://docs.metaflow.org/metaflow/basics#linear?

ok, I am running it, right now.

let's do another experiment - can you rename your flow -

ResolvabilityTrainingFlow

to something else and trying executing?

ok

It is likely that the gcs object was corrupted when written to (or being read from) gcs which is resulting in this error

running right now. The dataframes in raw_dfs are not so small: memory usage of ~250MB each (x8).

that should be okay - as long as you have sufficient resources on your workstation

still running. It might take a few more minutes.

Hi simon - I can continue helping here. I think the error msg "CRC check failed" comes from Gzip - we perform Gzip on all artifacts, regardless of application format. What would be great to verify is whether the Gzipped object in google storage itself is corrupted (or not). Would it be possible for you to find the google cloud storage blob itself, download it and calculate it's SHA1 hash. E.g. For the artifact

ResolvabilityTrainingFlow_2/data/8f/8f698a5010170f55432265e787943140dafa3

the SHA1 hash should be

8f698a5010170f55432265e787943140dafa3

+ is this running in Kubernetes? Or running locally (even if using GCP for storage?)

@elegant-carpenter-7681 We have a good idea what the issue is. There seems to be a bug with Gzip in Python stdlib, on M1 Macs (are you on M1?), which means artifacts > 2GB are not gzipped right (see the github issue link above). Immediate workaround is: try to stay under 2GB per artifact. In the meantime we are exploring potential improvements in metaflow to hide this Python x M1 issue internally.

@handsome-receptionist-54781 thank you very much for your help. My current setup is Macbook Pro M1, I am running the flow locally with GCP for Storage.

Got you... no additional debug info is needed from you at this point. As I mentioned above.... fastest way is to use sub 2GB artifacts

If I move to an remote machine, this should solve the issue right ?

would that be feasible for your use case?

Yes. In our tests, Linux on x86 CPU works fine.