is there some limitation on what kind of objects are able to Outerbounds #ask-metaflow

is there some limitation on what kind of objects a...

wooden-dusk-90720

01/27/2023, 4:39 PM

is there some limitation on what kind of objects are able to be serialized in

self

? I'm getting an error about "weakly-referenced object no longer exists", which I believe is coming from my pytorch model. I'm also not directly creating any weakly referenced objects in my code (to the best of my knowledge anyway)

Copy code

2023-01-27T03:45:37.502-08:00	File "/metaflow/metaflow/datastore/task_datastore.py", line 282, in pickle_iter
	2023-01-27T03:45:37.502-08:00	blob = pickle.dumps(obj, protocol=2)
	2023-01-27T03:45:37.502-08:00	ReferenceError: weakly-referenced object no longer exists

✅ 1

ancient-application-36103

01/28/2023, 1:20 AM

@wooden-dusk-90720 is this related to this thread?

wooden-dusk-90720

01/28/2023, 1:21 AM

Yea it is. I think dill works fine, but haven't confirmed yet

victorious-lawyer-58417

01/28/2023, 1:21 AM

hmm, usually

pytorch

models serialize fine (see e.g. this example). It's a complex project though so it might be something specific to your project

victorious-lawyer-58417

01/28/2023, 1:24 AM

dill

is pretty magical - sometimes it appears to work but the results can be unexpected. If you want a more robust method, using the native serialization method of the modeling library is usually the best bet For instance here's an example for Keras - then you can do

Copy code

self.model = save_model(my_model)

and

Copy code

my_model = load_model(self.model)

colossal-jewelry-36870

02/07/2023, 1:55 PM

Also replied in the linked thread, but wanted to mention that the issue seemed linked to the `Trainer`class's method

.tune()

from PyTorch Lightning more precisely

🙌 1

Open in Slack

Previous Next