https://outerbounds.com/ logo
Join Slack
Powered by
Hiya folks :slightly_smiling_face: I'm making a m...
# ask-metaflow
c
Hiya folks 🙂 I'm making a metaflow flow right now to train and validate a Markov model! I tried to create a step that saved my BigQuery client as a class attribute, as per the metaflow documentation, to use it in multiple steps (both for training and for validating the model). I got an error stating that the object is unpicklable. Investigating that error message eventually led me to discover that Metaflow pickles objects intended to be kept as artifacts. So then I instead instantiated a new BQ client in each of the steps of the flow that require one. That got me past that error. I then get this at the end of the step that instantiates the model and attempts to save it as an attribute on the flow class (the model class inherits directly from 
object
😞
Copy code
_pickle.PicklingError: Can't pickle <function root at 0x11e52ccc0>: attribute lookup root on __main__ failed
✅ 1
s
👋🏽 @cold-carpenter-83628 when you are instantiating your BQ client are you assigning it to self?
self.bq_client = <create my awesome BQ client>
c
@shy-address-41011 I am! Or I was. I was doing 
self.client = bigquery.Client(project="mozdata", location="us")
That's when I was getting the 'this object is unpicklable' error at step 2: 
instantiate_client
. When I moved that line to step 4 where I actually need the client, my print messages indicated that I was getting all the way to the end of step 3 (by which point no BigQuery client is created yet), but then I get 
_pickle.PicklingError: Can't pickle <function root at 0x11e52ccc0>: attribute lookup root on __main__ failed
Currently step 3 looks like
Copy code
@step
    def instantiate_model(self):
        '''
        Initializes the LTV Model with the flow parameters
        '''
        print("Instantiating LTV Model...")

        self.ltv_model = LTVModel(country=self.country)

        print("Model instantiated. providing Training params")


        train_params = {
            'min_sample_id': self.train_min_sample_id,
            'max_sample_id': self.train_max_sample_id,
            'first_cohort_date': f'"{self.train_first_cohort_date}"',
            'last_cohort_date': f'"{self.train_last_cohort_date}"',
            'last_observation_date': f'"{self.train_last_observation_date}"'
        }
        self.ltv_model.train_params = train_params

        print("Now the test params")


        test_params = {
            'min_sample_id': self.test_min_sample_id,
            'max_sample_id': self.test_max_sample_id,
            'first_cohort_date': f'"{self.test_first_cohort_date}"',
            'last_cohort_date': f'"{self.test_last_cohort_date}"',
            'last_observation_date': f'"{self.test_last_observation_date}"'
        }
        self.ltv_model.test_params = test_params

        print("are we getting here?")

        self.next(self.train_model)
and we are, in fact, getting to "are we getting here?" 😂
s
the code flow ending up at "are we getting here" - actually makes sense.. because metaflow will try to store / serialize the attributes you have assigned to self - only at the end of the step .. In case you try to assign some object to self which is not 
pickleable
then you will come to know only towards the end of the step
c
Ahhhh, gotcha 🙂. That's helpful context, thank you!
s
this has a high level arch which might be helpful
at the risk of sounding foolish - the LTV Model you are instantiating - can you make that pickleable?
c
So...that's the interesting part. That is pickleable. We do it in jupyter notebooks sometimes.
Upon doing more looking into pickling errors, it seems that this error might be a result of pickle attempting to find the model on the 
__main__
path in spite of the fact that it is an an attribute on the flow class.
s
oh interesting
c
Specifically that 41 upvotes answer explains it
s
ha.. I am out of my depth at this point 😄
c
But that begets the question....if that's the case, why does pickling ever work as a means of storing a thing set as an instance attribute on another class... 😅
d
hey @cold-carpenter-83628 — not 100% sure but it probably indicates that one of your objects has a function which are typically not picklable.
👀 1
I am not 100% sure what the bigquery client contains but it may also be a bad idea to try to save it across steps as it may have internal state that isn’t transferable that way.
Another option may be to instead pickle the attributes used to create the client so you can recreate it across steps if you want.
c
@dry-beach-38304 ah. The model does, indeed, have functions!
Now, that said, we do pickle it successfully in other venues outside a metaflow flow.
d
we don’t do anything special for pickling (we just call pickle) but it’s hard for me to pinpoint a more specific problem (it’s really dependent on the object).
c
@dry-beach-38304 indeed, I think this has something to do with attempting to pickle an object that is an attribute of another object. Something like this from the SO answer linked above could also be true for functions:
Copy code
The problem is that when namedtuple() returns a type object, it isn't aware of the fact that it's being assigned to a class member - and thus, it tells the type object that its type name should be __main__.Bar, even though it should really be __main__.Foo.Bar.
d
you can always fake things too if you want (like everything in python). Again, not sure of fthis specific use case but see in this PR how we muck around with things to make them picklable: https://github.com/Netflix/metaflow/pull/1664/files#diff-2e2fbb1b0cdd94c0e4915dd8e2395262974ba0a6429707ef20a4ae93fc78ff44. That said, you are better off saving as artifacts objects that are nice and picklable
c
Indeed; Im in a jupyter notebook now attempting to replicate the issue with pickling the ltv model. I suspect you're right that just generating the bigquery client whenever i need it is preferable over saving one. They don't take long to generate at all.
UPDATE! Got this working. Instead of pickling the model, I'm pickling a table of probabilities from which I can rehydrate the model in a follow-on step.
yay 1
8 Views
Open in Slack
PreviousNext
Powered by