A model can be passed through different

steps

?

@step

def train(self):

self.model.train(X,y)

self.next(predict)

@step

def predict(self):

self.model.predict #model previously trained

self.next(end)

It depends if the model can be pickled. Scikit-learn Estimators can be used like this.

Yes these LGBM and Catboost models So:

self.model.predict #model previously trained

so this line is going to read a model trained in the previous step

Sometimes GBM models are tricky with serialization. If you hit issues, here is a pattern for manual version of how

self.

does upload / download. Please lmk if you can share examples with LGBM & Catboost 🙂

I will once i have something. Thanks!!

Ahh, ya in that example. Are you working with different data storage layer? The point is to serialize the model as bytes and put it somewhere that both tasks in the Metaflow DAG can read/write from.

IIRC both lightgbm/catboost model artifacts can be pickled, so should be good to let

self.model

do its thing you'll only need to go beyond that if you need to export the model artifact into a more interoperable format (e.g. exporting to ONNX or dumping the weights to JSON to run as part of LTR plugin in elasticsearch), or if you want to utilize something like intermediate model checkpointing during training (e.g. create a model artifact every n iterations, which can be combined with

@retry

but requires some DIY serialization)

Yes you are right @fresh-laptop-72652 both lightgbm/catboost can be pickled. Without using Metaflow I’d use it’s respective methods

save

and

load

but if I’m not mistaken this is “saving” a

json

or a

pkl

file somewhere and then load it back again for predictions. What I thought

self.

would do is saving the model trained from

@step train

to

@step predict

without having to serialize an object and dumping it in

local system

or

S3

Maybe you’d be kind to explain me re:“let

self.model

do its thing”

ah gotcha! yea it sounds like you've got the right idea for it think of each

@step

as potentially executing on totally different computers – the only way to achieve that is by serializing at the end of the train step and then deserializing those artifacts in the subsequent predict step. in the broader realm of DAGs / workflow orchestrators, this is often called "state transition" since it has to do with how objects get passed around and made available. with Metaflow, the developer experience for state transition is handled through the use of

self.

-- there's some good examples in the docs to cover it https://docs.metaflow.org/metaflow/basics#artifacts the gist is that at the end of each

@step

, all the objects you've attached as attributes onto

self

get pickled/compressed/hashed and saved into storage (e.g. local storage on your laptop, or more likely cloud-based blob storage like S3) those saved artifacts can then be lazy-loaded into subsequent steps as needed, and the python objects are reinstantiated from those pickle files. at the end of the day, there's tons of ways you can save a model, serialize it's weights, or whichever term you prefer – and likewise there's many different formats depending on the model (e.g. pkl, onnx, json, pt, etc). There's pros and cons to different formats, but when it comes to saving nearly any arbitrary python object and loading it elsewhere in python, it's hard to beat pickle -- which is why Metaflow uses it for outta-the-box state transition. You can of course still use the other serialization methods, but it's just not baked into the DX of Metaflow hope that helps!