Hey. I’d like to try the k8s integration. Sending ...
# dev-metaflow
s
Hey. I’d like to try the k8s integration. Sending a message here per the request in the blog post.
1
👀 1
s
hey Josh! we are happy to help
for starters, do you use AWS?
s
Yes, we use AWS
And EKS if that matters, although not exclusively
s
nice, that's supported well
a
great! here's the draft of the documentation for it https://docs.google.com/document/d/1L_4Fws1KoGg_dtSTaRlAcREX1F8FPS4ZaYk7eJyu_jA/edit#heading=h.89vcutz9qrc3 tl;dr is as long as you have a valid K8S config, it should just work. Instead of
python myflow.py run
you can do
python myflow.py run --with kubernetes
and it'll run the steps on K8S cluster from kubeconfig
(and you don't have to use
--with
, you can alternatively annotate steps with
@kubernetes
decorator)
s
Thank you, will give this a try and report back. Any rough edges or gotchas I should know about now?
a
You shouldn't run into any obvious issues with basic K8S/EKS. What's missing is mostly things like adding more options so users can configure GPU node pools to their liking, or more advanced RBAC setups