Question about using custom

image

in

@kubernetes

decorator together

@pypi

decorator Hey everyone! We have a situation where most of our steps share a lot of packages but still require custom installations every now and then. So we have a base Docker image that is built with all of the common dependencies, but we would like to use the

@pypi

decorator to install the custom deps on the fly. Is this currently possible? I did a quick and dirty example flow with a custom base image and a custom dependency installed in the

@pypi

decorator and the code inside the step was not able to import PyTorch, even though it is available in the custom image.

@kubernetes(tolerations=[{"key": "something", "operator": "Equal", "value": "another_value", "effect": "NoSchedule"}], gpu=1, image="pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime")
    @pypi(python="3.10.0",
        packages={
            "implicit": "0.7.2",
        },
    )
    @step
    def gpu(self):

I searched in the docs and was able to find some information, but I am not sure if

system-wide

packages in the snippet refers to the container images packages or something else. Can anyone shed some light on whether or not the setup I am describing above is achievable with Metaflow? Thank you!

That’s not possible with the oss version of @pypi but with fast bakery https://outerbounds.com/blog/containerize-with-fast-bakery