Hi everyone Does anyone know how to pass a Parameter value t Outerbounds #ask-metaflow

Hi everyone, Does anyone know how to pass a Param...

jolly-dawn-58625

01/29/2025, 5:04 PM

Hi everyone, Does anyone know how to pass a Parameter value to a decorator? e.g.

Copy code

...
node_selector_params = Parameter(
    "node_selector_params",
    help="Node selector dictionary",
    default={"key": "value"},
    type=JSONType,
)
...
...
@kubernetes(
    service_account="...",
    image="...",
    image_pull_policy="Always",
    cpu=1,
    gpu=1,
    node_selector=<DYNAMIC_DICT_HERE>
    memory=4096,
    disk=10240,
)
...
...

✅ 1

square-wire-39606

01/29/2025, 5:05 PM

Hi! I think this might be what you are looking for https://docs.metaflow.org/metaflow/configuring-flows/basic-configuration

🙌 1

jolly-dawn-58625

01/29/2025, 5:06 PM

I could not get my head around that doc.

jolly-dawn-58625

01/29/2025, 5:06 PM

I could use a config, but then I won't be able to use a trigger and set it?

square-wire-39606

01/29/2025, 5:08 PM

Correct - these are deploy time variables

jolly-dawn-58625

01/29/2025, 5:09 PM

One way of doing this is getting the val from the parameter and setting

METAFLOW_KUBERNETES_NODE_SELECTOR

for the next step, right?

jolly-dawn-58625

01/29/2025, 5:12 PM

I just don't want to build the image each time I want to experiment with a new node 🙂

square-wire-39606

01/29/2025, 5:15 PM

Curious - what’s the use case? I would be happy to explore alternatives

jolly-dawn-58625

01/29/2025, 5:17 PM

My goal is to pick the best (performance/cost) gpu instances for the job, and compare them.

jolly-dawn-58625

01/29/2025, 5:18 PM

Using karpenter + node_selector to get the exact gpu instance I need provisioned.

jolly-dawn-58625

01/29/2025, 5:21 PM

AI/ML images can get huge in size, and building them takes time. So for each "experiment" there is a lag and interruption in the development process. Metaflow is great, and thank you! I may be asking for too much flexibility here 🙂

ancient-application-36103

01/29/2025, 6:17 PM

re: baking images ultra fast, we do have a solution in outerbounds 🙂

ancient-application-36103

01/29/2025, 6:18 PM

also, re: configs - here is another pattern that can run multiple runs with different configs that you can potentially embed inside a deployed flow

jolly-dawn-58625

01/29/2025, 6:20 PM

Thank you so much, I'll look into these.

5 Views

Open in Slack

Previous Next