hello, I'm trying to deploy Metaflow to Azure for the first time. I'm following these instructions, which point to these permissions to be granted for the deployment. After granting all of the permissions and running the

terraform apply ...

command, I get a lot of errors like the following:

│ Cannot register provider Microsoft.KeyVault with Azure Resource Manager: resources.ProvidersClient#Register: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client '<redacted>' with object id '<redacted>' does not have authorization to perform action 'Microsoft.KeyVault/register/action' over scope '<redacted>' or the scope is invalid. If access was recently granted, please refresh your credentials.".

I get this error for a lot of

Microsoft.*/register/action

actions, and I'm not quite sure how to go about fixing this. Would anyone be able to help me figure out what permissions I'm missing for this specifically? (I'm new to Azure here so this might be pretty basic)

Hi Ruben, sorry for the delay here

yep! let me re-run it to produce all of the output

do you set any env vars locally for azure sdk's to get a verbose output?

i haven't set anything like that! just the normal credential setup... although I think I put the

logging.basicConfig

level to info a long time ago. i'm not sure why it's getting 404s and not logging the errors leading to the run failure though...

also, it seems that the error is with launching the kubernetes job. do you have a valid kubeconfig locally that allows you to submit jobs to kubernetes?

this one i might not know how to check 🙂

have you been able to get any flow running, say without

--with kubernetes

?

only locally, it's my first time trying out the cloud deployment

what is the output that you get when you do

python flow.py run

?

it succeeds but i don't see any logs in the UI

sure - i will try to be around, but my responses might be delayed. what i would recommend is getting a simple flow working (such as one here) without

kubernetes

first.

ah okay! sounds good, thank you! I appreciate it!

it is likely that it is an issue with how the deployment is wired. were the logs visible without the pypi_base decorator - say when running a very simple flow with a bunch of print statements?

hm... interesting! okay, I have a really simple flow with this result:

(metaflow) ruben@ruben-A6:~/work/test-metaflow/model_flows/virtual-patient$ METAFLOW_PROFILE=azure python test.py run --with kubernetes
Metaflow 2.10.8+netflix-ext(1.1.1) executing TestFlow for user:ruben
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
2024-07-03 00:35:08.550 Workflow starting (run-id 12):
2024-07-03 00:35:10.397 [12/start/24 (pid 69362)] Task is starting.
2024-07-03 00:35:12.953 [12/start/24 (pid 69362)] [pod t-47ed747d-pkdsf-pp4cg] Task is starting (Pod is running, Container is running)...
2024-07-03 00:36:25.506 [12/start/24 (pid 69362)] Kubernetes error:
2024-07-03 00:36:25.592 [12/start/24 (pid 69362)] Error (exit code 1). This could be a transient error. Use @retry to retry.
2024-07-03 00:36:25.593 [12/start/24 (pid 69362)] 
2024-07-03 00:36:25.707 [12/start/24 (pid 69362)] Task failed.
2024-07-03 00:36:25.824 Workflow failed.
2024-07-03 00:36:25.824 Terminating 0 active tasks...
2024-07-03 00:36:25.824 Flushing logs...
    Step failure:
    Step start (task-id 24) failed.

I can run it locally:

(metaflow) ruben@ruben-A6:~/work/test-metaflow/model_flows/virtual-patient$ python test.py run
Metaflow 2.10.8+netflix-ext(1.1.1) executing TestFlow for user:ruben
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
2024-07-03 00:38:23.560 Workflow starting (run-id 1719959903550385):
2024-07-03 00:38:23.563 [1719959903550385/start/1 (pid 69802)] Task is starting.
2024-07-03 00:38:23.768 [1719959903550385/start/1 (pid 69802)] Task finished successfully.
2024-07-03 00:38:23.771 [1719959903550385/main/2 (pid 69805)] Task is starting.
2024-07-03 00:38:23.938 [1719959903550385/main/2 (pid 69805)] This is the main function
2024-07-03 00:38:23.973 [1719959903550385/main/2 (pid 69805)] This is the main function
2024-07-03 00:38:23.973 [1719959903550385/main/2 (pid 69805)] This is the main function
2024-07-03 00:38:23.973 [1719959903550385/main/2 (pid 69805)] This is the main function
2024-07-03 00:38:23.973 [1719959903550385/main/2 (pid 69805)] This is the main function
2024-07-03 00:38:23.973 [1719959903550385/main/2 (pid 69805)] This is the main function
2024-07-03 00:38:23.973 [1719959903550385/main/2 (pid 69805)] This is the main function
2024-07-03 00:38:23.974 [1719959903550385/main/2 (pid 69805)] This is the main function
2024-07-03 00:38:23.974 [1719959903550385/main/2 (pid 69805)] This is the main function
2024-07-03 00:38:23.974 [1719959903550385/main/2 (pid 69805)] This is the main function
2024-07-03 00:38:23.974 [1719959903550385/main/2 (pid 69805)] Task finished successfully.
2024-07-03 00:38:23.978 [1719959903550385/end/3 (pid 69808)] Task is starting.
2024-07-03 00:38:24.178 [1719959903550385/end/3 (pid 69808)] Task finished successfully.
2024-07-03 00:38:24.179 Done!

with this code:

class TestFlow(FlowSpec):
    @step
    def start(self) -> None:
        self.next(self.main)

    @step
    def main(self) -> None:
        for _ in range(10):
            print("This is the main function")

        self.next(self.end)

    @step
    def end(self) -> None:
        pass

if you dump this into a

job.yaml

-

apiVersion: batch/v1
kind: Job
metadata:
  name: test-job
spec:
  template:
    spec:
      containers:
      - name: test
        image: busybox
        command: ["echo", "Hello, Kubernetes!"]
      restartPolicy: Never

and then execute

kubectl apply -f job.yaml

followed by

kubectl get jobs

- what is the output?

(metaflow) ruben@ruben-A6:~/work$ kubectl apply -f job.yaml job.batch/test-job created
(metaflow) ruben@ruben-A6:~/work$ kubectl get jobs 
NAME               COMPLETIONS   DURATION   AGE
t-14365adb-shg28   0/1           11h        11h
t-47ed747d-pkdsf   0/1           7h15m      7h15m
t-51796fab-pwjsm   0/1           7h11m      7h11m
t-b19d1a41-g47z8   0/1           7h21m      7h21m
t-bc2fb6c3-shdcz   0/1           10h        10h
t-ca8d42f2-26p8d   0/1           10h        10h
t-e6478428-4rnq8   0/1           10h        10h
t-fa2a45d6-5qjbb   0/1           7h18m      7h18m
t-ff0eb83d-7wbpb   0/1           10h        10h
test-job           1/1           5s         5s

(i changed my terminal size after running so i think there are some collapsed lines)

Yeah it seems that something is off with your setup - metaflow is able to submit jobs to Kubernetes but then those jobs don’t make much progress

hmm... any suggestions on how to debug that setup issue?

Can you check the pod logs for any of the jobs that start with t-?

yeah, I was just checking that! I can find the jobs here:

The pod might have expired. Can you try executing the flow once again and checking the pod logs and events for the newest job?

ah okay! maybe this is the underlying error?

The container seems to have started

it seems like i might need to enable this service to check the logs?

You can also use kubectl from your terminal to fetch the pod logs

ah let me try that