https://outerbounds.com/ logo
Join Slack
Powered by
Hi all! We have Metaflow deployed in GCS. I want t...
# ask-metaflow
d
Hi all! We have Metaflow deployed in GCS. I want to create e2e tests of flows, which mimics exactly how flows run in practice in k8s. In practice, each flow is deployed and multiple containers are running for each flow which complicates a bit the way to have e2e testing. Anyone with experience with creating tests for flows?
h
Which part are you struggling with? We use deployer/runner to orchestrate our e2e tests
d
Thanks! The struggle with running e2e in GitHub Actions is that you can't run "locally" since each step has its own container image/deps. Using Runner makes sense! Are you using Runner with 
decospecs=['kubernetes']
to trigger flows on K8s from CI and validate artifacts? Does CI just need metaflow + cluster creds? If you can share more about the setup and how you compare artifacts, it'd be helpful. I essentially want to compare multiple intermediate outputs of the flow.
h
we use our own testing infra internally at Netflix so our parameters are different (we actually setup default decospecs in our internal extension), but what you described should work
👍 1
as to how our tests are setup, we wrap everything in pytest and programmatically fetch the artifacts to do assertions
🙌 1
thankyou 1
here's an example
Copy code
def test_hello_world(
    exec_mode, decospecs, titus_image, titus_gpu_image, tag, scheduler_config
):
    run = execute_test_flow(
        flow_name="flows/basic/helloworld.py",
        exec_mode=exec_mode,
        decospecs=decospecs,
        titus_image=titus_image,
        titus_gpu_image=titus_gpu_image,
        tag=tag,
        scheduler_config=scheduler_config,
        test_name="hello_world",
    )

    assert run.successful, "Run was not successful"
    assert (
        run["hello"].task.data.message == "Metaflow says: Hi!"
    ), "Hello world strings didn't match"
👀 1
5 Views
Open in Slack
PreviousNext
Powered by