https://outerbounds.com/ logo
Join Slack
Powered by
Hi all! I am working on remotely (from a cloud no...
# ask-metaflow
h
Hi all! I am working on remotely (from a cloud notebook) get access to our metaflow api which is hosted behind a Cloudflare Access protection. To gain access to the api, I need to pass headers to help bypass the protection, but no matter what, the headers don't seem to make their way with the call. this is how I am defining the 
METAFLOW_SERVICE_HEADERS
in the config file
Copy code
"METAFLOW_SERVICE_HEADERS": {
        "origin": "<https://xyz.com>",
        "referer": "<https://xyz.com>",
        "CF-Access-Client-Id": "my-client-id",
        "CF-Access-Client-Secret": "my-client-secret"
    }
any idea on what I am missing? TIA
a
i believe you may need to encode the value into a string
h
trying now
Copy code
"METAFLOW_SERVICE_HEADERS": "{\"origin\":\"<https://xyz.com>\",\"referer\":\"<https://xyz.com>\",\"CF-Access-Client-Id\":\"my-client-id\",\"CF-Access-Client-Secret\": \"my-client-secret\"}"
didn't work either... is that how you'd proceed @square-wire-39606?
s
can you help me with the error trace?
h
there is no error trace, I am just getting a 403 which outputs the cloudflare protection page, meaning the header is not being passed 😞
s
can you trace using wireshark to check which headers are being passed?
h
I am not sure how else I could test, especially since I am running from a cloud hosted environment
s
ah
okay
let me also trace the code to see if we have any recent regressions
which version of metaflow are you on?
h
metaflow = "2.12.10"
s
can you try with a newer version - this one is quite old 🙂
h
sure, what's the newest? I'm going to have to rebuild my image and it'll take a while, but afaik, this is how to pass headers since a few years ago no?
Is there a way to print the configuration from within the metaflow cli in python?
a
yep - that's the mechanism. newest is 2.13.5
🙌 1
@thankful-ambulance-42457 do you know if anything else needs to happen to pass the headers?
thankyou 1
h
what about validating the configurations from within code @square-wire-39606, so you have a trick for that?
t
easiest way to check whether the config is read correctly is via:
Copy code
python -c "from metaflow.metaflow_config import SERVICE_HEADERS; print(SERVICE_HEADERS)"
(avoid this if the sensitive headers shouldn't be printed though) the value of 
METAFLOW_SERVICE_HEADERS
should be a json string, but the example you gave should work just fine.
🙌 1
h
thank you. I have been struggling a lot with this issue today.
To confirm, the URL that the python code when importing metaflow hits is the metadata service, or the backend-ui service? I am able to see results from my https://backend-ui-service/api/runs etc... but I cannot figure out how to validate that same result from the metadata service endpoint https://metadata-service.ourdomain/ping is the only endpoint that I found that answers, can one of you give me a few examples of how I could test thatout?
I was able to make this work! I can hit the api from code by passing the proper headers. Now I am at the point that I would like to configure the UI service to add the SERVICE_HEADERS so by default the UI can hit the backend-ui-service which is protected behind Cloudflare Access. I tried adding them as an environment variable to the metaflow UI service container, but it doesn't seem to be taken into consideration by the UI service
Bump, anyone have an idea on how to configure headers to be passed every time the UI calls the backend-ui api?
a
@thankful-ambulance-42457 any thoughts?
t
hmm. will have to look at this more thoroughly. @hundreds-midnight-75494 are you hosting the static ui assets via the docker image that has nginx inside for routing? or are you letting the ui backend also host the static frontend assets?
h
Copy code
- name: METAFLOW-UI-IMAGE
    newName: public.ecr.aws/outerbounds/metaflow_ui
    newTag: 1.3.13
this is the image we're using for the UI, the backend and metadata services are separate
t
hmm. yep I think that feature is missing, and the dockerfile needs to be modified in order for the requests to pass the correct headers
h
Copy code
- name: METAFLOW_SERVICE_HEADERS
              valueFrom:
                secretKeyRef:
                  name: metaflow-cf-service-token
                  key: data
this is how I am passing this to the container from k8s
assuming that'll also be how the container would access the headers?
t
yup, but I'm guessing whats happening here is the frontend which is funneling api calls through itself (nginx) to the actual backend, is not passing all headers forward. theres probably a simple toggle in nginx.conf that can enable this though
out and about now on my phone so will check this once I'm at my workstation again :)
h
yeah that makes sense
awesome, thank you for the response, looking forward to enabling this, it'll actually help us remove convoluted WAF policies we temporarily put in place to protect our backend api 🙂
4 Views
Open in Slack
PreviousNext
Powered by