https://outerbounds.com/ logo
Join Slack
Powered by
:wave::skin-tone-2: There is a way in AWS SageMake...
# ask-metaflow
m
👋🏻 There is a way in AWS SageMaker to integrate Tensorboard to visualize experiments like https://docs.aws.amazon.com/sagemaker/latest/dg/studio-tensorboard.html. Wonder if anyone has attempted to do something similar with the Metaflow UI? Thinking that there might be a way to use the JupyterServer /proxy/ URL to redirect.
✅ 1
v
including a link to a Tensorboard (e.g. tensorboard.dev) run should be very easy in a 
@card
you should be able to include it in an iframe in a custom card - it would be a fun experiment 🙂
m
thanks @straight-shampoo-11124
let me try a few things
actually to clarify, I don't mean to display tensorboard inside of the metaflow UI. I meant that instead of tensorboard, we want metaflow UI to be accessible (maybe as a sidecar?) from a JupyterLab environment
v
ah, understood. Given how extensible JupyterLab is, I am quite positive it's possible
something like jupyterlab_iframe maybe. I don't have a recipe for you right now (maybe others here have tried it?) but definitely let us know how it goes if you end up setting it up
👀 1
m
Thanks @straight-shampoo-11124 I tried it out, but jupyterlab_iframe is not very customizable and gives some issues displaying content (I suspect CORS related) I'm trying out https://jupyter-server-proxy.readthedocs.io/en/latest/arbitrary-ports-hosts.html with some progress • Using https://d-xxx.studio.us-east-1.sagemaker.aws/jupyter/default/proxy/metaflow.test-domain.autodesk.com:80 • /aws/sagemaker/studio AWS logs show 200 GET in proxying • /metaflow-ui AWS logs also show 200 GET • however, the page is blank and the browser console shows 404 not found errors for the static assets (chunked js, css files) Any ideas? I suspect it's either still CORS related or some other security headers/policies I may need to edit inside of the metaflow-ui service
v
if you copy-paste the proxied URL in the browser, are you able to see it?
m
Yup, it works perfectly well. This is a pattern for us to provide the UI from inside the sagemaker JupyterLab environment
v
m
Oh gotcha, so: • https://metaflow.test-domain.autodesk.com works fine • https://d-xxx.studio.us-east-1.sagemaker.aws/jupyter/default/proxy/metaflow.test-domain.autodesk.com:80 is blank with 404 errors in the browser console for the CSS/JS files ◦ note that port 443 doesn't work (yet) for the jupyter-server-proxy ◦ and we added a port 80 listener on the ALB to make networking progress ◦ however, logs on both the metaflow-ui and jupyter service show 
200
GET requests
v
yeah, going through the jupyter proxy might be hard - i bet they don't support websockets either
if the Metaflow UI was directly accessible to the client (e.g. through VPN), then it would be easier to get the iframe working too
do you have Metaflow UI accessible at https://metaflow.test-domain.autodesk.com ?
m
Websocket might be supported https://github.com/jupyterhub/jupyter-server-proxy/blob/main/jupyter_server_proxy/websocket.py Iframe from within a notebook using IPython works, but that is via the client browser connection. We're looking to shrink the allowlist to be from the AWS notebook network only. and jupyter_iframe would be good (it uses the remote host to proxy, using tornado). jupyter-server-proxy also uses tornado Yup, that URL is a dummy but it works for me/users via VPN allowlist.
v
I don't have experience with the Jupyter proxy. I'd load the proxy URL with devtools open in my browser and see where it fails
👍 1
m
all good, thanks anyway for suggestions
FWIW - that is what I did for both jupyter_iframe and jupyter-server-proxy. Both use tornado under the hood to proxy. Both in the browser devtools/console show 
404
errors upon loading css/js assets, but networking wise the requests succeed with 
200
v
hmm, I wonder if it gets confused with paths somehow going through the proxy
m
possible! looking into that
👍 1
v
the proxy approach may be tricky. If you are able to expose another port on your Sagemaker instances and access the UI through a proxy/tunnel behind that port, it'd have a higher chance of succeeding
👌 1
👀 1
you could have just 
nginx
behind that port with a remote backend pointing at your internal Metaflow service. This way the client will only have to talk to the sagemaker instance
there wouldn't be any auth on that path though, which may or may not be a problem
m
so exposing another port on the same jupyter server with nginx works (nicer url this way too), but hits a similar problem as before. It seem like it is definitely due to what you said about 
confused with paths somehow going through the proxy
• The main index.html loads fine in both direct and sagemaker proxy • However, for static assets like favicon.ico, manifest.json, and css/js files ◦ https://metaflow.test-domain.autodesk.com/ works fine ◦ https://d-xxx.studio.us-east-1.sagemaker.aws/jupyter/default/metaflow/ does not ▪︎ the browser will try to load https://d-xxx.studio.us-east-1.sagemaker.aws/manifest.json ▪︎ but it should be loading from https://d-xxx.studio.us-east-1.sagemaker.aws/*jupyter/default/metaflow/*manifest.json
so perhaps in our vendored metaflow-ui service, the code for https://github.com/Netflix/metaflow-ui/blob/master/public/index.html#L5 can be changed from absolute to relative paths 🤔
v
interesting. We can look into this early next week and see if there's an easy fix
✅ 1
if you have time, you can try changing that path by yourself meanwhile
m
thanks and no rush, but making the app support base path like https://aimstack.readthedocs.io/en/latest/using/sagemaker_notebook_ui.html would open up more integrations for Metaflow UI. in the meantime I can try to play with path rewrites using nginx and/or the jupyter side. So the PUBLIC_URL variable or publicPath if using webpack
FYI - the nginx config that worked
Copy code
server {

    listen 3000;

    location /jupyter/default/metaflow/ {
        resolver 8.8.8.8;
        proxy_pass <https://metaflow.test-domain.autodesk.com/>;
        proxy_redirect / /jupyter/default/metaflow/;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
        proxy_set_header Accept-Encoding "";

        rewrite ^/jupyter/default/metaflow/(.+) /$1 break;

        sub_filter_once off;
        sub_filter_types *;
        sub_filter "='/" "='/jupyter/default/metaflow/";
        sub_filter '="/' '="/jupyter/default/metaflow/';
        sub_filter '/static/' '/jupyter/default/metaflow/static/';
        sub_filter '.aws/' '.aws/jupyter/default/metaflow/';
    }

}
s
awesome! I’ll get back to you re: paths soon
m
great thanks. So that works for the most part - and I'm now debugging : • refreshing - the sub_filters don't seem to be replacing the links created in the JS SPA. So clicking around always works, but the browser URL is missing the base path ``jupyter/default/metaflow/` • index.html is not autofilling upon initial page load to a first query like 
?timerange_start=1679702400000
, so user has to click the top left home button first
b
@most-leather-78733 would it be helpful to have an env var with your base path that can be prepended to link URL's and API URL's?
m
that would be amazing
so something to help us with 
{scheme}://{host}:{port}/{OPTIONAL_BASE_PATH_FOR_PROXY}/{PATH}?{QUERY}
I think this would open up more integration options for Metaflow and I've seen this done on other apps
b
Would you be able to work-with/test from a branch of 
metaflow-ui
?
m
yea CC @acoustic-van-30942
b
thankyou 1
m
thanks @brave-lion-15961 will do.
Copy code
This change does not affect API URL's nor websocket URL's. They should be able to be controlled through REACT_APP_METAFLOW_SERVICE
in my case do you suggest: • for metaflow ui ◦ 
export REACT_APP_BASE_PATH="/jupyter/default/metaflow"
◦ 
export REACT_APP_METAFLOW_SERVICE="/jupyter/default/metaflow/api"
b
I think 
REACT_APP_METAFLOW_SERVICE
should be the full URL, including the domain name
a
@most-leather-78733 following up here - did this work for you? If not, we are happy to explore alternates; otherwise, we would love to ship this PR soon.
m
Hi @ancient-application-36103 I was out at a conference all week, but this is is in our upcoming sprint. Will let you know.
a
Fantastic
m
Just a heads up that it's getting pushed back a little bit, but we still plan to test it in our environments.
👍🏽 1
👍🏼 1
we merged our fork of your PR branch. will update
🎉 2
So first try didn't seem to work. We built the ui container using this change and redeployed it. Ziqi who worked on it will join this slack 🙂 CC @acoustic-van-30942 Question: is it reasonable to expect that if we DO set 
REACT_APP_BASE_PATH=/jupyter/default/metaflow
but the path doesn't have 
/jupyter/default/metaflow
, it should break?
b
It should redirect to a URL that does start with 
/jupyter/default/metaflow
e.g. 
/My0?direction=asc&group=true&order=startTime&status
will become 
/jupyter/default/metaflow/MyFlow/20?direction=asc&group=true&order=startTime&status
In my local case, I set 
export REACT_APP_BASE_PATH=/giraffe
Screen Recording 2023-05-25 at 2.37.00 PM.mov
It shouldn't break
m
@fast-dog-88545 Can you describe your findings?
f
@brave-lion-15961 Thanks for the video, although we do set 
REACT_APP_BASE_PATH=/jupyter/default/metaflow
, the path without 
/jupyter/default/metaflow
will not be automatically redirected to one start with 
/jupyter/default/metaflow
. If I manually add 
/jupyter/default/metaflow
, it will think we are accessing the data of a Flow named 
/jupyter/default/metaflow
.
b
I will see if I can re-create this locally
We may need to walk through this together - would you be open to a call?
f
Thanks you so much! Please let me know if there anything I can do for help
Sure
b
In this video, I ran the UI with 
export REACT_APP_BASE_PATH=/jupyter/default/metaflow
. When I navigate to 
<http://localhost:3000/>
it automatically changes to 
<http://localhost:3000/jupyter/default/metaflow/?_group_limit=30&_limit=30&_order=-t[…]us=failed%2Crunning%2Ccompleted&timerange_start=1682935200000>
Is this different to what you are doing?
Screen Recording 2023-05-31 at 3.01.46 PM.mov
A possible, add complication is that MFGUI remembers some filter parameters in localStorage. Maybe that is affecting you?
f
Yes it's different. As Tony said, https://metaflow.test-domain.autodesk.com works for us, so after set 
REACT_APP_BASE_PATH=/jupyter/default/metaflow
, should we expect it automatically changes to https://metaflow.test-domain.autodesk.com/jupyter/metaflow/default? (but it doesn't)
Let me look into the MFGUI. I don't think so
b
Are you able to edit the code? If you add 
console.log('process.env.REACT_APP_BASE_PATH: ', process.env.REACT_APP_BASE_PATH);
into 
App.tsx
you can check to see if the variable is getting through properly.
f
Good idea, let me try and check
b
Any more information?
m
seems to have worked for us. there was an issue getting the env-var into the docker runtime using AWS ECS qq: You didn't have to edit your metaflow-ui/Dockerfile with new 
ENV
lines, right?
b
Good to hear. I run the UI locally, so I just 
export
the env var, myself - so I don't edit the Dockerfile
f
@brave-lion-15961 can I know how you run it locally, is it by 
docker build
and then run the built image? If it's in this way, will 
docker run -e REACT_APP_BASE_PATH=/jupyter/default/metaflow
works instead of using 
export
(This is how AWS ECS works)? I am trying to verify this but have problems in running metaflow UI locally, maybe because my computer is using M2 chip.
b
Running locally is not the recommended way for everyone - I do it because I am actively modifying the code, so it is quicker to develop. I run the 
metaflow_service
using docker than then just run the UI portion with 
yarn start
. There are some extra credentials that I add to my dockerfile to allow the UI to connect properly. My laptop has an M1 chip.
f
Oh yes I am following this document which may just run the UI portion with 
yarn start
and keeps encountering problems. And also get error 
io_setup() failed
when I try to directly run docker image, this seems related to M1 chip by searching in google Anyway this modification works well. Thanks for your help!
b
We should probably dive into your M2 problems in another thread as they are sure to affect my folks in the near future.
If you're happy with the modification, I'll add some documentation and merge it in to the next release - thanks for testing!
thankyou 1
f
That's great, I'd love to report what problems I had when trying to deploy metaflow UI to my M2 computer
b
🚨 Released https://github.com/Netflix/metaflow-ui/releases/tag/v1.3.3 to allow overriding the UI's base path. Thanks for your help
m
thank you for this collab
Open in Slack
PreviousNext
Powered by