https://outerbounds.com/ logo
Join Slack
Powered by
Hello MF team! <@U03CUHZ5JSW> <@U01TPLB5GA2> I am...
# ask-metaflow
s
Hello MF team! @User @narrow-lion-2703 I am setting up my own metaflow deployment from the Helm chart and I notice that my UI refuses to work no matter what I do. The metaflow deployment is over a preexisting GKE deployment in VPC native mode and it targets the 
default
network. The UI says there is no connection and keeps outputting the following 500 error. I was sort of hoping for some pointers. Is this a CORS error?
Copy code
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="/favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="Build and manage real-life data science projects with ease."/><link rel="apple-touch-icon" href="/logo192.png"/><link rel="manifest" href="/manifest.json"/><title>Metaflow UI</title><link href="/static/css/main.95ba8333.chunk.css" rel="stylesheet"><script>window.METAFLOW_SERVICE="/api";window.MF_DEFAULT_TIME_FILTER_DAYS="";</script></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div><script>!function(e){function t(t){for(var n,l,f=t[0],i=t[1],a=t[2],c=0,s=[];c<f.length;c++)l=f[c],Object.prototype.hasOwnProperty.call(o,l)&&o[l]&&s.push(o[l][0]),o[l]=0;for(n in i)Object.prototype.hasOwnProperty.call(i,n)&&(e[n]=i[n]);for(p&&p(t);s.length;)s.shift()();return u.push.apply(u,a||[]),r()}function r(){for(var e,t=0;t<u.length;t++){for(var r=u[t],n=!0,f=1;f<r.length;f++){var i=r[f];0!==o[i]&&(n=!1)}n&&(u.splice(t--,1),e=l(l.s=r[0]))}return e}var n={},o={1:0},u=[];function l(t){if(n[t])return n[t].exports;var r=n[t]={i:t,l:!1,exports:{}};return e[t].call(r.exports,r,r.exports,l),r.l=!0,r.exports}l.m=e,l.c=n,l.d=function(e,t,r){l.o(e,t)||Object.defineProperty(e,t,{enumerable:!0,get:r})},l.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},l.t=function(e,t){if(1&t&&(e=l(e)),8&t)return e;if(4&t&&"object"==typeof e&&e&&e.__esModule)return e;var r=Object.create(null);if(l.r(r),Object.defineProperty(r,"default",{enumerable:!0,value:e}),2&t&&"string"!=typeof e)for(var n in e)l.d(r,n,function(t){return e[t]}.bind(null,n));return r},l.n=function(e){var t=e&&e.__esModule?function(){return e.default}:function(){return e};return l.d(t,"a",t),t},l.o=function(e,t){return Object.prototype.hasOwnProperty.call(e,t)},l.p="/";var f=this["webpackJsonpmetaflow-ui"]=this["webpackJsonpmetaflow-ui"]||[],i=f.push.bind(f);f.push=t,f=f.slice();for(var a=0;a<f.length;a++)t(f[a]);var p=i;r()}([])</script><script src="/static/js/2.f31c38de.chunk.js"></script><script src="/static/js/main.c1d36590.chunk.js"></script></body></html>
✅ 1
b
Are you able to see your runs in the UI?
s
@brave-lion-15961 No
the status chit in the upper right hand corner says 
Waiting for connection
runs complete normally however
and I can see the metadata in the postgresql server
b
Can you open the Network tab in DevTools? And then share a sample failed request, with the URL and the response
s
Screenshot 2023-05-26 at 7.22.07 AM.png
Screenshot 2023-05-26 at 7.23.32 AM.png
b
Do you see any XHR requests (non-websocket)?
s
No
I don't see any activity that looks like the service is actually reaching out
b
You can filter by clicking on the Fetch/XHR tab and reloading
Do you see anything at all on the UI? Or a blank page?
s
the page comes up
hold on
Screenshot 2023-05-26 at 9.12.50 AM.png
b
Thanks
s
i found one XHR
b
That page should make some XHR requests to get the runs, in addition to the websockets. I want to see what the response is for those
That may tell us whether it is a CORS issue, or the backend server is not running properly
s
Screenshot 2023-05-26 at 9.14.21 AM.png
looks like it came through as 200
Screenshot 2023-05-26 at 9.15.23 AM.png
so this wouldn't be a CORS issue
presumably?
b
What does Preview show?
s
hold on
white screen labeled 
You need to enable JavaScript to run this app.
I think that error doesn't make any sense
b
Is it wrapped in any HTML?
Seems strange because the XHR request can’t be made without js
s
yes
b
However if it is a HTML page with that text, it could be hitting the wrong thing
s
yes
it seems like the ui server is pointed somewhere wrong
b
Do you have access to the ui_backend logs? Probably running in Docker
s
python client can see runs and metadata
Copy code
27.0.0.1 - - [26/May/2023:13:45:00 +0000] "GET /?_group_limit=30&_limit=30&_order=-ts_epoch&status=failed%2Crunning HTTP/1.1" 200 1194 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15" "-"
127.0.0.1 - - [26/May/2023:13:45:00 +0000] "GET /manifest.json HTTP/1.1" 200 210 "<http://localhost:3000/?_group_limit=30&_limit=30&_order=-ts_epoch&status=failed%2Crunning>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15" "-"
127.0.0.1 - - [26/May/2023:13:45:01 +0000] "GET /api/ws HTTP/1.1" 200 1194 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15" "-"
127.0.0.1 - - [26/May/2023:13:45:01 +0000] "GET /api/features HTTP/1.1" 200 1194 "<http://localhost:3000/?_group_limit=30&_limit=30&_order=-ts_epoch&status=failed%2Crunning>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15" "-"
127.0.0.1 - - [26/May/2023:13:45:01 +0000] "GET /api/plugin HTTP/1.1" 200 1194 "<http://localhost:3000/?_group_limit=30&_limit=30&_order=-ts_epoch&status=failed%2Crunning>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15" "-"
127.0.0.1 - - [26/May/2023:13:45:01 +0000] "GET /api/version HTTP/1.1" 200 1194 "<http://localhost:3000/?_group_limit=30&_limit=30&_order=-ts_epoch&status=failed%2Crunning>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15" "-"
127.0.0.1 - - [26/May/2023:13:45:01 +0000] "GET /api/links HTTP/1.1" 200 1194 "<http://localhost:3000/?_group_limit=30&_limit=30&_order=-ts_epoch&status=failed%2Crunning>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15" "-"
127.0.0.1 - - [26/May/2023:13:45:01 +0000] "GET /api/notifications?start:le=1685108701297&end:ge=1685108701297 HTTP/1.1" 200 1194 "<http://localhost:3000/?_group_limit=30&_limit=30&_order=-ts_epoch&status=failed%2Crunning>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15" "-"
127.0.0.1 - - [26/May/2023:13:45:01 +0000] "GET /api/runs?_order=-ts_epoch&_limit=30&_group_limit=31&status=failed%2Crunning&_page=1 HTTP/1.1" 200 1194 "<http://localhost:3000/?_group_limit=30&_limit=30&_order=-ts_epoch&>
but the ui backend looks healthy
Copy code
NFO:CacheAsyncClient:cache_data/artifact_search:Pending stream keys: ['data:stream:GetTask:1dbd86097c532333ef24cee4c865f52f0eb0f51b']
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/ping HTTP/1.1" 200 187 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/ping HTTP/1.1" 200 4
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/hello/tasks/147 HTTP/1.1" 200 478 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/hello/tasks/147 HTTP/1.1" 200 293
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/hello/tasks/147/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 659 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/hello/tasks/147/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 474
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/hello/tasks/147/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 659 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/hello/tasks/147/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 474
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/hello/tasks/147/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 659 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/hello/tasks/147/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 474
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/start/tasks/146 HTTP/1.1" 200 478 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/start/tasks/146 HTTP/1.1" 200 293
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/start/tasks/146/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 659 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/start/tasks/146/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 474
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/start/tasks/146/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 659 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/start/tasks/146/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 474
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/start/tasks/146/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 659 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/start/tasks/146/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 474
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/end/tasks/148 HTTP/1.1" 200 476 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/end/tasks/148 HTTP/1.1" 200 291
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/end/tasks/148/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 657 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/end/tasks/148/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 472
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/end/tasks/148/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 657 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/end/tasks/148/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 472
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/end/tasks/148/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 657 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/end/tasks/148/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 472
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/_parameters/tasks/145 HTTP/1.1" 200 484 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/_parameters/tasks/145 HTTP/1.1" 200 299
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/_parameters/tasks/145/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 665 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/_parameters/tasks/145/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 480
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/_parameters/tasks/145/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 665 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/_parameters/tasks/145/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 480
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8083
INFO:aiohttp.access:127.0.0.1 [26/May/2023:15:46:06 +0000] "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/_parameters/tasks/145/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 665 "-" "python-requests/2.28.2"
INFO:CacheAsyncClient:cache_data/artifact_search:Message: DEBUG:urllib3.connectionpool:<http://localhost:8083> "GET /api/metadata/flows/HelloCloudFlow/runs/41/steps/_parameters/tasks/145/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 480
INFO:CacheAsyncClient:cache_data/artifact_search:{'op': 'worker_terminate', 'keys': 4, 'stream_key': 'data:stream:GetTask:1dbd86097c532333ef24cee4c865f52f0eb0f51b', 'idempotency_token': '030ab255ef37ef6cfea26c675c3434c662c078c7'}
b
That all looks right.
s
yeah
the UI frontend is silent. Backend looks healthy
completely baffling right?
b
See if you can use curl for that XHR request to take the browser/cors out of the equation
You can “copy as curl” from DevTools
s
same
Copy code
(base) jaredthompson@Jareds-MacBook-Pro metaflow-workload-identity-service-account % curl '<http://localhost:3000/api/runs?_order=-ts_epoch&_limit=30&_group_limit=31&_page=1>' \
  -H 'Accept: */*' \
  -H 'Accept-Language: en-US,en;q=0.9' \
  -H 'Connection: keep-alive' \
  -H 'Referer: <http://localhost:3000/?_group_limit=30&_limit=30&_order=-ts_epoch&status=completed%2Cfailed%2Crunning>' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36' \
  -H 'sec-ch-ua: "Google Chrome";v="113", "Chromium";v="113", "Not-A.Brand";v="24"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  --compressed
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="/favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="Build and manage real-life data science projects with ease."/><link rel="apple-touch-icon" href="/logo192.png"/><link rel="manifest" href="/manifest.json"/><title>Metaflow UI</title><link href="/static/css/main.95ba8333.chunk.css" rel="stylesheet"><script>window.METAFLOW_SERVICE="/api";window.MF_DEFAULT_TIME_FILTER_DAYS="";</script></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div><script>!function(e){function t(t){for(var n,l,f=t[0],i=t[1],a=t[2],c=0,s=[];c<f.length;c++)l=f[c],Object.prototype.hasOwnProperty.call(o,l)&&o[l]&&s.push(o[l][0]),o[l]=0;for(n in i)Object.prototype.hasOwnProperty.call(i,n)&&(e[n]=i[n]);for(p&&p(t);s.length;)s.shift()();return u.push.apply(u,a||[]),r()}function r(){for(var e,t=0;t<u.length;t++){for(var r=u[t],n=!0,f=1;f<r.length;f++){var i=r[f];0!==o[i]&&(n=!1)}n&&(u.splice(t--,1),e=l(l.s=r[0]))}return e}var n={},o={1:0},u=[];function l(t){if(n[t])return n[t].exports;var r=n[t]={i:t,l:!1,exports:{}};return e[t].call(r.exports,r,r.exports,l),r.l=!0,r.exports}l.m=e,l.c=n,l.d=function(e,t,r){l.o(e,t)||Object.defineProperty(e,t,{enumerable:!0,get:r})},l.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},l.t=function(e,t){if(1&t&&(e=l(e)),8&t)return e;if(4&t&&"object"==typeof e&&e&&e.__esModule)return e;var r=Object.create(null);if(l.r(r),Object.defineProperty(r,"default",{enumerable:!0,value:e}),2&t&&"string"!=typeof e)for(var n in e)l.d(r,n,function(t){return e[t]}.bind(null,n));return r},l.n=function(e){var t=e&&e.__esModule?function(){return e.default}:function(){return e};return l.d(t,"a",t),t},l.o=function(e,t){return Object.prototype.hasOwnProperty.call(e,t)},l.p="/";var f=this["webpackJsonpmetaflow-ui"]=this["webpackJsonpmetaflow-ui"]||[],i=f.push.bind(f);f.push=t,f=f.slice();for(var a=0;a<f.length;a++)t(f[a]);var p=i;r()}([])</script><script src="/static/js/2.f31c38de.chunk.js"></script><script src="/static/js/main.c1d36590.chunk.js"></script></body></html>%
exactly the same thing
1. Not the browser 2. Not the backend 3. The backend communicates to the service... 4. UI not communicating to backend despite everything else in the network working
b
You don’t have any proxies or nginx setup in the way? What you are seeing is the payload for the SPA homepage. This could happen if the requests are being redirected to /
s
that's very helpful
There are no proxies/nginx servers
this goes straight through to the port that's exposed
is it something in the metaflow config?
a
The current setup Jared is sharing is just using the 
forward_metaflow_ports.py
to access GKE resources locally (with the target values pointed to the corresponding GKE deployments) https://github.com/outerbounds/metaflow-tools/blob/72c5ecae49952dfc4020c163780ceff231f4b92a/scripts/forward_metaflow_ports.py We did try exposing the UI over a google hosted VPN via GKE service (type loadbalancer); however, that ended up having same issue w/ui-static. We haven’t put any proxy of our own in front of it
👍 1
Also, hey all! Nice to join the discussion, thanks for all the support!
👍 1
a
are you using any k8s ingress there? its kind of required to get HTTP path routing right
a
Only other suspicious thing encountered in the ui-backend when ssh’ing into the pod was found in the cache for artifact search, related to GCS 
_check_and_init_gs_dep
Untitled.cpp
It seems we currently only have the Cluster IP k8 Services defined, not a k8 ingress @sparse-agent-19997
🤔 1
a
yes and pick and install some k8s ingress controller in your cluster (nginx or maybe gke ingress in your case, or something else)
the idea there is that when you open ingress host in your browser, it makes sure that 
/api
requests are routed to UI backend container, and everything else goes to UI static container (but looks like the same host to your browser)
✅ 2
s
Ok, that's a good place to start. I didn't check to see if an ingress was installed over the Cluster
🙌 1
I had the helm Ingress installed in v9 and was getting the same error @agreeable-chef-18387
c
Just had this error: because I did not have an ingress controller installed! Thanks for the insight!
6 Views
Open in Slack
PreviousNext
Powered by