https://outerbounds.com/ logo
Join Slack
Powered by
I'm hoping someone can help we with the UI setup. ...
# ask-metaflow
c
I'm hoping someone can help we with the UI setup. I'm trying to run this.
Copy code
docker run -d  \
-e MF_METADATA_DB_HOST='host' \
-e MF_METADATA_DB_PORT=5432 \
-e MF_METADATA_DB_USER='master' \
-e MF_METADATA_DB_PSWD='password' \
-e MF_METADATA_DB_NAME='metaflow' \
-p 8083:8083 \
netflixoss/metaflow_metadata_service \
/opt/latest/bin/python3 -m services.ui_backend_service.ui_server; \

docker run -d  -p 3000:3000 -e METAFLOW_SERVICE=<http://localhost:8083/api> metaflow-ui:latest
But I get the following attached result. I've checked the DB connection and it appears good. I've tried a bunch of different ports and variations of the METAFLOW_SERVICE=url. with and without /api
✅ 1
a
can you see what the error is in devtools
or anything in ui backend logs
c
Metaflow-UI docker container
Copy code
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/03/08 00:04:10 [notice] 1#1: using the "epoll" event method
2023/03/08 00:04:10 [notice] 1#1: nginx/1.23.3
2023/03/08 00:04:10 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
2023/03/08 00:04:10 [notice] 1#1: OS: Linux 5.15.0-1028-aws
2023/03/08 00:04:10 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/03/08 00:04:10 [notice] 1#1: start worker processes
2023/03/08 00:04:10 [notice] 1#1: start worker process 34
172.17.0.1 - - [08/Mar/2023:00:04:13 +0000] "GET /?_group_limit=30&_limit=30&_order=-ts_epoch&status=completed%2Cfailed%2Crunning&timerange_start=1675641600000 HTTP/1.1" 200 1208 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15" "-"
172.17.0.1 - - [08/Mar/2023:00:04:13 +0000] "GET /manifest.json HTTP/1.1" 200 210 "<http://localhost:3000/?_group_limit=30&_limit=30&_order=-ts_epoch&status=completed%2Cfailed%2Crunning&timerange_start=1675641600000>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15" "-"
172.17.0.1 - - [08/Mar/2023:03:26:11 +0000] "GET / HTTP/1.1" 200 1208 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15" "-"
172.17.0.1 - - [08/Mar/2023:03:26:11 +0000] "GET /manifest.json HTTP/1.1" 304 0 "<http://localhost:3000/>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15" "-"
Metaflow service
Copy code
ERROR:AsyncPostgresDB:ui:Exception occured
Traceback (most recent call last):
  File "/opt/latest/lib/python3.7/site-packages/aiopg/pool.py", line 309, in _acquire
    await self._fill_free_pool(True)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/pool.py", line 343, in _fill_free_pool
    **self._conn_kwargs,
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 1225, in _connect
    await self._poll(self._waiter, self._timeout)  # type: ignore
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 884, in _poll
    raise exc
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 881, in _poll
    await asyncio.wait_for(self._waiter, timeout)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 409, in wait_for
    await waiter
concurrent.futures._base.CancelledError
a
looks like it can't connect to the db. If the database is running on the same host I don't think by default you'd be able to connect to it from another container
c
The database is an RDS instance. I installed 
psql
inside the 
metaflow_metadata_service
container and was able to connect and run
Copy code
select * from runs_vs;
a
interesting, how did you run migrations to create those tables?
..and you were able to run flows, i.e. except for UI things work?
c
I setup everything else with the cloudformation template. Everything else works correctly.
a
especially the last 3, and 
METAFLOW_SERVICE_URL
will be localhost:8083 for you too (i know its a little weird but metadata backend service includes a copy of metadata service in it, so it works)
c
Like this ?
Copy code
docker run -d \
-e MF_METADATA_DB_HOST='RDS HOST' \
-e MF_METADATA_DB_PORT=5432 \
-e MF_METADATA_DB_USER='master' \
-e MF_METADATA_DB_PSWD='*******' \
-e MF_METADATA_DB_NAME='metaflow' \
-e UI_ENABLED='1' \
-e PATH_PREFIX='/api' \
-e MF_DATASTORE_ROOT='<s3://------->-/metaflow' \
-e METAFLOW_DATASTORE_SYSROOT_S3='<s3://----->-/metaflow' \
-e LOG_LEVEL='DEBUG' \
-e METAFLOW_SERVICE_URL='<http://localhost:8083/api/metadata>' \
-e METAFLOW_DEFAULT_DATASTORE='s3' \
-e METAFLOW_DEFAULT_METADATA='service' \
-p 8083:8083 \
netflixoss/metaflow_metadata_service \
/opt/latest/bin/python3 -m services.ui_backend_service.ui_server;
\
docker run -d  -p 3000:3000 -e METAFLOW_SERVICE=<http://localhost:8083/> metaflow-ui:latest
a
yes
c
Same result 
waiting for connection
Copy code
/root/services/ui_backend_service/ui_server.py:50: DeprecationWarning: loop argument is deprecated
  _app = web.Application(loop=loop)
/root/services/ui_backend_service/ui_server.py:51: DeprecationWarning: loop argument is deprecated
  app = web.Application(loop=loop) if len(PATH_PREFIX) > 0 else _app
INFO:AsyncPostgresDB:ui:Setting up notify trigger for flows_v3
   Keys: ['flow_id']
INFO:AsyncPostgresDB:ui:Setting up notify trigger for runs_v3
   Keys: ['flow_id', 'run_number', 'last_heartbeat_ts']
INFO:AsyncPostgresDB:ui:Setting up notify trigger for steps_v3
   Keys: ['flow_id', 'run_number', 'step_name']
INFO:AsyncPostgresDB:ui:Setting up notify trigger for tasks_v3
   Keys: ['flow_id', 'run_number', 'step_name', 'task_id']
INFO:AsyncPostgresDB:ui:Setting up notify trigger for artifact_v3
   Keys: ['flow_id', 'run_number', 'step_name', 'task_id', 'attempt_id', 'name']
INFO:AsyncPostgresDB:ui:Setting up notify trigger for metadata_v3
   Keys: ['flow_id', 'run_number', 'step_name', 'task_id', 'field_name', 'value']
INFO:AsyncPostgresDB:ui:Connection established.
   Pool min: 1 max: 10

INFO:AsyncPostgresDB:ui:cache:Connection established.
   Pool min: 1 max: 10

INFO:AsyncPostgresDB:ui:notify:Connection established.
   Pool min: 1 max: 10

INFO:ListenNotify:Connection acquired
INFO:AsyncPostgresDB:ui:heartbeat:Connection established.
   Pool min: 1 max: 10

INFO:AsyncPostgresDB:ui:websocket:Connection established.
   Pool min: 1 max: 10

INFO:AutoCompleteApi:15 cached tags in memory consuming 0 Mb
INFO:AsyncPostgresDB:global:Connection established.
   Pool min: 1 max: 10

INFO:root:Metadata service available at <http://0.0.0.0:8083/api/metadata>
INFO:Plugin:Init plugins
INFO:Plugin:Plugins ready: []
INFO:CacheAsyncClient:cache_data/artifact_search:Message: CACHE [2023-03-08T17:31:44.518014] IO ERROR: (None) Cache initialized with 0 permanents objects, 0 disposable objects, totaling 0 bytes.
INFO:CacheAsyncClient:cache_data/artifact_search:{'op': 'worker_terminate', 'keys': 1, 'stream_key': None, 'idempotency_token': '3bcf3b24ec95b2826fe1ccbef3c92f3bc1bec9e9'}
INFO:CacheAsyncClient:cache_data/artifact_search:Pending stream keys: []
INFO:CacheAsyncClient:cache_data/dag:Message: CACHE [2023-03-08T17:31:45.525046] IO ERROR: (None) Cache initialized with 0 permanents objects, 0 disposable objects, totaling 0 bytes.
INFO:CacheAsyncClient:cache_data/dag:{'op': 'worker_terminate', 'keys': 1, 'stream_key': None, 'idempotency_token': 'a23a718088ddeae883cad9f6980b2951d9ac059c'}
INFO:CacheAsyncClient:cache_data/dag:Pending stream keys: []
INFO:CacheAsyncClient:cache_data/log:Message: CACHE [2023-03-08T17:31:46.529103] IO ERROR: (None) Cache initialized with 0 permanents objects, 0 disposable objects, totaling 0 bytes.
INFO:CacheAsyncClient:cache_data/log:{'op': 'worker_terminate', 'keys': 1, 'stream_key': None, 'idempotency_token': '29ed52ab9e05e6ef2f72e7c7bb03b9bc7bb36446'}
INFO:CacheAsyncClient:cache_data/log:Pending stream keys: []
INFO:CacheStore:Preloading 32 runs
INFO:CacheAsyncClient:cache_data/artifact_search:{'op': 'worker_create', 'keys': 1, 'stream_key': 'data:stream:GetParameters:edc9b7d5e5ecd873aafd753f425a180c9f2172c0', 'idempotency_token': '222abf772912619bd1b767fdcea1816a9c0c1048'}
INFO:CacheAsyncClient:cache_data/artifact_search:Pending stream keys: ['data:stream:GetParameters:edc9b7d5e5ecd873aafd753f425a180c9f2172c0']
INFO:aiohttp.access:127.0.0.1 [08/Mar/2023:17:31:46 +0000] "GET /api/metadata/flows/EvaluateVideoFlow/runs/499/steps/_parameters HTTP/1.1" 200 423 "-" "python-requests/2.28.2"
INFO:aiohttp.access:127.0.0.1 [08/Mar/2023:17:31:46 +0000] "GET /api/metadata/flows/EvaluateVideoFlow/runs/499/steps/_parameters/tasks HTTP/1.1" 200 473 "-" "python-requests/2.28.2"
INFO:aiohttp.access:127.0.0.1 [08/Mar/2023:17:31:46 +0000] "GET /api/metadata/flows/EvaluateVideoFlow/runs/499/steps/_parameters/tasks/2698/artifacts HTTP/1.1" 200 4169 "-" "python-requests/2.28.2"
INFO:AutoCompleteApi:15 cached tags in memory consuming 0 Mb
It does seem to be reading the service
Copy code
INFO:aiohttp.access:127.0.0.1 [08/Mar/2023:17:41:58 +0000] "GET /api/metadata/ping HTTP/1.1" 200 187 "-" "python-requests/2.28.2"
INFO:aiohttp.access:127.0.0.1 [08/Mar/2023:17:41:58 +0000] "GET /api/metadata/flows/EvaluateVideoFlow/runs/499/steps/process_video/tasks/2700 HTTP/1.1" 200 473 "-" "python-requests/2.28.2"
INFO:aiohttp.access:127.0.0.1 [08/Mar/2023:17:41:58 +0000] "GET /api/metadata/flows/EvaluateVideoFlow/runs/499/steps/process_video/tasks/2700/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 683 "-" "python-requests/2.28.2"
INFO:aiohttp.access:127.0.0.1 [08/Mar/2023:17:41:58 +0000] "GET /api/metadata/flows/EvaluateVideoFlow/runs/499/steps/process_video/tasks/2700/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 683 "-" "python-requests/2.28.2"
INFO:aiohttp.access:127.0.0.1 [08/Mar/2023:17:41:58 +0000] "GET /api/metadata/flows/EvaluateVideoFlow/runs/499/steps/process_video/tasks/2700/artifacts/_task_ok/attempt/0 HTTP/1.1" 200 683 "-" "python-requests/2.28.2"
a
^ that looks pretty healthy, where do you see waiting for connection?
ah in the UI in the top right
👍 1
also can you check out what devtools in the browser say? i wonder if its something between browser connecting to the API now
c
devtools?
a
yep developer tools in chrome, where it shows you all network requests and such
other browsers have it too under slightly different names
looking for this view
c
Screenshot 2023-03-08 at 9.54.46 AM.png
I built the 
metaflow-ui
image with version 1.2.5 and 1.2.4. I'm going to try going back to 1.1.4
a
do you see any errors at all in that view if you refresh the page? can you tell if its connecting to the right port etc. (where backend is listening)
especially the requests in the beginning of the page load
c
Screenshot 2023-03-08 at 10.15.10 AM.png
I'm using an ssh port forward is that causing this?
a
can you see if those failed requests are actually trying to connect to localhost:8083 where backend is listening?
its usually either its not trying to connect to the correct URL for the backend, or those requests are prevented by the browser due to CORS, but you should be able to see which one is it in devtools if you click around and look at specific errors
c
Screenshot 2023-03-08 at 10.51.38 AM.png
a
hmm im not 100% sure from this screenshot still but can you open that request url in browser and/or 
curl
it ?
or maybe somewhere in devtools it has more details for the error
like what does this 
net:::ERR_...
bit say exactly
c
I can't reproduce that error exactly
Whats the difference between port 3000 and 8083? Whats running on each?
a
:3000 has metaflow-ui container that has all the "static" front end assets like js files, html and such
:8083 is UI backend API container (htat includes metadata service api on a sub path)
c
I see. I forwarded 8083 as well. and now I get CORS errors
a
so you open :3000 in your browser and js code will call to :8083
you also get those 404 which i suspect because this paramter for the UI container is not right
Copy code
METAFLOW_SERVICE=<http://localhost:8083/>
do you have set to 
<http://localhost:8083/>
or 
<http://localhost:8083/api>
?
c
METAFLOW_SERVICE=<http://localhost:8083/>
a
can you try 
/api
? i think you'd still get CORS errors but no 404
✅ 1
for CORS try setting 
ORIGIN_TO_ALLOW_CORS_FROM=*
on the ui backend container
c
Adding the 
/api
worked ✅
Thank you for the help 🙂
You'all are the best 🥳
👍 1
8 Views
Open in Slack
PreviousNext
Powered by