https://outerbounds.com/ logo
Join Slack
Powered by
Hello all, I'm experiencing various errors from t...
# ask-metaflow
a
Hello all, I'm experiencing various errors from the the ECS containers, both for the UI and metadata services. Any insight or suggestions on what could be causing these errors. This is from deploying the terraform template to AWS.
✅ 1
Error from the metadata ui -
Copy code
DEBUG:asyncio:Using selector: EpollSelector
/root/services/ui_backend_service/ui_server.py:50: DeprecationWarning: loop argument is deprecated
  _app = web.Application(loop=loop)
/root/services/ui_backend_service/ui_server.py:51: DeprecationWarning: loop argument is deprecated
  app = web.Application(loop=loop) if len(PATH_PREFIX) > 0 else _app
ERROR:root:Encountered an exception: [Errno 2] No such file or directory
ERROR:AsyncPostgresDB:ui:Exception occurred
Traceback (most recent call last):
  File "/root/services/data/postgres_async_db.py", line 83, in _init
    echo=AIOPG_ECHO)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/pool.py", line 300, in from_pool_fill
    await self._fill_free_pool(False)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/pool.py", line 343, in _fill_free_pool
    **self._conn_kwargs,
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 1225, in _connect
    await self._poll(self._waiter, self._timeout)  # type: ignore
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 884, in _poll
    raise exc
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 881, in _poll
    await asyncio.wait_for(self._waiter, timeout)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 423, in wait_for
    raise futures.TimeoutError()
concurrent.futures._base.TimeoutError
ERROR:root:Encountered an exception: [Errno 2] No such file or directory
ERROR:AsyncPostgresDB:ui:Exception occurred
Traceback (most recent call last):
  File "/root/services/data/postgres_async_db.py", line 83, in _init
    echo=AIOPG_ECHO)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/pool.py", line 300, in from_pool_fill
    await self._fill_free_pool(False)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/pool.py", line 343, in _fill_free_pool
    **self._conn_kwargs,
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 1225, in _connect
    await self._poll(self._waiter, self._timeout)  # type: ignore
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 884, in _poll
    raise exc
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 881, in _poll
    await asyncio.wait_for(self._waiter, timeout)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 423, in wait_for
    raise futures.TimeoutError()
concurrent.futures._base.TimeoutError
ERROR:root:Encountered an exception: [Errno 2] No such file or directory
ERROR:AsyncPostgresDB:ui:Exception occurred
Traceback (most recent call last):
  File "/root/services/data/postgres_async_db.py", line 83, in _init
    echo=AIOPG_ECHO)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/pool.py", line 300, in from_pool_fill
    await self._fill_free_pool(False)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/pool.py", line 343, in _fill_free_pool
    **self._conn_kwargs,
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 1225, in _connect
    await self._poll(self._waiter, self._timeout)  # type: ignore
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 884, in _poll
    raise exc
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 881, in _poll
    await asyncio.wait_for(self._waiter, timeout)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 423, in wait_for
    raise futures.TimeoutError()
concurrent.futures._base.TimeoutError
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/services/ui_backend_service/ui_server.py", line 152, in <module>
    main()
  File "/root/services/ui_backend_service/ui_server.py", line 129, in main
    the_app = app(loop, DBConfiguration())
  File "/root/services/ui_backend_service/ui_server.py", line 54, in app
    loop.run_until_complete(async_db._init(db_conf=db_conf, create_triggers=DB_TRIGGER_CREATE))
  File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/root/services/data/postgres_async_db.py", line 98, in _init
    raise e
  File "/root/services/data/postgres_async_db.py", line 83, in _init
    echo=AIOPG_ECHO)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/pool.py", line 300, in from_pool_fill
    await self._fill_free_pool(False)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/pool.py", line 343, in _fill_free_pool
    **self._conn_kwargs,
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 1225, in _connect
    await self._poll(self._waiter, self._timeout)  # type: ignore
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 884, in _poll
    raise exc
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 881, in _poll
    await asyncio.wait_for(self._waiter, timeout)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 423, in wait_for
    raise futures.TimeoutError()
concurrent.futures._base.TimeoutError
Error from the metadata service:
Copy code
2
connecting
booting...
[Errno 111] Connection refused
1
connecting
booting...
[Errno 111] Connection refused
ran out of retries to get migration version, exiting
Failed to get env version
a
it looks like it can't connect to the database (RDS instance)
any notable differences between your setup and let's say minimal example ? I guess you're using private subnets, anyhting else?
a
Potentially. I'll investigate. I am using a VPC and private subnets that already exist in the AWS account as opposed to creating new ones and I'm not importing 
"outerbounds/metaflow/aws"
. I refactored some of the code to meet our own custom security requirements.
a
i see, yeah its likely either connectivity between RDS and the subnet where ECS service is running, or maybe something off with postgres hostname provided to ECS container but thats less common
a
Thanks for the lead!
👍 1
@average-beach-28850 - we created an ec2 instance in the same subnet and security group as ecs service and looks like the connectivity was successful, as shown below:
Copy code
[root@ip-10-91-115-53 ~]# telnet <http://metaflow-metaflow-ampsdemo.cc1tsv3ywqpu.us-west-2.rds.amazonaws.com|metaflow-metaflow-ampsdemo.cc1tsv3ywqpu.us-west-2.rds.amazonaws.com> 5432
Trying 10.91.115.82...
Connected to metaflow-metaflow-ampsdemo.cc1tsv3ywqpu.us-west-2.rds.amazonaws.com.
Could it be the python handler within the ecs container?
a
Hmm interesting, every time I saw that error before it was def network connection issue. I’d maybe double check if the Postgres host is passed properly to the container. Otherwise I’ll try to deploy the template using private subnet in our dev account here and see if I can replicate this
a
Thanks @average-beach-28850. Out of curiosity, we pulled the docker image 
netflixoss/metaflow_metadata_service
inside the test ec2 instance we created that is inside the same subnet & using the same security group as the RDS instance. Then we ran the docker container, as follows:
Copy code
docker pull netflixoss/metaflow_metadata_service

docker run -e MF_METADATA_DB_HOST='<instance_name>.<http://us-east-1.rds.amazonaws.com|us-east-1.rds.amazonaws.com>' \
-e MF_METADATA_DB_PORT=5432 \
-e MF_METADATA_DB_USER='metaflow' \
-e MF_METADATA_DB_PSWD='BLAHBLAHBLAH' \
-e MF_METADATA_DB_NAME='metaflow' \
-it -p 8082:8082 -p 8080:8080 metaflow_metadata_service
We were able to recreate the same error, unfortunately:
Copy code
5
connecting
booting...
[Errno 111] Connection refused
/root/services/migration_service/migration_server.py:16: DeprecationWarning: loop argument is deprecated
  app = web.Application(loop=loop)
ERROR:asyncio:Exception in callback Connection._ready(<weakref at 0...x7f70c88ac198>) at /opt/latest/lib/python3.7/site-packages/aiopg/connection.py:779
handle: <Handle Connection._ready(<weakref at 0...x7f70c88ac198>) at /opt/latest/lib/python3.7/site-packages/aiopg/connection.py:779>
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 838, in _ready
    self._fileno, self._ready, weak_self  # type: ignore
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 334, in add_writer
    return self._add_writer(fd, callback, *args)
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 294, in _add_writer
    (reader, handle))
  File "/usr/lib/python3.7/selectors.py", line 389, in modify
    self._selector.modify(key.fd, selector_events)
FileNotFoundError: [Errno 2] No such file or directory
4
connecting
booting...
[Errno 111] Connection refused
3
connecting
booting...
[Errno 111] Connection refused
2
connecting
booting...
[Errno 111] Connection refused
1
connecting
booting...
[Errno 111] Connection refused
ran out of retries to get migration version, exiting
Failed to get env version
One thing we noted though is that it's trying to find this port in this line. Do we need to pass this in to the environment variable of the container?
a
Can you try connecting with psql from that host as well? Also I found one report where this cryptic error was apperently due to wrong password https://github.com/aio-libs/aiopg/issues/837#issuecomment-1410238500
a
You're absolutely 100% right. I did some digging and there is indeed a password inconsistency (plus it's using special characters). Will fix that and should resolve the issue. Thanks so much for that tidbit!
Getting a different error now First off, I can confirm that I can connect to postgresql RDS through 
psql
. Unfortunately, I am getting this new error though.
Copy code
DEBUG:asyncio:Using selector: EpollSelector
/root/services/ui_backend_service/ui_server.py:50: DeprecationWarning: loop argument is deprecated
  _app = web.Application(loop=loop)
/root/services/ui_backend_service/ui_server.py:51: DeprecationWarning: loop argument is deprecated
  app = web.Application(loop=loop) if len(PATH_PREFIX) > 0 else _app
INFO:AsyncPostgresDB:ui:Setting up notify trigger for flows_v3
   Keys: ['flow_id']
ERROR:AsyncPostgresDB:ui:Exception occurred
Traceback (most recent call last):
  File "/root/services/data/postgres_async_db.py", line 86, in _init
    await table._init(create_triggers=create_triggers)
  File "/root/services/data/postgres_async_db.py", line 163, in _init
    await PostgresUtils.setup_trigger_notify(db=self.db, table_name=self.table_name, keys=self.trigger_keys)
  File "/root/services/data/postgres_async_db.py", line 462, in setup_trigger_notify
    commands=_commands
  File "/root/services/data/postgres_async_db.py", line 394, in create_trigger_if_missing
    await cur.execute(command)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 426, in execute
    await self._conn._poll(waiter, timeout)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 881, in _poll
    await asyncio.wait_for(self._waiter, timeout)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    return fut.result()
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 788, in _ready
    state = self._conn.poll()
psycopg2.errors.UndefinedTable: relation "public.flows_v3" does not exist
10.91.115.86 - - [05/Feb/2023:07:40:33 +0000] "GET / HTTP/1.1" 200 1194 "-" "ELB-HealthChecker/2.0" "-"
/opt/latest/lib/python3.7/site-packages/aiopg/pool.py:479: ResourceWarning: Unclosed 1 connections in <aiopg.pool.Pool object at 0x7f4e348a8198>
  f"Unclosed {left} connections in {self!r}", ResourceWarning
INFO:AsyncPostgresDB:ui:Setting up notify trigger for flows_v3
   Keys: ['flow_id']
ERROR:AsyncPostgresDB:ui:Exception occurred
Traceback (most recent call last):
  File "/root/services/data/postgres_async_db.py", line 86, in _init
    await table._init(create_triggers=create_triggers)
  File "/root/services/data/postgres_async_db.py", line 163, in _init
    await PostgresUtils.setup_trigger_notify(db=self.db, table_name=self.table_name, keys=self.trigger_keys)
  File "/root/services/data/postgres_async_db.py", line 462, in setup_trigger_notify
    commands=_commands
  File "/root/services/data/postgres_async_db.py", line 394, in create_trigger_if_missing
    await cur.execute(command)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 426, in execute
    await self._conn._poll(waiter, timeout)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 881, in _poll
    await asyncio.wait_for(self._waiter, timeout)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    return fut.result()
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 788, in _ready
    state = self._conn.poll()
psycopg2.errors.UndefinedTable: relation "public.flows_v3" does not exist
/opt/latest/lib/python3.7/site-packages/aiopg/pool.py:479: ResourceWarning: Unclosed 1 connections in <aiopg.pool.Pool object at 0x7f4e348a8550>
  f"Unclosed {left} connections in {self!r}", ResourceWarning
INFO:AsyncPostgresDB:ui:Setting up notify trigger for flows_v3
   Keys: ['flow_id']
ERROR:AsyncPostgresDB:ui:Exception occurred
Traceback (most recent call last):
  File "/root/services/data/postgres_async_db.py", line 86, in _init
    await table._init(create_triggers=create_triggers)
  File "/root/services/data/postgres_async_db.py", line 163, in _init
    await PostgresUtils.setup_trigger_notify(db=self.db, table_name=self.table_name, keys=self.trigger_keys)
  File "/root/services/data/postgres_async_db.py", line 462, in setup_trigger_notify
    commands=_commands
  File "/root/services/data/postgres_async_db.py", line 394, in create_trigger_if_missing
    await cur.execute(command)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 426, in execute
    await self._conn._poll(waiter, timeout)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 881, in _poll
    await asyncio.wait_for(self._waiter, timeout)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    return fut.result()
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 788, in _ready
    state = self._conn.poll()
psycopg2.errors.UndefinedTable: relation "public.flows_v3" does not exist
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/services/ui_backend_service/ui_server.py", line 152, in <module>
    main()
  File "/root/services/ui_backend_service/ui_server.py", line 129, in main
    the_app = app(loop, DBConfiguration())
  File "/root/services/ui_backend_service/ui_server.py", line 54, in app
    loop.run_until_complete(async_db._init(db_conf=db_conf, create_triggers=DB_TRIGGER_CREATE))
  File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/root/services/data/postgres_async_db.py", line 98, in _init
    raise e
  File "/root/services/data/postgres_async_db.py", line 86, in _init
    await table._init(create_triggers=create_triggers)
  File "/root/services/data/postgres_async_db.py", line 163, in _init
    await PostgresUtils.setup_trigger_notify(db=self.db, table_name=self.table_name, keys=self.trigger_keys)
  File "/root/services/data/postgres_async_db.py", line 462, in setup_trigger_notify
    commands=_commands
  File "/root/services/data/postgres_async_db.py", line 394, in create_trigger_if_missing
    await cur.execute(command)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 426, in execute
    await self._conn._poll(waiter, timeout)
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 881, in _poll
    await asyncio.wait_for(self._waiter, timeout)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    return fut.result()
  File "/opt/latest/lib/python3.7/site-packages/aiopg/connection.py", line 788, in _ready
    state = self._conn.poll()
psycopg2.errors.UndefinedTable: relation "public.flows_v3" does not exist
a
can you post metaflow-service logs as well? I think there is a race condition with how db migrations are applied for the first time. Those migrations are supposed to be executed by the metadata service container not UI backend. We fixed it in the Kubernetes-based version because K8S has some nice features for this (init containers), but in ECS its still done in a bit convoluted way unfortunately What I'd try since you already have psql there is check what tables are available ( 
\dt
in psql), if there is nothing then migrations didn't run at all. If there are some tables it may help to DROP all of them and restart metadata service to create them again
a
Hi Oleg, With regards to the metaflow service logs, It's the same 
connection refused
error.
I checked the 
metadata service container
environment variables and all looks consistent. Will now check the tables available.
Unfortunately, not finding any tables, so looks like the migrations didn't run then.
a
ok we should figure out how to get metadata service to connect to the db first, it looks like it failed to 🤔 has to be something either with network connectivity or the confg again. i can actually hop on a zoom call sometime tomorrow or day after to debug together if that helps, or we can continue here
thankyou 1
a
I think I know what the issue is after I toiled around with different security groups for the EC2 instance I deployed to test the RDS connectivity. It was my mistake! I think I passed in a custom security group for the ECS services. I only did that because I was testing one module at a time.
Okay resolved the security group discrepancy. Now I'm getting a different error, but this one is a bit more descriptive.
Copy code
2023/02/06 05:14:50 goose run: pq: no pg_hba.conf entry for host "10.91.115.57", user "metaflow", database "metaflow", SSL off
/root/services/migration_service/migration_server.py:16: DeprecationWarning: loop argument is deprecated
  app = web.Application(loop=loop)
/opt/latest/lib/python3.7/site-packages/aiopg/pool.py:479: ResourceWarning: Unclosed 1 connections in <aiopg.pool.Pool object at 0x7fe279f2df98>
  f"Unclosed {left} connections in {self!r}", ResourceWarning
/opt/latest/lib/python3.7/site-packages/aiopg/pool.py:479: ResourceWarning: Unclosed 1 connections in <aiopg.pool.Pool object at 0x7fe279eed9e8>
  f"Unclosed {left} connections in {self!r}", ResourceWarning
/root/services/migration_service/migration_server.py:26: DeprecationWarning: Application.make_handler(...) is deprecated, use AppRunner API instead
  handler = the_app.make_handler()
Running initial migration..
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "/opt/latest/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
    resp = await request_handler(request)
  File "/opt/latest/lib/python3.7/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/root/services/migration_service/api/admin.py", line 53, in version
    version = await ApiUtils.get_latest_compatible_version()
  File "/root/services/migration_service/api/utils.py", line 62, in get_latest_compatible_version
    raise Exception("Failed to run initial migration")
Exception: Failed to run initial migration
INFO:aiohttp.access:127.0.0.1 [06/Feb/2023:05:14:50 +0000] "GET /version HTTP/1.1" 500 244 "-" "python-requests/2.27.1"
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/services/migration_service/get_virtual_env.py", line 39, in <module>
    r.raise_for_status()
  File "/opt/latest/lib/python3.7/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: <http://localhost:8082/version>
a
this is another connection error but this time i think it may be a mismatch between SSL settings on the client and the server. open source Metaflow Service currently doesn't use SSL, and you'll see that in that tf template RDS instance it doesn't require it. Any chance you have this parameter set to require ssl?
a
Thanks Oleg! Great catch. We were using an internally developed tf module to create RDS and it had SSL on by default. Can confirm that turning that off did the trick.
👍 1
a
so did it all work in the end?
a
It did. Thanks so much Oleg! Much obliged. Excited to try out some flows.
a
great! let us know how that goes :)
a
Seem to be experiencing some trouble with IAM permissions with batch. Created a different thread for this. Currently trying to troubleshoot the issue.
5 Views
Open in Slack
PreviousNext
Powered by