Hello
We are trying to run metaflow only on kubern...
# ask-metaflowk
kind-potato-31471
07/11/2023, 12:32 PMHello
We are trying to run metaflow only on kubernetes. We defined s3)endpoint_url to our s3 service (not aws). While running command
!python test_dag.py --datastore=s3 --with kubernetes run
we get the error
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Looks like the problem is with the env var METAFLOW_DATASTORE_SYSROOT_S3 in config.json. When we set it, everything fails. Without this variable flow doesn't work
ā
1
b
bulky-afternoon-92433
07/11/2023, 12:41 PMWhich version of Metaflow are you running? There was a bug affecting S3 operations with non-AWS S3 providers that should be fixed in 2.9.9 (affecting since 2.9.5)
k
kind-potato-31471
07/11/2023, 12:43 PM2.9.9
kind-potato-31471
07/11/2023, 12:46 PMCan you give me issue link?
b
bulky-afternoon-92433
07/11/2023, 12:48 PMJust to be sure that this is not related to the ServerSideEncryption issues with the S3 client, have you tried an older version of Metaflow to see if the problem persists? 2.9.4 would be a good test.
Also if you could mention which S3 service you are using, this will help with trying to reproduce the error.
k
kind-potato-31471
07/11/2023, 12:49 PMok we can try
we use ceph
š 1
kind-potato-31471
07/11/2023, 12:52 PMtried 2.9.4, the error still exists
kind-potato-31471
07/11/2023, 12:59 PMHere are the steps to reproduce the error:
1) Define all environmental variables:
{
"METAFLOW_DEFAULT_METADATA": "service",
"METAFLOW_KUBERNETES_CONTAINER_REGISTRY": "metaflow-metaflow-service",
"METAFLOW_KUBERNETES_NAMESPACE": "metaflow",
"METAFLOW_KUBERNETES_SECRETS": "test",
"METAFLOW_KUBERNETES_SERVICE_ACCOUNT": "default",
"METAFLOW_SERVICE_INTERNAL_URL": "http://metaflow-metaflow-ui-static.metaflow.svc.local:3000",
"METAFLOW_SERVICE_URL": "http://metaflow.example.com/",
"METAFLOW_DATASTORE_S3_ROOT": "http://s3.example.com"
}
2) !python test_dag.py --datastore=s3 --with kubernetes run
3)
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
b
bulky-afternoon-92433
07/11/2023, 1:31 PMsome further questions on the matter:
ā¢ does the error happen during step execution, or immediately when submitting the flow?
ā¦ any details in the error as to where the JSONDecodeError is raised from?
ā¢ are all the services reachable from the environment that is throwing the error? (both the S3 bucket, and the metadata service)
k
kind-potato-31471
07/11/2023, 3:52 PM1. We found out that we used wrong url for the ui. Now we can register all the flows, run them in kubernetes. Moreover, we faced 2 new issues:
2. 1) When we open dag tab, we still see JSON decode error
3. 2) Step 'Start' generates the error about metaflow service version. It is saying to use 2.0.6 version or higher, but our version is 2.3.5
b
bulky-afternoon-92433
07/11/2023, 4:54 PMWould the tips in https://outerbounds-community.slack.com/archives/C02116BBNTU/p1679361473224039?thread_ts=1679347750.946389&cid=C02116BBNTU this thread be useful for your case as well?
depending on how you have deployed the stack, basically making sure that
METAFLOW_SERVICE_URLk
kind-potato-31471
07/12/2023, 1:11 PMall good, thx!
š 1
b
bulky-afternoon-92433
07/12/2023, 1:14 PMexcellent š
22 Views