https://outerbounds.com/ logo
Join Slack
Powered by
Hi! In my k8s deployment, I have tried to change t...
# ask-metaflow
s
Hi! In my k8s deployment, I have tried to change the datastore location, now I have this in my helm chart:
Copy code
metaflow-ui:
  uiBackend:
    metaflowDefaultDatastore: "s3"
    metaflowDatastoreSysRootS3: "<s3://metaflow>"
    metaflowS3EndpointURL: "<http://minio.caic:9000>"
plus AWS_SECRET_* values through 
env
. Previously I had DefaultDatastore set to "local" I have restarted both deployments for mataflow-ui. Now I'm running yet another flow, and in UI it still shows me Since this run uses local datastore instead of cloud services, some information may be incomplete. And of course there are no data in my minio bucket Is my s3 configuration correct? Do I need to drop the database or deploy the backend service too to see the effect?
actually even if I completely redeploy my setup, even with the values shown above ^ it still shows me I'm using local datastore
OK, seems I was doing it wrong. S3 storage needs to be contacted from my client ... but then I'm not sure what's the relation of these values set in helm chart and my local env vars. Anyway, • I set AWS_* values for creds, both on my client and in the helm chart for metaflow-ui app • I ran 
metaflow configure aws
locally to configure s3 access for client, this sets METAFLOW_DATASTORE_SYSROOT_S3 locally • local run of 
metaflow configure show
shows me
Copy code
METAFLOW_DATASTORE_SYSROOT_S3=<s3://metaflow-eu-1>
METAFLOW_DATATOOLS_S3ROOT=<s3://metaflow-eu-1/data>
METAFLOW_DEFAULT_DATASTORE=s3
which seems correct. I make sure these are set in 
metaflow-metaflow-ui
pod as well • 
python example_flow.py --datastore s3 run
is stuck in `2025-06-27 125936.759 Workflow starting (run-id 7)`and nothing else What am I doning wrong? How can I see some debug output so I can find out what am I missing?
OK, solved some parts, obviously problems between keyboard and monitor; I had some old env variable set up that still pointed to my minio server instead of S3
OK, so I got further. Now I am able to run the flow and some artifacts are stored in s3. Great. When I go to UI and try to see the details, there's no DAG and on stderr/stdout parts it shows "Cache returned None for log content and raised no errors. The cache server might be experiencing issues." When I check the logs of the -ui pod, I see messages like 
The AWS Access Key Id you provided does not exist in our records.
I'm not sure if it is related to the part where I do not see the artifacts in the UI, (after all these messages are not reported as errors in the log), but I've seen people pointing out this issue several times here. What's the problem? I'm passing same AWS_ACCESS_KEY_ID on the cli as I'm passing as env variables to the metaflow-ui pod; so what is it complaining about?
@hundreds-zebra-57629 could you look at this issue as well, please?
Update: I think the issue is with my vpc endpoint
h
Hey Jiri, it is possible that the UI is trying to connect to the AWS S3 endpoint instead of your minio endpoint or the environment variables are being correctly set. A couple of suggessions: 1. Can you exec into the backend UI pod and inspect the environment variables? Confirm the aws credentials and METAFLOW_S3_ENDPOINT_URL environment variables are being correctly set. 2. Can you compare your helm values input to the input values here and see if there are any diffs?
s
I switched from S3 to minio, so nothing is minio related ATM. And now I really think the problem is that my endpoint is not configured to access desired S3 bucket. Because if I do provide the endpoint value, I get errors like 
botocore.exceptions.SSLError: SSL validation failed for <https://vpce-xyz>
(I hope the client does NOT try to verify certs if I pass S3_VERIFY_CERTIFICATE="False") So I need to figure out in my setup
Open in Slack
PreviousNext
Powered by