Hey, Metaflow community running into issues --with...
# ask-metaflowm
microscopic-airline-4836
05/02/2023, 7:30 PMHey, Metaflow community running into issues --with-kubernetes on Minio using Metaflow. Is that the s3 credentials? I have scanned through the posts and as one user suggested passed S3 creds as an @environment variable using
@environment Copy code
Setting up task environment.
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package..✅ 1
microscopic-airline-4836
05/02/2023, 7:32 PMComplete logs,
Copy code
Validating your flow...
The graph looks good!
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
2023-05-02 14:11:14.260 Workflow starting (run-id 38):
2023-05-02 14:11:17.621 [38/start/500 (pid 75928)] Task is starting.
2023-05-02 14:11:20.275 [38/start/500 (pid 75928)] INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
2023-05-02 14:11:20.417 [38/start/500 (pid 75928)] INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
2023-05-02 14:11:21.466 [38/start/500 (pid 75928)] [pod t-cjnhp-fr65d] Task is starting (Pod is running, Container is running)...
2023-05-02 14:12:58.104 [38/start/500 (pid 75928)] Kubernetes error:
2023-05-02 14:12:58.104 [38/start/500 (pid 75928)] Error (exit code 1). This could be a transient error. Use @retry to retry.
2023-05-02 14:12:58.490 [38/start/500 (pid 75928)]
2023-05-02 14:12:58.904 [38/start/500 (pid 75928)] Task failed.
2023-05-02 14:12:59.419 Workflow failed.
2023-05-02 14:12:59.419 Terminating 0 active tasks...
2023-05-02 14:12:59.419 Flushing logs...
Step failure:
Step start (task-id 500) failed.v
victorious-lawyer-58417
05/02/2023, 8:24 PMdo you have the usual
AWS_SECRET_ACCESS_KEYAWS_ACCESS_KEY_IDvictorious-lawyer-58417
05/02/2023, 8:24 PMif you have, you don't need to use
@environmentm
microscopic-airline-4836
05/02/2023, 8:24 PMYes, I do. I can see logs and artifacts being stored in Minio. Thanks, for your reply
microscopic-airline-4836
05/02/2023, 8:25 PMIt exhausts with
Copy code
Setting up task environment.
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package...
fatal error: Unable to locate credentials
Downloading code package...
fatal error: Unable to locate credentials
Failed to download code package from <s3://metaflow/HelloFlow/data/59/59f713ccb5b727f22a0c970e987d3b6461e37a92> after 6 tries. Exiting.v
victorious-lawyer-58417
05/02/2023, 8:26 PMdoes that happen when you execute
--with kubernetes@kubernetesm
microscopic-airline-4836
05/02/2023, 8:27 PMYes, that happens with the
@kubernetesv
victorious-lawyer-58417
05/02/2023, 8:29 PMright, you need to make the credentials available to all pods too. See instructions here (the instructions talk about Airflow but they apply to other setups too)
m
microscopic-airline-4836
05/02/2023, 8:31 PMMuch appreciated. I am missing the secrets parameter, will try it out. Thanks again!
👍 1
v
victorious-lawyer-58417
05/02/2023, 8:32 PMlet us know if it keeps causing trouble - we can help debug it further
👍 1
m
microscopic-airline-4836
05/03/2023, 6:42 PMSetting the secret did resolve the issue. Thanks @straight-shampoo-11124
v
victorious-lawyer-58417
05/03/2023, 6:46 PMawesome!
3 Views