Hey guys, I’m trying to connect to my AWS resource...
# ask-metaflow
a
Hey guys, I’m trying to connect to my AWS resources deployed using CDK. I started with a simple example of uploading a file to S3. When using
S3(s3root=s3://...)
directly it works perfectly while using the config with
S3(run=self)
doesn’t.
Copy code
# This works:
with S3(s3root='s3://<my-bucket-name>/metaflow/') as s3:
    url = s3.put_files([('stockfish', 'stockfish')])
    print("File saved to", url)

# This doesn't work:
with S3(run=self) as s3:
    url = s3.put_files([('stockfish', 'stockfish')])
    print("File saved to", url)
My config:
Copy code
{
  "METAFLOW_BATCH_JOB_QUEUE": "JobQueue<hash>",
  "METAFLOW_DATASTORE_SYSROOT_S3": "s3://<my-bucket-name>/metaflow",
  "METAFLOW_DATATOOLS_SYSROOT_S3": "s3://<my-bucket-name>/metaflow/data",
  "METAFLOW_DEFAULT_DATASTORE": "s3",
  "METAFLOW_ECS_S3_ACCESS_IAM_ROLE": "<My-Batch-IAM-Role>"
}
Run Command:
Copy code
AWS_PROFILE=<my-profile> python helloaws.py run
Error Message:
Copy code
2022-11-22 13:37:16.949 [1669120599412313/start/1 (pid 29059)] File "/Users/basti/opt/anaconda3/envs/deepcheat/lib/python3.9/genericpath.py", line 155, in _check_arg_types
2022-11-22 13:37:16.949 [1669120599412313/start/1 (pid 29059)] raise TypeError("Can't mix strings and bytes in path components") from None
2022-11-22 13:37:16.949 [1669120599412313/start/1 (pid 29059)] TypeError: Can't mix strings and bytes in path components
Any help or ideas would be highly appreciated 🙂
1