https://outerbounds.com/ logo
Join Slack
Powered by
related to my previous question on having flows ac...
# ask-metaflow
c
related to my previous question on having flows across AWS regions sharing the same metadata service, I am now using the metaflow metadata client to retrieve flow information and running into a strange bug: with the same metaflow config and version, same boto3 and aws config, one computer is able to retrieve artifacts from a different region, while another computer times-out with
Copy code
An error occurred (IllegalLocationConstraintException) when calling the GetObject operation: The x-central-1 location constraint is incompatible for the region specific endpoint this request was sent to.
✅ 1
i have checked metaflow and boto3 versions match, aws and metaflow config files are the same (
sort | md5sum
), and there are no AWS or METAFLOW environment variables present
the mwe.py is simply
Copy code
from metaflow import Run, namespace
from metaflow.plugins import aws
import metaflow
import boto3

print('metaflow version', metaflow.__version__)
print('boto3 version', boto3.__version__)
print('s3 endpoint', aws.aws_client.get_aws_client('s3').meta.endpoint_url)

namespace(None)

run = Run('PermissionValidationFlow/20074')
print(list(list(run.steps())[::-1][0].tasks())[0].data._artifacts['region'].data)
on computer 029:
Copy code
$ python mwe.py
metaflow version 2.8.0
boto3 version 1.24.35
s3 endpoint <https://s3.ap-southeast-2.amazonaws.com>
GBR
on computer 022:
Copy code
$ python mwe.py
metaflow version 2.8.0
boto3 version 1.24.35
s3 endpoint <https://s3.ap-southeast-2.amazonaws.com>
...
Error: An error occurred (IllegalLocationConstraintException) when calling the GetObject operation: The x-central-1 location constraint is incompatible for the region specific endpoint this request was sent to.
at the same time, using 
aws s3 ls
on the target bucket on all machines fail with the same 
IllegalLocationConstraintException
error, so it seems the "working" behaviour is the anomaly, not the failure case.
though if i can make it work everywhere it would be much nicer!
on 022, the "magic" computer, the mwe.py can be updated to retrieve artifacts from runs across all regions without any special effort
a
interesting. what happens if you set the 
AWS_DEFAULT_REGION
to the region of the bucket?
c
interestingly, it just started to work without effort on my part
this normally happens to other people... i am not sure what just happened @ancient-application-36103, i didn't change the mwe.py (in fact i copied it from the slack message to be sure), and
Copy code
$ env | grep AWS | wc -l
0
but now
Copy code
$ python mwe.py
metaflow version 2.8.0
boto3 version 1.24.35
s3 endpoint <https://s3.ap-southeast-2.amazonaws.com>
GBR
works on computer 022
@ancient-application-36103 let's say you fixed it by looking... for now.... xD
ooooh, it is the client cache
i happen to pick flows for testing that had its metadata cached from previous dev work, so it "appears" to work on 029 b/c it never needed to consult S3
purging the client cache on both 022 and 029 brings consistent behaviour now: not being able to access artifacts stored in a different region
which is a bummer: our internal dashboard (written before metaflow's own) only works now for flows executed in the same region. It sees flows from other regions, but is unable to access their data
a
I would love to understand this issue a bit more deeply. would you be available in another 30 mins for a quick chat?
c
sure thing @ancient-application-36103
hit me up in my DMs
a
will do after my current meeting..
c
@ancient-application-36103 quick update, I managed got my dashboard to work reasonably well flows from any known region without modifying metaflow code by: 1. set METAFLOW_S3_RETRY_COUNT to 0 so I know immediately if a flow is in a different region 2. after instantiating a Run/Step/Task object, access some property that requires S3 artifacts to be fetched, e.g. 
task.stderr
3. if (2) fails due to incorrect region, try again with the next region 4. otherwise return the object Not ideal but for an internal dashboard the UX is fast enough (regions ordered by usage) that no one is complaining yet :)
among us party 1
3 Views
Open in Slack
PreviousNext
Powered by