https://outerbounds.com/ logo
Join Slack
Powered by
Hi all! I am trying to run the 05-helloaws/helloaw...
# ask-metaflow
e
Hi all! I am trying to run the 05-helloaws/helloaws.py tutorial but I am getting:
Copy code
Metaflow 2.6.3 executing HelloAWSFlow for user:sderosa
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
    S3 access failed:
    S3 operation failed.
    Key requested: <s3://bag-replay/metaflow/HelloAWSFlow/1673376143152351/_parameters/0/0.attempt.json>
    Error: Unable to locate credentials
However, i am able to upload/download objects from the 
<s3://bag-replay/metaflow/>
directory Update: I actually get the same error when running the helloworld tutorial 
00-helloworld/helloworld.py
✅ 1
b
Do you have 
METAFLOW_DATASTORE_SYSROOT_S3
METAFLOW_DATATOOLS_S3ROOT
, and 
METAFLOW_DEFAULT_DATASTORE
defined in a 
.metaflowconfig/config.json
file?
e
yes i do
and FWIW i am able to trigger aws batch jobs via aws cli
alright, i just hard coded some values in my 
~/.aws/credentials
file which are universal for our aws account (i didnt need to do this yesterday, not sure what happened)
so now i am able to run the 
00-helloworld
tutorial again (woo hoo)
but now, when trying to run the 
05-helloaws
tutorial i get :
Copy code
> python helloaws.py run
Metaflow 2.6.3 executing HelloAWSFlow for user:sderosa
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2023-01-10 14:05:52.161 Workflow starting (run-id 1071):
2023-01-10 14:05:53.043 [1071/start/8100 (pid 118683)] Task is starting.
2023-01-10 14:05:54.787 [1071/start/8100 (pid 118683)] HelloAWS is starting.
2023-01-10 14:05:56.038 [1071/start/8100 (pid 118683)] 
2023-01-10 14:05:56.038 [1071/start/8100 (pid 118683)] Using metadata provider: service@https://bg758e3hbg.execute-api.us-east-2.amazonaws.com/api/
2023-01-10 14:05:56.038 [1071/start/8100 (pid 118683)] 
2023-01-10 14:05:56.038 [1071/start/8100 (pid 118683)] The start step is running locally. Next, the
2023-01-10 14:05:56.038 [1071/start/8100 (pid 118683)] 'hello' step will run remotely on AWS batch.
2023-01-10 14:05:56.038 [1071/start/8100 (pid 118683)] If you are running in the Netflix sandbox,
2023-01-10 14:05:56.039 [1071/start/8100 (pid 118683)] it may take some time to acquire a compute resource.
2023-01-10 14:05:56.660 [1071/start/8100 (pid 118683)] Task finished successfully.
2023-01-10 14:05:57.566 [1071/hello/8101 (pid 118743)] Task is starting.
2023-01-10 14:05:58.485 [1071/hello/8101 (pid 118743)] Traceback (most recent call last):
2023-01-10 14:05:58.486 [1071/hello/8101 (pid 118743)] File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/metaflow/plugins/aws/batch/batch_cli.py", line 275, in step
2023-01-10 14:05:58.487 [1071/hello/8101 (pid 118743)] batch.launch_job(
2023-01-10 14:05:58.487 [1071/hello/8101 (pid 118743)] File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/metaflow/plugins/aws/batch/batch.py", line 324, in launch_job
2023-01-10 14:05:58.488 [1071/hello/8101 (pid 118743)] self.job = job.execute()
2023-01-10 14:05:58.488 [1071/hello/8101 (pid 118743)] File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/metaflow/plugins/aws/batch/batch_client.py", line 138, in execute
2023-01-10 14:05:58.488 [1071/hello/8101 (pid 118743)] response = self._client.submit_job(**self.payload)
2023-01-10 14:05:58.489 [1071/hello/8101 (pid 118743)] File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/client.py", line 320, in _api_call
2023-01-10 14:05:58.489 [1071/hello/8101 (pid 118743)] return self._make_api_call(operation_name, kwargs)
2023-01-10 14:05:58.490 [1071/hello/8101 (pid 118743)] File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/client.py", line 595, in _make_api_call
2023-01-10 14:05:58.490 [1071/hello/8101 (pid 118743)] request_dict = self._convert_to_request_dict(
2023-01-10 14:05:58.490 [1071/hello/8101 (pid 118743)] File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/client.py", line 631, in _convert_to_request_dict
2023-01-10 14:05:58.490 [1071/hello/8101 (pid 118743)] request_dict = self._serializer.serialize_to_request(
2023-01-10 14:05:58.490 [1071/hello/8101 (pid 118743)] File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/validate.py", line 291, in serialize_to_request
2023-01-10 14:05:58.490 [1071/hello/8101 (pid 118743)] raise ParamValidationError(report=report.generate_report())
2023-01-10 14:05:58.491 [1071/hello/8101 (pid 118743)] botocore.exceptions.ParamValidationError: Parameter validation failed:
2023-01-10 14:05:58.545 [1071/hello/8101 (pid 118743)] Unknown parameter in containerOverrides: "resourceRequirements", must be one of: vcpus, memory, command, environment
2023-01-10 14:05:58.647 [1071/hello/8101 (pid 118743)] Task failed.
2023-01-10 14:05:58.829 This failed task will not be retried.
    Internal error:
    The end step was not successful by the end of flow.
so sad 1
im not sure why I'd be getting 
unkown parameter in containerOverrides: "resourceRequirements"
when im just running the script as is, no extra params or custom changes
b
Can you confirm your 
metaflow
version?
Copy code
❯ metaflow help
Metaflow (2.7.12)
I see from your log above that you are on 
2.6.3
I would recommend first upgrading metaflow if that is possible - 
pip install --upgrade metaflow
e
alrighty i upgraded to 
2.7.18
, still seeing the most recent issue i shared above
b
Is it possible that your AWS account does not have Batch setup?
Also is 
AWS_DEFAULT_REGION
set to the right region?
e
im pretty sure its all setup correct, i am able to successfully trigger jobs using the 
aws batch submit-job
cli without any issues
b
what do you see for 
echo $AWS_DEFAULT_REGION
?
e
Copy code
> echo $AWS_DEFAULT_REGION
us-east-2
👍🏽 1
b
Can you share the output when running with the new version of metaflow again?
e
Copy code
> python helloaws.py run           
Metaflow 2.7.18 executing HelloAWSFlow for user:sderosa
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2023-01-10 14:31:50.492 Workflow starting (run-id 1079):
2023-01-10 14:31:51.278 [1079/start/8121 (pid 126083)] Task is starting.
2023-01-10 14:31:53.199 [1079/start/8121 (pid 126083)] HelloAWS is starting.
2023-01-10 14:31:53.260 [1079/start/8121 (pid 126083)] 
2023-01-10 14:31:53.260 [1079/start/8121 (pid 126083)] Using metadata provider: service@https://bg758e3hbg.execute-api.us-east-2.amazonaws.com/api
2023-01-10 14:31:54.411 [1079/start/8121 (pid 126083)] 
2023-01-10 14:31:54.411 [1079/start/8121 (pid 126083)] The start step is running locally. Next, the
2023-01-10 14:31:54.411 [1079/start/8121 (pid 126083)] 'hello' step will run remotely on AWS batch.
2023-01-10 14:31:54.411 [1079/start/8121 (pid 126083)] If you are running in the Netflix sandbox,
2023-01-10 14:31:54.411 [1079/start/8121 (pid 126083)] it may take some time to acquire a compute resource.
2023-01-10 14:31:55.095 [1079/start/8121 (pid 126083)] Task finished successfully.
2023-01-10 14:31:56.021 [1079/hello/8122 (pid 126114)] Task is starting.
2023-01-10 14:31:56.950 [1079/hello/8122 (pid 126114)] Traceback (most recent call last):
2023-01-10 14:31:56.951 [1079/hello/8122 (pid 126114)] File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/metaflow/plugins/aws/batch/batch_cli.py", line 275, in step
2023-01-10 14:31:56.952 [1079/hello/8122 (pid 126114)] batch.launch_job(
2023-01-10 14:31:56.952 [1079/hello/8122 (pid 126114)] File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/metaflow/plugins/aws/batch/batch.py", line 322, in launch_job
2023-01-10 14:31:56.952 [1079/hello/8122 (pid 126114)] self.job = job.execute()
2023-01-10 14:31:56.953 [1079/hello/8122 (pid 126114)] File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/metaflow/plugins/aws/batch/batch_client.py", line 138, in execute
2023-01-10 14:31:56.953 [1079/hello/8122 (pid 126114)] response = self._client.submit_job(**self.payload)
2023-01-10 14:31:56.953 [1079/hello/8122 (pid 126114)] File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/client.py", line 320, in _api_call
2023-01-10 14:31:56.954 [1079/hello/8122 (pid 126114)] return self._make_api_call(operation_name, kwargs)
2023-01-10 14:31:56.954 [1079/hello/8122 (pid 126114)] File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/client.py", line 595, in _make_api_call
2023-01-10 14:31:56.954 [1079/hello/8122 (pid 126114)] request_dict = self._convert_to_request_dict(
2023-01-10 14:31:56.955 [1079/hello/8122 (pid 126114)] File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/client.py", line 631, in _convert_to_request_dict
2023-01-10 14:31:56.955 [1079/hello/8122 (pid 126114)] request_dict = self._serializer.serialize_to_request(
2023-01-10 14:31:56.955 [1079/hello/8122 (pid 126114)] File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/validate.py", line 291, in serialize_to_request
2023-01-10 14:31:56.956 [1079/hello/8122 (pid 126114)] raise ParamValidationError(report=report.generate_report())
2023-01-10 14:31:56.956 [1079/hello/8122 (pid 126114)] botocore.exceptions.ParamValidationError: Parameter validation failed:
2023-01-10 14:31:57.008 [1079/hello/8122 (pid 126114)] Unknown parameter in containerOverrides: "resourceRequirements", must be one of: vcpus, memory, command, environment
2023-01-10 14:31:57.117 [1079/hello/8122 (pid 126114)] Task failed.
2023-01-10 14:31:57.302 This failed task will not be retried.
    Internal error:
    The end step was not successful by the end of flow.
b
Can you share the listing of 
/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/metaflow/plugins/aws/batch/batch_cli.py
?
particularly lines 173-188
a
it is possible that you are running a very old version of boto3?
e
173-178:
Copy code
def echo(msg, stream="stderr", batch_id=None):
        msg = util.to_unicode(msg)
        if batch_id:
            msg = "[%s] %s" % (batch_id, msg)
        ctx.obj.echo_always(msg, err=(stream == sys.stderr))
i just upgraded 
boto3
, now metaflow cant find it.
Copy code
> pip show boto3   
Name: boto3
Version: 1.26.47
Summary: The AWS SDK for Python
Home-page: <https://github.com/boto/boto3>
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages
Requires: s3transfer, jmespath, botocore
Required-by: metaflow

 ~/external-repos/metaflow/metaflow/tutorials/05-helloaws  master !3 ?2 .................................................................................................................................................................................. 3s  15:12:21 
> python helloaws.py run
Metaflow 2.7.18 executing HelloAWSFlow for user:sderosa
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
    Flow failed:
    You need to install 'boto3' in order to use S3.
a
can you do 
python -m pip install -U boto3
e
yeah that didnt do anything, just says requirements are already satisfied. I tried uninstalling then reinstalling, and I get the same result
ok maybe there is something with how my metaflow cache ? trying to run the 
00-hellworld
flow, i get
Copy code
~/external-autoyard-repos/metaflow/metaflow/tutorials/00-helloworld  master !3 ?2 .................................................................................................................................................................................... 15:17:55 
> python helloworld.py run
Metaflow 2.7.18 executing HelloFlow for user:sderosa
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
    Flow failed:
    You need to install 'boto3' in order to use S3.
despite there not being any reference to 
s3
in that flow tutorial
a
can you share what your metaflow config looks like? It seems that your backing datastore is set to 
s3
but metaflow is unable to locate 
boto3
for communicating with AWS
e
Copy code
{
  "METAFLOW_DEFAULT_DATASTORE": "s3",
  "METAFLOW_DEFAULT_METADATA": "service",
  "METAFLOW_DATASTORE_SYSROOT_S3": "<s3://pcvml.data.metaflow.dev/metaflow>",
  "METAFLOW_DATATOOLS_S3ROOT": "s3://.pcvml.data.metaflow.dev/data",
  "METAFLOW_BATCH_JOB_QUEUE": "arn:aws:batch:us-east-2:043394610237:job-queue/pcvml-metaflow-fargate-dev",
  "METAFLOW_BATCH_INF_JOB_QUEUE": "arn:aws:batch:us-east-2:043394610237:job-queue/pcvml-metaflow-inference-dev",
  "METAFLOW_ECS_S3_ACCESS_IAM_ROLE": "arn:aws:iam::043394610237:role/pcvml-metaflow-batch-job-dev",
  "METAFLOW_SERVICE_URL": "<https://bg758e3hbg.execute-api.us-east-2.amazonaws.com/api/>",
  "METAFLOW_SERVICE_INTERNAL_URL": "<http://nlb-fd4375d-c0ba4b930f24ea57.elb.us-east-2.amazonaws.com/>",
  "METAFLOW_EVENTS_SFN_ACCESS_IAM_ROLE": "arn:aws:iam::043394610237:role/event-bridge-role-174cfbf",
  "METAFLOW_SFN_IAM_ROLE": "arn:aws:iam::043394610237:role/step-function-role-cd2ac4a",
  "METAFLOW_SFN_DYNAMO_DB_TABLE": "stepfunctions-state-b24f9fd",
  "METAFLOW_ECS_FARGATE_EXECUTION_ROLE": "arn:aws:iam::043394610237:role/ecs-task-execution-role-346fe87",
  "METAFLOW_UI_URL": "<http://metaflow-ui-alb-dev-1170540126.us-east-2.elb.amazonaws.com:8083>"
}
a
Copy code
METAFLOW_DEFAULT_DATASTORE": "s3"
indicates that Metaflow artifacts be logged in s3
e
ah ok, so thats why a flow that doesnt access s3 needs connection ?
a
the flows accesses s3 in the sense that the data artifacts generated by the flow live in s3
even though you might not be accessing anything in s3 directly within your user code
you can set 
METAFLOW_DEFAULT_DATASTORE": "local"
if you don't want to stash data artifacts in s3
it seems that something in your python set up is preventing successful lookup of boto3 package
what is the output of
Copy code
python -c "import boto3; print(boto3.__version__)"`
e
Copy code
> python -c "import boto3; print(boto3.__version__)" 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/boto3/__init__.py", line 17, in <module>
    from boto3.session import Session
  File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/session.py", line 26, in <module>
    import botocore.client
  File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/client.py", line 15, in <module>
    from botocore import waiter, xform_name
  File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/waiter.py", line 18, in <module>
    from botocore.docs.docstring import WaiterDocstring
  File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/docs/__init__.py", line 15, in <module>
    from botocore.docs.service import ServiceDocumenter
  File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/docs/service.py", line 14, in <module>
    from botocore.docs.client import ClientDocumenter, ClientExceptionsDocumenter
  File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/docs/client.py", line 14, in <module>
    from botocore.docs.example import ResponseExampleDocumenter
  File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/docs/example.py", line 13, in <module>
    from botocore.docs.shape import ShapeDocumenter
  File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/docs/shape.py", line 19, in <module>
    from botocore.utils import is_json_value_header
  File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/utils.py", line 37, in <module>
    import botocore.httpsession
  File "/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/httpsession.py", line 22, in <module>
    from urllib3.util.ssl_ import (
ImportError: cannot import name 'PROTOCOL_TLS' from 'urllib3.util.ssl_' (/home/sderosa/.pyenv/versions/3.8.10/lib/python3.8/site-packages/urllib3/util/ssl_.py)
yeah unfortunately i think you're right
im going to just create a new env
a
that would definitely be the path of least resistance 🙂
e
alright, ill ping back if i run into issues.
a
sounds good
e
thanks @square-wire-39606 and @brave-lion-15961 for your help!
2 Views
Open in Slack
PreviousNext
Powered by