https://outerbounds.com/ logo
Join Slack
Powered by
Hello all, There's a bug in Metaflow `2.12.37` . ...
# ask-metaflow
a
Hello all, There's a bug in Metaflow 
2.12.37
. Here's the error when I try to deploy and trigger a flow on Step Functions:
Copy code
Traceback (most recent call last):
  File "/opt/conda/lib/python3.12/site-packages/metaflow/cli.py", line 554, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/opt/conda/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/metaflow/cli_components/utils.py", line 69, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/metaflow/plugins/aws/step_functions/step_functions_cli.py", line 194, in create
    flow = make_flow(
           ^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/metaflow/plugins/aws/step_functions/step_functions_cli.py", line 329, in make_flow
    decorators._init(obj.flow, only_non_static=True)
  File "/opt/conda/lib/python3.12/site-packages/metaflow/decorators.py", line 561, in _init
    deco.init()
  File "/opt/conda/lib/python3.12/site-packages/metaflow/plugins/pypi/conda_decorator.py", line 71, in init
    self.attributes["libraries"],
      ~~~~~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'libraries'
Downgrading to Metaflow 
2.12.33
fixed the issue
👀 1
✅ 1
a
Does downgrading to 2.12.36 help?
d
We found the issue. Will be issuing a fix shortly
The fix is here: https://github.com/Netflix/metaflow/pull/2167. I’ll be merging once tests pass. Feel free to try that version out as well. 2.12.36 should also work and there is no feature change (user facing) between 2.12.36 and 2.12.37 so you should alos be able to use that version.
thankyou 1
a
Thanks a bunch!
d
No prob. Sorry for missing that. Let me know if you face more issues with 2.12.37+fix. It’s paving the way for some pretty cool things :).
🙏 1
no problem 1
2.12.38 is on its way out with the fix.
thankyou 1
s
Maybe an issue related to this, in 2.12.38 :
Copy code
Metaflow 2.12.38 executing CarbonQuantificationFlow for user:gitlab
Project: carbon_quantification, Branch: test.test_v1_1_0
  Internal error
Traceback (most recent call last):
 File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/metaflow/cli.py", line 554, in main
  start(auto_envvar_prefix="METAFLOW", obj=state)
 File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
  return self.main(args, kwargs)
 File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 782, in main
  rv = self.invoke(ctx)
 File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/metaflow/cli_components/utils.py", line 65, in invoke
  click.Command.invoke(self, ctx)
 File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
  return ctx.invoke(self.callback, ctx.params)
 File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
  return callback(args, kwargs)
 File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/metaflow/tracing/__init__.py", line 27, in wrapper_func
  return func(args, kwargs)
 File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/metaflow/_vendor/click/decorators.py", line 21, in new_func
  return f(get_current_context(), args, kwargs)
 File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/metaflow/cli.py", line 468, in start
  decorators._init_step_decorators(
 File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/metaflow/decorators.py", line 614, in _init_step_decorators
  deco.step_init(
 File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/metaflow/plugins/pypi/pypi_decorator.py", line 45, in step_init
  self._user_defined_attributes = ***
TypeError: 'set' object is not a mapping
To be more specific, we saw that 2.12.37 brought new changes when developing custom decorators, something that we do, but the specific changes that you did do not seem to apply to our case. But we have a config for SFN that is like this : echo $METAFLOW_AWS_CONFIG >~/.metaflowconfig/config.json And in the variable :
Copy code
{
  "METAFLOW_BATCH_JOB_QUEUE": "job-queue-XXXXX",
  "METAFLOW_DATASTORE_SYSROOT_S3": "<s3://anonymous-bucket/metaflow>",
  "METAFLOW_DATATOOLS_S3ROOT": "<s3://anonymous-bucket/metaflow/data>",
  "METAFLOW_DEFAULT_DATASTORE": "s3",
  "METAFLOW_DEFAULT_METADATA": "service",
  "METAFLOW_ECS_S3_ACCESS_IAM_ROLE": "arn:aws:iam::111111111111:role/AnonymousS3TaskRole",
  "METAFLOW_EVENTS_SFN_ACCESS_IAM_ROLE": "arn:aws:iam::111111111111:role/AnonymousEventBridgeRole",
  "METAFLOW_SERVICE_AUTH_KEY": "xxx",
  "METAFLOW_SERVICE_INTERNAL_URL": "<http://internal-service-url/>",
  "METAFLOW_SERVICE_URL": "<https://external-service-url/api/>",
  "METAFLOW_SFN_DYNAMO_DB_TABLE": "AnonymousDynamoDBTable",
  "METAFLOW_SFN_IAM_ROLE": "arn:aws:iam::111111111111:role/AnonymousStepFunctionsRole",
  "METAFLOW_ECS_FARGATE_EXECUTION_ROLE": "arn:aws:iam::111111111111:role/AnonymousECSTaskExecutionRole",
  "METAFLOW_CONDA_DEPENDENCY_RESOLVER": "micromamba",
  "METAFLOW_DEFAULT_SECRETS_BACKEND_TYPE": "aws-secrets-manager"
}
t
still verifying some bits n pieces before this can go in, but https://github.com/Netflix/metaflow/pull/2168 should fix your issue @stocky-fountain-57774
🙌 1
3 Views
Open in Slack
PreviousNext
Powered by