Hi, we are trying to debug intermittent error we get when trying to create a argo workflow. We have had people in my team who get error like. This doesn't happen all the time but it does happen from time to time.

Metaflow service error
Metadata request failed: 403 - forbidden error

Does it auto-resolve? How frequently does it happen?

Sometimes it happens multiple times in a row for few colleagues. But yeah it auto resolves

can you walk us through how you have deployed this service? we do not expect this issue to happen intermittently.

We have used the terraform provided by outerbound. This is pretty much the terraform code we are using

module "metaflow_datastore" {
  source  = "outerbounds/metaflow/aws//modules/datastore"
  version = "~> 0.10.0"

  force_destroy_s3_bucket = true

  resource_prefix = local.resource_prefix
  resource_suffix = local.resource_suffix

  metadata_service_security_group_id = module.metaflow_metadata_service.metadata_service_security_group_id
  metaflow_vpc_id                    = local.vpc.vpc_id
  standard_tags                      = {}
  subnet1_id                         = local.vpc.private_subnets[0]
  subnet2_id                         = local.vpc.private_subnets[1]
}

module "metaflow_common" {
  source  = "outerbounds/metaflow/aws//modules/common"
  version = "~> 0.10.0"
}


module "metaflow_metadata_service" {
  source  = "outerbounds/metaflow/aws//modules/metadata-service"
  version = "~> 0.10.0"

  resource_prefix = local.resource_prefix
  resource_suffix = local.resource_suffix

  access_list_cidr_blocks          = []
  database_name                    = module.metaflow_datastore.database_name
  database_password                = module.metaflow_datastore.database_password
  database_username                = module.metaflow_datastore.database_username
  datastore_s3_bucket_kms_key_arn  = module.metaflow_datastore.datastore_s3_bucket_kms_key_arn
  enable_api_basic_auth            = true
  fargate_execution_role_arn       = aws_iam_role.ecs_execution_role.arn
  metaflow_vpc_id                  = local.vpc.vpc_id
  metadata_service_container_image = module.metaflow_common.default_metadata_service_container_image
  rds_master_instance_endpoint     = module.metaflow_datastore.rds_master_instance_endpoint
  s3_bucket_arn                    = module.metaflow_datastore.s3_bucket_arn
  standard_tags                    = {}
  subnet1_id                       = local.vpc.private_subnets[0]
  subnet2_id                       = local.vpc.private_subnets[1]
  vpc_cidr_blocks                  = [local.vpc.cidr_block]
  with_public_ip                   = false
}