https://outerbounds.com/ logo
Join Slack
Powered by
Hello Metaflow friends! I ran the <terraform for M...
# ask-metaflow
f
Hello Metaflow friends! I ran the terraform for Metaflow on GCP and then wanted to try out @kubernetes, so ran tutorial 5. The container on the 
@kubernetes
step failed to start with a 403 (GCS object forbidden). I ran the flow as a user that has permissions to assume the service account identity (and also tried the other method of using the key in 
metaflow_gsa_key_default.json
). The permissions on the service account look correct. When I describe the pod, the service account listed is 
default
. The 
gsa-metaflow-default
service account shows up as default in cloud console. Any ideas on what else to try?
v
hi Julie - good to see you here 🙂 meow wave
👋 1
if it's 403 GCS object forbidden indicates that somehow your user is not able to write objects to your storage bucket
make sure you have run
Copy code
gcloud auth application-default login
you can try to see if you can access GCS by running e.g.
Copy code
gcloud storage ls <gs://your-bucket-name>
I wonder if it gives 403 too?
f
Thanks, @straight-shampoo-11124! No, it doesn't give a 403. But somehow, the "default" service account that runs the container is getting a 403. The roles look right. There's also the ksa which has access to it.
u
To try to isolate the issue - are you able to run without kubernetes? (i.e. no decorators for @kubernetes, and no 
--with=kubernetes
)
u
Also are you able to share the contents of your 
~/.metaflowconfig/config.json
?
f
yes, able to run w/o kubernetes
Copy code
{
  "METAFLOW_DATASTORE_SYSROOT_GS": "<gs://storage>-[REDACTED]-metaflow-default/tf-full-stack-sysroot",
  "METAFLOW_DEFAULT_DATASTORE": "gs",
  "METAFLOW_DEFAULT_METADATA": "service",
  "METAFLOW_SERVICE_INTERNAL_URL": "<http://metadata-service.default:8080/>",
  "METAFLOW_SERVICE_URL": "<http://127.0.0.1:8080/>"
}
I'm using the port forwarding script that came with the GCP terraform templates.
u
Please add these two entries:
u
Copy code
"METAFLOW_KUBERNETES_NAMESPACE": "default",
  "METAFLOW_KUBERNETES_SERVICE_ACCOUNT": "ksa-metaflow",
f
Interesting. I don't know how those entries were removed. I copied the 
config.json
from the gcp/terraform directory after running terraform. But I later ran 
metaflow configure
. I wonder if it was overwritten/removed somehow.
It does work now. Without the 
ksa-metaflow
and the default namespace entry, what namespace/account would be used instead?
thank you for the help!
u
I think the 
default
may already be used without the entry.
u
but 
ksa-metaflow
is key though.
u
(if unset, it will be whatever the default svc acct is for that namespace)
f
Interesting. I'm not sure whether I've uncovered a bug, because I can't be entirely sure if I touched the config file at some point. I do know that I copied it from gcp/terraform and then ran metaflow configure. If that doesn't reproduce the issue, then it could have just been user error.
u
If you copy the file over you should not need to run metaflow configure. Yeah perhaps metaflow configure removed some keys. E.g. it prompts for namespace from the user.
f
possibly. I don't remember what I entered
Following up: I wanted to see what would happen if I ran 
metaflow configure gcp
with an existing config.json. First, it asked me if I wanted to edit an existing config. I said yes. Then, it asked about GCS bucket, metadata service and auth key. It did not prompt me about the kubernetes service account or namespace, and didn't ask about argo either. Those settings were then removed from config.json. Is this expected behavior?
v
do you remember if it showed a message like
Copy code
Final note! Metaflow can scale your flows by executing your steps on Kubernetes.
You may use Google Kubernetes Engine (GKE) or a self-managed Kubernetes cluster on Google Compute Engine VMs. If/when your Kubernetes cluster is ready for use, please run 'metaflow configure kubernetes'.
in the end? It should have. The idea is that you can optionally 
configure kubernetes
separately, since you can use GCP locally without Kubernetes.
f
I don't remember seeing that message. I can try again to see if it shows up.
v
thanks! Let me know if you don't see it - it sounds like a bug if you don't
2 Views
Open in Slack
PreviousNext
Powered by