https://outerbounds.com/ logo
Join Slack
Powered by
hey so trying to get secrets to work in AWS batch ...
# ask-metaflow
f
hey so trying to get secrets to work in AWS batch using the command line 
with
so that different people on the team can have different creds. 🧵
✅ 1
my command line:
Copy code
poetry run python classifier/pull_data_flow.py --environment=pypi –-with 'secrets:sources=["dev/mmr-snowflake"]' run
my error:
Copy code
Using python3 (3.11.7)
Usage: pull_data_flow.py [OPTIONS] COMMAND [ARGS]...
Try 'pull_data_flow.py --help' for help.

Error: No such command '–-with'.
as I read the blog post, the above should work, right? https://outerbounds.com/blog/metaflow-secrets/
a
correct - it should. what does pull_data_flow.py look like?
f
hrm, not sure how much of that I can post on this channel
not sure how relevant that is though, since it appears to be an error with the parsing of the command line itself
a
okay, so no enhancements to 
__name__=="__main__":
bits?
f
Copy code
if __name__ == "__main__":
    ClassificationPreprocessing()
which... 
class ClassificationPreprocessing(FlowSpec):
Copy code
@secrets(sources=['dev/mmr-snowflake'])
    @batch(image='python:3.10.13-slim', cpu=16, memory=2**16, gpu=0)
    @conda(packages=PREPROCESSING_PACKAGES, python=PYTHON_VERSION)
    @step
    def pull_ocr(self):
        import pandas as pd
        import ast
        from shared.execute_sql import SnowflakeOAuthQuery, SqlArgs
a
cool - we are checking internally - that command isn't working for a helloflow either - likely has to do with escaping
f
I also tried 
@environment(SNOWFLAKE_INFO)
but that just straight up failed, not sure how to structure env vars
ah, the slash there?
a
@environment(vars=SNOWFLAKEINFO)
should work
f
ah!
s
hi @faint-zoo-41552 what metaflow version are you using?
f
2.12.4
s
@faint-zoo-41552 do you have the @secrets decorator on the pull_ocr step as well?
in addition to passing it via the --with secrets?
f
no
I comment that out when I try to use the with command
👍🏽 1
s
if you put it in the flow and run the flow are you able to access the secret?
f
no
that's giving me some other shenanigans that I think might be related to batch, something about not being able to copy
trying to repro locally
s
maybe try the following flow:
Copy code
from metaflow import FlowSpec, step
import os


class HelloWorldFlow(FlowSpec):
    @step
    def start(self):
        for name, value in os.environ.items():
            print("{0}: {1}".format(name, value))
        self.next(self.end)
    
    
    @step
    def end(self):
        pass

if __name__ == '__main__':
    HelloWorldFlow()
and run it like :
Copy code
python helloflow.py --with 'secrets:sources=["'REPLACE_SECRET_HERE'"]' run
f
Copy code
mark@mmr-laptop:~/src/HypHub/models/lease_sales_classifier$ poetry run python helloflow.py --with 'secrets:sources=["'REPLACE_SECRET_HERE'"]' run
The currently activated Python version 3.12.1 is not supported by the project (~3.11).
Trying to find and use a compatible version. 
Using python3 (3.11.7)
Metaflow 2.12.4 executing HelloWorldFlow for user:mark
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2024-06-20 16:27:43.688 Workflow starting (run-id 1870):
2024-06-20 16:27:47.653 [1870/start/11341 (pid 936231)] Task is starting.
2024-06-20 16:27:49.748 [1870/start/11341 (pid 936231)] <flow HelloWorldFlow step start> failed:
2024-06-20 16:27:51.493 [1870/start/11341 (pid 936231)] Flow failed:
2024-06-20 16:27:51.620 [1870/start/11341 (pid 936231)] Failed to retrieve secret 'REPLACE_SECRET_HERE': An error occurred (ResourceNotFoundException) when calling the GetSecretValue operation: Secrets Manager can't find the specified secret.
2024-06-20 16:27:51.620 [1870/start/11341 (pid 936231)] 
2024-06-20 16:27:51.721 [1870/start/11341 (pid 936231)] Task failed.
2024-06-20 16:27:51.873 Workflow failed.
2024-06-20 16:27:51.873 Terminating 0 active tasks...
2024-06-20 16:27:51.873 Flushing logs...
    Step failure:
    Step start (task-id 11341) failed.
so that's an expected failure
that worked
now how do I delete that out of my shared env since I just published my secret
to the metaflow cluster
s
aah.. I dont think that is super easy.. your best bet It might be easier to rotate out the secret
a
@faint-zoo-41552 by publishing the secret, do you mean leaking it out as part of your task logs?
f
well you just had me print it out 🙂
and now that print statement went to the cluster
a
You can scrub the logs using Python flow.py logs scrub
That will wipe the logs from metaflow’s store - but if you are pushing logs anywhere else as well, you will have to hunt those down
f
Copy code
mark@mmr-laptop:~/src/HypHub/models/lease_sales_classifier$ poetry run python helloflow.py logs scrub
The currently activated Python version 3.12.1 is not supported by the project (~3.11).
Trying to find and use a compatible version. 
Using python3 (3.11.7)
Metaflow 2.12.4 executing HelloWorldFlow for user:mark
Usage: helloflow.py logs scrub [OPTIONS] INPUT_PATH
Try 'helloflow.py logs scrub --help' for help.

Error: Missing argument 'INPUT_PATH'.
what's the INPUT_PATH for the cluster logs?
a
run-id/step-name/task-id
f
ok, all redacted, thanks
👍🏼 1
will still rotate because yeah
👍🏼 1
👍🏽 1
ok I think something's wonky with my flow then, I'll try to fix it
a
cool - let us know if you still run into any issues
is this also the same flow?
f
no
we have many problems atm
a
ah - hopefully all easily solvable 🙂
f
hopefully
yeah idk what's the deal with why my flow isn't working, as I add in components it's still all working..
when I try to scrub some other logs, I'm getting this:
Copy code
Metaflow 2.12.4 executing ClassificationPreprocessing for user:mark
Project: extraction, Branch: user.mark
    Invalid command:
    No Tasks found at the given path -- either none exist or they have not finished yet.
    If you know the task has finished, you can supply --include-not-done to force scrub it.
I know that that job exists because I can see it in the cluster
a
what was the input path?
f
the flow is in the classifier directory
from my current pwd
should flows always be run in a root directory?
a
python flow.py logs scrub run-id/step-name/task-id
would be the command
f
yep more pebkac
thanks
7 Views
Open in Slack
PreviousNext
Powered by