https://outerbounds.com/ logo
Join Slack
Powered by
Hi, I am hitting some problems in aws batch. The ...
# ask-metaflow
p
Hi, I am hitting some problems in aws batch. The batch Job says its state is RUNNING, but nothing happens (no further logs, CPU usage is zero %) Metaflow reports that the task has failed. I see logs from the pip decorator we use before each step and then nothing. Any idea how to debug? Basically the error 'seems' to happen (more?) when using step functions. This is a parallel step so perhaps 30 out of 100 steps succeeded. Doing another run, I notice that Metaflow reports runs failed even though they later succeed (eg when i refresh the ui)
👋 2
a
Hi @plain-baker-2104 - some of the state tracking in Metaflow UI is eventually consistent (when running on Step Functions). This happens when the time between a step finishing and next step beginning exceeds a certain threshold (because the job is waiting in a queue etc).
👋 1
regarding no logs if you use 
@pip
- can you share your implementation of 
@pip
?
p
Copy code
def pip(libraries):
    """Work around to use pip in metaflow steps,
    see here: <https://github.com/Netflix/metaflow/issues/24>"""

    def decorator(function):
        @functools.wraps(function)
        def wrapper(*args, **kwargs):
            import subprocess
            import sys

            for library, version in libraries.items():
                print("Pip Install:", library, version)
                subprocess.run(
                    [
                        sys.executable,
                        "-m",
                        "pip",
                        "install",
                        # "--quiet",
                        library + "==" + version,
                    ]
                )
            return function(*args, **kwargs)

        return wrapper

    return decorator
So when I say no logs, I mean that there appears no logs after Pip has installed the specified libraries. So having commented out 
--quiet
, the last log line I see is 
Successfully installed XXX
. [and I have added a log step at the start of my start step] which is never output
I am currently ssh'd into the instance. any suggestions what `i should look at?
a
what's the output of 
ps -ef | grep metaflow
?
f
might also be useful to cross-reference the cloudwatch logs from the batch jobs
p
cloudwatch (app) logs don't show anything after the pip install
f
is the behavior consistent when running locally vs on batch? I'd also caution a bit about something like 
@pip
in combination with a 
foreach
fanout, as you might be spamming pypi servers or be downloading more than intended each time depending on the package (e.g. large libs or pretrained models), none of which will be persisted/versioned by metaflow or in the base python environment in the docker image. also if you have both 
@conda
and 
@pip
decorators, this decorator needs to be placed after the 
@conda
decorator to work correctly.
p
I don't run these big parallel runs locally. I only run on step functions, and then when they fail I resume them in batch from by terminal. I haven't observed this behaviour locally. nor is it consistent - some fraction of runs work. I don't really believe its to do with 
pip
. we have been using it without problems for ? 2 years. I was assuming its related to a silent fail in the initialisation of the metaflow step.
7 Views
Open in Slack
PreviousNext
Powered by