Hello everyone. Maybe this is a newbie question. I just set up my metaflow to connect to AWS Batch, and now it is running fine using AWS Batch. However, I found out that even if I run the flow without the

--with batch

flag it still runs on Batch? That is, the following command will try to run on Batch.

python kmeans_flow_v1.py run

How do I run my flow locally, even though metaflow is connected to batch? Appreciate any help here!

this idea might help you

@fast-jewelry-98070 - do you have an

@batch

decorator on the steps within your flow? that could be the reason why your step is running on AWS Batch.

--with batch

is syntactic sugar that applies

@batch

decorator to all your steps in the flow.

I actually didn't annotate any steps with

@batch

decorator, and it still runs on batch without

--with batch

can you paste the logs that you see in the terminal here?

The code:

from metaflow import FlowSpec, step, Parameter


class KmeansFlow(FlowSpec):
    num_docs = Parameter('num-docs', help='Number of documents to load', default=1000)

    @step
    def start(self):
        import scale_data
        scale_data.load_yelp_reviews(self.num_docs)
        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == '__main__':
    KmeansFlow()

command line:

(ml-engineer-python) wenjunsun@Wenjuns-MacBook-Air chapter5 % python kmeans_flow_v1.py run
Metaflow 2.8.5 executing KmeansFlow for user:wenjunsun
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
2023-05-16 19:25:08.302 Workflow starting (run-id 3):
2023-05-16 19:25:09.476 [3/start/8 (pid 25683)] Task is starting.
2023-05-16 19:25:30.250 [3/start/8 (pid 25683)] Task finished successfully.
2023-05-16 19:25:32.074 [3/end/9 (pid 26015)] Task is starting.
2023-05-16 19:25:38.519 [3/end/9 (pid 26015)] Task finished successfully.
2023-05-16 19:25:39.019 Done!

Actually I think I am mistaken. It seems like it isn't running on batch without

--with batch

. Previously I was on VPN so running the above command gives

requests.exceptions.SSLError: HTTPSConnectionPool(host='<http://ff8msu7xfa.execute-api.us-east-1.amazonaws.com|ff8msu7xfa.execute-api.us-east-1.amazonaws.com>', port=443): Max retries exceeded with url: /api/flows/KmeansFlow (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))

which makes me think it is trying to run it on batch, which I guess it isn't?