Hi everyone, How can we define a separate and independent environment for metaflow steps using pip? I am using this pip decorator (https://github.com/Netflix/metaflow/issues/24#issuecomment-571976372 ) to define environments for steps, but instead of generating an independent environment for each step it just overwrites the existing environment.

For ex, in this sample code in the start step the ruamel.yaml version is 0.17.19, in step process_message I install ruamel.yaml 0.17.15 and in step end it stays 0.17.15 instead of going back to 0.17.19.

class LinearFlow(FlowSpec):
    @step
    def start(self):
        self.message = 'Thanks for reading.'
        pkgs = freeze.freeze()
        for pkg in pkgs:
            if 'yaml' in str(pkg):
                print(pkg)

        self.next(self.process_message)

    @pip(libraries={'ruamel.yaml':'0.17.15'})
    @step
    def process_message(self):
        print('the message is: %s' % self.message)
        pkgs = freeze.freeze()
        for pkg in pkgs:
            if 'yaml' in str(pkg):
                print(pkg)
        self.next(self.end)

    @step
    def end(self):
        pkgs = freeze.freeze()
        for pkg in pkgs:
            if 'yaml' in str(pkg):
                print(pkg)
        print('the message is still: %s' % self.message)

output:

2023-03-17 20:43:24.495 Workflow starting (run-id 1679085804401214):
2023-03-17 20:43:24.540 [1679085804401214/start/1 (pid 19437)] Task is starting.
2023-03-17 20:43:26.363 [1679085804401214/start/1 (pid 19437)] ruamel.yaml==0.17.19
2023-03-17 20:43:26.516 [1679085804401214/start/1 (pid 19437)] ruamel.yaml.clib==0.2.7
2023-03-17 20:43:26.526 [1679085804401214/start/1 (pid 19437)] Task finished successfully.
2023-03-17 20:43:26.581 [1679085804401214/process_message/2 (pid 19455)] Task is starting.
2023-03-17 20:43:28.051 [1679085804401214/process_message/2 (pid 19455)] 03/17/2023 20:43:28 - INFO - decorators -   pip-install ruamel.yaml:  with version 0.17.15
2023-03-17 20:43:30.225 [1679085804401214/process_message/2 (pid 19455)] Collecting ruamel.yaml==0.17.15
2023-03-17 20:43:30.299 [1679085804401214/process_message/2 (pid 19455)] Downloading ruamel.yaml-0.17.15-py3-none-any.whl (108 kB)
2023-03-17 20:43:31.076 [1679085804401214/process_message/2 (pid 19455)] Requirement already satisfied: ruamel.yaml.clib>=0.1.2 in /fsx/jay/workspace/metis_eval/removetmux/metis/env/lib/python3.6/site-packages (from ruamel.yaml==0.17.15) (0.2.7)
2023-03-17 20:43:31.542 [1679085804401214/process_message/2 (pid 19455)] Installing collected packages: ruamel.yaml
2023-03-17 20:43:31.542 [1679085804401214/process_message/2 (pid 19455)] Attempting uninstall: ruamel.yaml
2023-03-17 20:43:31.542 [1679085804401214/process_message/2 (pid 19455)] Found existing installation: ruamel.yaml 0.17.19
2023-03-17 20:43:31.672 [1679085804401214/process_message/2 (pid 19455)] Uninstalling ruamel.yaml-0.17.19:
2023-03-17 20:43:36.769 [1679085804401214/process_message/2 (pid 19455)] Successfully uninstalled ruamel.yaml-0.17.19
2023-03-17 20:43:38.613 [1679085804401214/process_message/2 (pid 19455)] Successfully installed ruamel.yaml-0.17.15
2023-03-17 20:43:38.682 [1679085804401214/process_message/2 (pid 19455)] the message is: Thanks for reading.
2023-03-17 20:43:39.024 [1679085804401214/process_message/2 (pid 19455)] ruamel.yaml==0.17.15
2023-03-17 20:43:39.164 [1679085804401214/process_message/2 (pid 19455)] ruamel.yaml.clib==0.2.7
2023-03-17 20:43:39.173 [1679085804401214/process_message/2 (pid 19455)] Task finished successfully.
2023-03-17 20:43:39.221 [1679085804401214/end/3 (pid 19518)] Task is starting.
2023-03-17 20:43:40.963 [1679085804401214/end/3 (pid 19518)] ruamel.yaml==0.17.15
2023-03-17 20:43:40.965 [1679085804401214/end/3 (pid 19518)] ruamel.yaml.clib==0.2.7
2023-03-17 20:43:40.965 [1679085804401214/end/3 (pid 19518)] the message is still: Thanks for reading.
2023-03-17 20:43:41.118 [1679085804401214/end/3 (pid 19518)] Task finished successfully.
2023-03-17 20:43:41.126 Done!