I am trying to run a metaflow stage in the cloud on a specified image

FULL_IMAGE

. I get the following error:

2023-02-08 11:58:43.479 [1301/run_subsystem_with_bag/9070 (pid 8798)] [26f877a1-b061-4801-9954-86bb440d2ead]   File "/metaflow/bag_evaluator.py", line 17, in evaluate_bag
2023-02-08 11:58:43.479 [1301/run_subsystem_with_bag/9070 (pid 8798)] [26f877a1-b061-4801-9954-86bb440d2ead]     import rosbag
2023-02-08 11:58:43.479 [1301/run_subsystem_with_bag/9070 (pid 8798)] [26f877a1-b061-4801-9954-86bb440d2ead] ModuleNotFoundError: No module named 'rosbag'

However, if I can

docker run

FULL_IMAGE

locally, run

python

and

import rosbag

no problem. Any idea why this would work locally and fail in the cloud?

do you have a local env active that has

rosbag

installed? Hypothesis is the local run is picking up from the active env on the machine that launched the flow, but that env wouldn't be active on remote.

Also this thread might be useful

I do have a local python env which i use to kick off the flow. But the test i am referring to however i am running within the docker container that i am specifying in the

@batch

decorator of the flow. I.e. the flows stage has

@batch(cpu=8, memory=24576, image=FULL_IMAGE)
    # @retry
    @step

and locally i am running

docker run ... FULL_IMAGE...

root@d1979d8cff2f:/# python
Python 3.9.16 (main, Feb  8 2023, 17:58:19) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rosbag
Failed to load Python extension for LZ4 support. LZ4 compression will not be available.

yep! if you activate the environment explicitly in your docker file, then it should work

i just set my PYTHONPATH explicitly, instead of however it is auto-set in bash'