I want to run the default metaflow setup (cloudformation) but run it in private subnets with a NAT gateway. For some reason, I cannot get this to work, and always my second task fails with a very vague error: "Error: 'path'". This happens when the task is started by metaflow in the batch job, and I cannot get any additional info as to what the issue is. I tried making the changes to the stack as minimal as possible, just removing all assignment of public IP except for the ELB, then replace the subnets. But the issue remains exactly the same. Any ideas?

@thankful-ambulance-42457 any ideas?

I found it already. It had nothing to do with internal routing actually. The error occurred (I suspect) when unpickling a pandas dataframe. I did not add numpy to the conda dependencies, so that's why it could not bootstrap the conda environment.

glad to hear it got sorted :) was the error made more vague by executing within metaflow, or is it as sparse when running natively as well?

I didn't get the error when running it natively actually. That's also what made me think it was an error in my metaflow setup itself, not the flow