We're trying to justify using Metaflow on Batch over Sagemaker to our users, but one thing we noticed whilst training their model was that Sagemaker is slightly faster. I looked at the average time per epoch (100 epochs), and Batch was 10.67 minutes versus 9.09 minutes for Sagemaker training. The Batch training job on Metaflow is using the

p2.xlarge

instance type and the Sagemaker training job is using the

ml.p2.xlarge

instance type. Configuration for both was set the same. Any clue as to why the Sagemaker training job is slightly faster?

are you using

@conda

- that can potentially add ~30s to runtime - which shouldn't matter for longer training runs

Using

@conda_base

, and that's about 3-5 minutes but only counts as overhead at the start

correct - are you able to reliably reproduce these numbers over multiple training runs?

These are long training runs - 20+ hours for both

oh - these numbers are per epoch

Same configuration, same data, same sample size, same batch size. We are running a controlled experiment I believe

what are you specifying with

@resources

?

it's likely that the sagemaker instance had access to higher resources -

p2.xlarge

supports 4 vcpus and 60G RAM

I see the ec2 instance provisioned by batch and it looks like it has 4 cpus

Yes, the ec2 instance has 4 vCPUs but unless you define

cpu=4

in the

@batch

(or

@resources

) decorator, by default the job will only use 1 vcpu

Ohhhh - that makes sense. Thanks for clarifying Savin!

the performance between ec2 instance through AWS Batch vs Sagemaker should be the same because at the end of the day - it is the very same instance (but lot more expensive on sagemaker).

Got it! Yeah that's what I thought but I didn't know Metaflow only uses a portion of the resources used by the ec2 instance, but yes that makes sense now. I thought the resource decorator helps determine which instance type is selected and then it uses the entirety of that designated instance

Ah no - the

@resources

and

@batch

decorator dictate how much resources of a single ec2 instance are allocated to a job. You can fit multiple jobs on a single instance - which will lead to further (and drastic) cost reductions compared to Sagemaker.

Understood, thanks again Savin!