That is so cool !
May I ask at which resolution you had those results ?
We managed to get up to 10x for very low resolutions (160) for a resnet101 but it usually plateaus for high resolutions (above 896x896) at a 1.7~1.9 speed-up.
Although using Int8 gives even higher speed-ups (~times 3.6 for 896x896 input), for some tasks it degrades the performance too much.
I think using TRTorch[1] can be quick way to generate both easy to use and fast inference models from PyTorch.
It compiles your model, using TensorRT, Ahead of Time and enables you to use the compiled model through torch.jit.load("your_trtorch_model.ts") in your application.
Once compiled, you no longer need to keep your model's code in the application (as for usual jit models).
The inference time is on par with TensorRT and it does the optimizations for you as well.
You can quantize your model to FP16 or Int8 using PTQ as well and it should give you an additional speed up inference wise.