HackerTrans
TopNewTrendsCommentsPastAskShowJobs

haolu7

no profile record

Submissions

8bit HippoAttention: Up to 3X Faster Compared to FlashAttentionV2

blog.hippoml.com
4 points·by haolu7·vor 2 Jahren·0 comments

comments

haolu7
·vor 4 Jahren·discuss
No, you don't need to train the model differently to use it with AITemplate. Here is an intro example to do inference with AITemplate with a very simple PyTorch model: https://facebookincubator.github.io/AITemplate/tutorial/how_.... For more advanced examples, check out https://github.com/facebookincubator/AITemplate/tree/main/ex...
haolu7
·vor 4 Jahren·discuss
AITemplate-PyTorch Stable Diffusion is the fastest stable diffusion inference solution by pushing image generation below one second on A100 (batch 1: 0.7s / 25 steps, 1.3s / 50 steps; batch 3: 1.6s / 25 steps, per image 0.55s; batch 16 7.9s / 25 steps, per image 0.49s) for the first time, 2.57X faster than Keras' XLA-based GPU compilation solution.

More benchmark numbers and repro at: https://github.com/facebookincubator/AITemplate/tree/main/ex...