| „[…] was a friend telling me his LaTeX thesis took 90 seconds to compile towards the end“
Sure, but in order to iterate you won’t have to compile the whole document but can just keep the chapter you are working on by structuring it with \includes
This was a great read, thanks a lot!
One a side note, any one has a good guess what tool/software they used to create the visualisations for matrix multiplications or memory outline?
Okay, I see that for inference. But for training it shouldn't matter because I need to hold on to all my activations for my backwards pass anyways?
But yeah, fair point!
"[...] modern neural network (NN) architectures have complex designs with many components [...]"
I find the Transformer architecture actually very simple compared to previous models like LSTMs or other recurrent models. You could argue that their vision counterparts like ViT are conceptually maybe even simpler than ConvNets?
Also, can someone explain why they are so keen to remove the skip connections? At least when it comes to coding, nothing is simpler than adding a skip connection and computationally the effect should be marginal?
Sure, but in order to iterate you won’t have to compile the whole document but can just keep the chapter you are working on by structuring it with \includes