"This is what's happening to the parameters of models when they're quantized down to sizes that are possible to run on your laptop. Instead of floats, small integers are what get stored and loaded into memory. When the time comes to use the quantized values, to generate an answer to a question for example, the values are dequantized on the fly. You might think this sounds slower, but we'll see later on that this actually ends up being faster as well as smaller."
I thought that most GPUs supported floating point math in these quantized formats, like they can natively do math on an float4 number (that's maybe packed, 2 float4s into a single byte, or more probably 16 float4s in an 8 byte array or maybe something even bigger)
Am I getting this wrong - is it instead the GPU pulls in the quantized numbers and then converts them back into 32-bit or 64-bit float to actually run through the ALUs on the GPU? (and the memory bandwidth savings make up for the extra work to convert them back into 32 bit numbers once you get them onto the GPU?)
Or is it some weird hybrid, like there is native support for float8 and Bfloat16, but if you want to use float2 you have to convert it to float4 or something the hardware can work with.
I am confused what actually happens in the vectorized ADD and MULT instructions in the GPU with these quantized numbers.
And every couple of days we see a post or a tweet about "Google/Apple/Microsoft just nuked my account with no notice and no recourse" so trusting them to sync passkeys rightfully makes some people nervous.
Is there any word on the screen material? I have a Pixel 3a that for the most part I really like, but one of the ways I think they got the price down was by using a glass that was much less scratch-resistant. My original Pixel never scratched, the 3a is scuff up badly after just 6 months. My iPhone 7 is a couple of years old and also has no scratches, so I'm hoping for a similarly scratch-resistant screen.
The Arpaci-Dusseaus are some of my favorite professors at the UW - they're not only first-rate researchers but fantastic teachers.
Remzi in particular has a very dry but hilarious sense of humor. His exams are a hoot (but are also great questions to see if you really know your material)
"This is what's happening to the parameters of models when they're quantized down to sizes that are possible to run on your laptop. Instead of floats, small integers are what get stored and loaded into memory. When the time comes to use the quantized values, to generate an answer to a question for example, the values are dequantized on the fly. You might think this sounds slower, but we'll see later on that this actually ends up being faster as well as smaller."
I thought that most GPUs supported floating point math in these quantized formats, like they can natively do math on an float4 number (that's maybe packed, 2 float4s into a single byte, or more probably 16 float4s in an 8 byte array or maybe something even bigger)
Am I getting this wrong - is it instead the GPU pulls in the quantized numbers and then converts them back into 32-bit or 64-bit float to actually run through the ALUs on the GPU? (and the memory bandwidth savings make up for the extra work to convert them back into 32 bit numbers once you get them onto the GPU?)
Or is it some weird hybrid, like there is native support for float8 and Bfloat16, but if you want to use float2 you have to convert it to float4 or something the hardware can work with.
I am confused what actually happens in the vectorized ADD and MULT instructions in the GPU with these quantized numbers.