LLMs are very good at lossless compression via arithmetic coding. But I didn't know that it was possible to go the reverse direction (do language modeling via a compressor). It's not super great quality, but I'm surprised it worked! Other compression algorithms (like PPMd) use variable n-grams under the hood, and should be much better (although less interesting due to already containing basic language models internally).
Yes, this is the case. During training, the model will get a sequence of text (ex, 512 tokens long) with a percentage of them masked out (with a special <MASK> token). It learns how to unmask those tokens to construct the original text.
In the case that you mentioned, if we had 4 <MASK> tokens in a row, all we are doing for decoding is predicting what those 4 tokens should be.
Generally, this does not seem to be a significant problem, as there are usually multiple ways to express an idea in varying lengths. Also, with confidence-aware parallel decoding, it can usually avoid the scenario you mentioned, as focusing on decoding the highest confident tokens will generally avoid such scenarios with a well trained model.
Thanks for your input. Lots of good points on the technical side. Will go through and make some edits later tonight or tomorrow.
> You're not hijacking anything, you are using the old APIs as intended and using the right terms in the description that follows.
When it comes to the use of the word "hijacking", I use it in to refer to the fact that using graphic shaders for general computation wasn't initially intended. When NVIDIA allowed programmable vertex and pixel shaders, they had no idea that it would be used for anything else other than graphics rendering. So when I say I "hijack" a fragment shader to compute layers of a neural network instead of as a part of a rendering pipeline, this is what I mean. I don't see a problem with this use of language.
If you couldn't tell, the post was about using shader programming for general-purpose computation, specifically. Yes, WebGL adds compute shaders, but the point of the article was to use the graphics pipeline specifically. If you say there are statements that are incorrect or inaccurate, pointing them out would be very much appreciated :)
Here's a link to the github repo. At the top of the README it has a demo of GPT-2 running and the visualizations of the attention matrices and transformer block outputs
Thanks for the comment! I did this as a final project in a graphics class where we mainly used WebGL for all the assignments. Seeing the improvements a WebGPU port would bring would be cool to see!
A few weeks back, I implemented GPT-2 using WebGL and shaders. Here's a write-up over how I made it, covering how I used textures and frame buffer objects to store and move around weights and outputs from calculations while using WebGL.
Not my repo, but I think there are others that allow this functionality via the same method the “iMessage Wrapped” project accessed your message history. Since they’re also MCP servers, they should work seamlessly with Claude
https://github.com/willccbb/imessage-mcp/tree/main
Yeah, right now the weights aren't even pushed to the repo so there's no way for the github page to load them in. I've looked a tiny bit into it and might end up making a fix for it, having it load the weights by fetching them from somewhere else on load, but probably not today.
This was a final project for a graphics class where we used WebGL a lot. Also I was just more familiar with OpenGL and haven't looked that much into webGPU