AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling(junzhan2000.github.io)
junzhan2000.github.io
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
https://junzhan2000.github.io/AnyGPT.github.io/
20 comments
We're clearly heading towards a "Generalist Multimodal Large Language Model" that autonomously selects the appropriate specialized LLM for any given task, rather than requiring us to switch between multiple LLMs. The combination of a mixture of experts approach and multimodality appears to be the way forward. Very excited for the future.
TFA is the exact opposite of that though? It's a single model that acts on a single representation, which is derived from (and mapped back to) different input (output) modalities.
Apparently used in GPT-4 and coming in Gemini 1.5.
I don't think it's accurate to call them language models anymore actually. Large Multimodal Model?
As long as the underlying representation is a sequence of (embeddings of) tokens , and the training depends on next-token prediction, they should still be called language models. The 'language' part of 'language models' is about the architecture, not the modalit(y/ies).
Is it a "language" model though, if you have a multimodal autoencoder that converts input into concept tokens, and you do sequence extension in concept space?
Unambiguously, yes.
To elaborate a bit, that's effectively what happens in a natural-language-only LLM. Text tokens are immediately mapped to wide embeddings, which are processed by a transformer, producing another embedding, which is decoded back to a discrete token. To a first approximation, unified multimodal models substitute f: text -> [embedding] with g: image -> [embedding,] and act on a sequence like [*f(t), *g(i), ...]. So... the multimodal model maps text/image parts to wide embeddings, which are processed by a transformer producing another embedding, which is decoded back to a discrete text/image part. Which sounds somewhat similar to the former :).
Of course the function g is a bit more complicated, and decoding takes more work, but the underlying machinery is unchanged. The ability to handle multi-modal inputs/outputs uniformly is exactly what's so exciting about work like this!
To elaborate a bit, that's effectively what happens in a natural-language-only LLM. Text tokens are immediately mapped to wide embeddings, which are processed by a transformer, producing another embedding, which is decoded back to a discrete token. To a first approximation, unified multimodal models substitute f: text -> [embedding] with g: image -> [embedding,] and act on a sequence like [*f(t), *g(i), ...]. So... the multimodal model maps text/image parts to wide embeddings, which are processed by a transformer producing another embedding, which is decoded back to a discrete text/image part. Which sounds somewhat similar to the former :).
Of course the function g is a bit more complicated, and decoding takes more work, but the underlying machinery is unchanged. The ability to handle multi-modal inputs/outputs uniformly is exactly what's so exciting about work like this!
You're right, that calling them language model is no longer accurate. But "Language Model" has a precise signification (which is that it tries to create a perfect representation of the language). Saying "multimodal model", you're not really saying what does it try to represent. "World Model" or "Human Model" would be more accurate at this point
World model sounds too much of physics than it deserves imho. If you want to name it after what's being modeled (which it doesn't need to be, it could also be named after the methods used eg), why not just call it something like general token model?
What they model is meaning. I'd call them "semantic models".
"Large Observation Model" has a nice ring to it
I’m fond of Large Sequence Model, which I’ve seen used a bit recently. If your data can be defined as a sequence of unique symbols, actions, letters, tokens, patches, whatever - a transformer is a great way to model it.
I'm partial to Large World Model (LWM).
Large Sequence Model (LSM) is OK too.
Large Sequence Model (LSM) is OK too.
Are there any MoE multimodal approaches?
Say I am between sophomore and junior, what's the best way to bootstrap from calc+linalg+stat to being able to competently configure and piece these components together into something like this? Any good lecturers or courses?
What's with the dummy github page? Anyway. This feels like the right step forward. Just like OpenAI, i have near religious faith in the transformers architecture. The question is how these modalities can work together better.
I feel like this type of capability or architecture might be the future of interactive agents. The quality of the voices leaves a little to be desired, but otherwise it seems very powerful.
Given how quickly we've been churning through sota architectures in the last decade or so, I'd not bet on anything we currently have being "the future"
The key takeaway for me:
Whether data is continuous or discrete, no matter its modality (text, video, music, etc.), we now have an array of proven methods for representing it with discrete tokens, enabling us to use existing sequence modeling architectures (Transformers, linear RNNs).
We live in interesting times!
Whether data is continuous or discrete, no matter its modality (text, video, music, etc.), we now have an array of proven methods for representing it with discrete tokens, enabling us to use existing sequence modeling architectures (Transformers, linear RNNs).
We live in interesting times!
Totally agree. Another takeaway is that data from different modality has some shared intrinisity, so one can be mapped to another.