MirageLSD: The First Live-Stream Diffusion (LSD) Model - A Vid2Vid running in real time, infinite generation, zero latency. Available now in a live-hosted unlimited demo at https://mirage.decart.ai!
1. Non-cherrypicked in its consistency (if you look at the demonstrations in the Oasis blog post you can find specific cases of consistency which is an anomaly rather than the norm)
2. Is live-inferenced at 20fps. If you use Runway v3 which is a comparably larger and higher quality model (resolution and consistency) it might take a minute or two generate 10 seconds of video.
3. Is served (relatively) reliably at consumer scale (with queues of 5-10k concurrent players) which means that in order to save on GPU cost, you increase batch size and decrease model size to “fit” more players in 1 GPU.
How does one select a good candidate for the draft model in speculative decoding? I imagine that there's some better intuition than just selecting the next parameter count down (i.e 70B -> 13B, 13B -> 7B).
Also how does that interact with MoE models? Do you have a mini version of the MoE, with smaller experts?
Is this still the case for sliding window attention/streaming LLMs, where you have a fixed length attention window rather than infinitely passing in new tokens for quadratic scaling? You even get better performance due to purposely downsampling non-meaningful attention sink tokens.
That's not so much a use case, but I get what you're saying. It's nice that you can find optimizations to shift down the pareto frontier of across the cost and latency dimension. The hard tradeoffs are for cases like inference batching where it's cheaper and higher throughput but slower for the end consumer.
What's a good use case for an order of magnitude decrease in price per token? Web scale "analysis" or cleaning of unstructured data?
If you were to serve this on a datacenter server, is the client to server roundtrip networking the slowest part of the inference? Curious if it would be faster to run this cloud GPUs on better hardware but farther compute, or locally with worse hardware.
What are some of the better use cases of fast inference? From my experience using ChatGPT, I don't need it to generate faster than I can read, but waiting for code generation is painful because I'm waiting for the whole code block to format correctly, be available to copy or execute (in the case of code interpreter). Anything else fall under this pattern?
Under the same conditions where enterprise versions of the API have significantly less latency and better reliability than personal. OpenAI can change anything about the underlying infrastructure.
I don’t believe that the answer is strictly no. There are still many questions around the fine tuning method and the scale of data, as well as expectations of task accuracy from the perspective of an end user.
Knowledge instillation is probably the holy grail of fine tuning. The hard part is:
1. Generalizing new facts. You can create a question answer pair of: “what is the population of the world in 2023?” “8 billion”, but it may not be able to pick up alternate phrasing or “does the world have 8 billion people on it?”
2. Catastrophic and behavioral forgetting. Continued fine tuning after RLHF and instruction fine tuning may result in the loss of the alignment and instruction following capabilities trained by OpenAI. At worst, it will start spewing random tokens like the example in the post.
I have not yet seen it successfully done, and I suspect that updating fractions (~.1%) of the original weights with PEFT methods won’t help.