I wonder if it would be possible to do something simple like prepending sentinel tokens with the year. Or, since they're training a model from scratch anyways, tweak the architecture to condition on a temporal embedding. That opens the door to cool stuff like: Generate a response from 2050.
> Branching predictions involves following a few logits to see what other tokens they lead to. This is often called MCTS (Monte Carlo Tree Search) and is a method that has been often tried in LLMs to middling success. One of the tradeoffs of branching is that it requires using inference compute in a way where the branches cannot benefit from each others compute.
I wonder if speculative decoding could help here? E.g. have some small model draft predictions for the branches and parallel and have to big model verify the most promising one.
Every 2.5-5 miles in SF = about once a ride. The city is only 7x7 after all. I've taken 4 Cruise rides, all within that range, and had a message pop up saying a human was intervening during one of them when the car had gotten stuck in front of some street nonsense in the Tenderloin. I'm not sure I would classify this as a "major scoop" unless there was evidence that humans were also intervening during situations that weren't apparent to the rider.
Note that this post is from May 28, before the release of gpt-4-0613. By "the last two updates", I believe the poster is referring to some UI changes, that possibly also included some underlying model changes(?)
Bossa nova is some of my favorite music to listen to while working. It's mellow, soothing, but never boring. Also, being in a foreign language helps minimize distraction. RIP
This seems like a corollary to Betteridge's law of headlines. If the article was about the Salesforce Tower, the headline would've said 'Salesforce Tower'.
A minor quibble with your use-case explanation: The advantage of a bloom filter isn't strictly time complexity. For example, a hash table would also have constant lookup time (best case), and would give a definitive answer on set membership. However, to store 1 million IPv6 addresses would take 16 MB. You can see very quickly that this would not scale very well to, say, a billion addresses stored in-memory on a laptop. With a bloom filter, we can shrink the amount of storage space required* while maintaining an acceptable, calculable false positive rate.
* IP addresses actually aren't a great use case for basic bloom filters, as they're fairly storage efficient to begin with, as opposed to a url for example. Taking your example, say we need to store 1 million IP addresses in our bloom filter and we're okay with a ~1% false positive rate. Well then, if we use a bloom filter with 2^23 bits (1 MB), the optimal number of hash functions is (2^23)/(10^6)*ln(2) = 6, yielding a false positive rate of (1 - exp(-6* 10^6 /2^23))^6 = ~1.8%. So we're using 6% of the storage space, but with a nearly 2% false positive rate.