A paper [1] we wrote in 2015 (cited by the authors) uses some more sophisticated data structures (compressed suffix trees) and Kneser–Ney smoothing to get the same "unlimited" context. I imagine with better smoothing and the same larger corpus sizes as the authors use this could improve on some of the results the authors provide.
Back then neural LMs were just beginning to emerge and we briefly experimented with using one of these unlimited N-gram models for pre-training but never got any results.
I find the requirement of using a laptop without being allowed to use an external monitor to view potentially very long documents (half screen!) for hours on a tiny screen to be ridiculous.
Funny enough "learning characters/words in the context of the vocabulary they are in" is exactly what NLP machine learning models use to learn "rich" word/text representations based on the "distribution hypothesis" which states that the choice of words in the same context share a common meaning.
From the first link: "Comparing OFDM to LTE today we find a better scalability to a much lower latency (an order of magnitude lower round-trip time [RTT] than LTE today) in OFDM."
Doesn't LTE already have quite good latency properties?
I thought the NVIDIA drivers for the more fancy cards (TITAN etc) are the same as for the gforce cards. Wouldn't this restriction apply to those cards as well? Doesn't make much sense to me...
> I find it hard to believe this. Their main index is certainly not all-RAM (there must be some flash and maybe even disk), and the throughput would just not be enough for something like BitFunnel.
From looking at the github repo it does look like the system runs entirely in main memory.
I was at the SIGIR'17 presentation of this paper (won best paper award btw) and have some comments in general:
- They mentioned (from what I remember) that they now use BitFunnel as they core of the complete Bing search engine not just the fresh parts.
- When I read the paper and looking at the code, it looks like their index doesn't include frequency information whereas your PEF code does. It is unclear what was counted in the experiments.
- If you look at the code, they are actually doing much more complicated stuff than just regular bloom filters by "bin packing" the hash positions for each term to reduce false positive rates (see https://github.com/BitFunnel/BitFunnel/issues/278 ). I'm nor sure if it is "fair" to compare a system developed by 10+ engineers over many years to a "phd student" code base developed over short period of time. I think the PEF code is excellent but I'm more talking about that engineering efforts can have a large impact on performance.
- I'm fairly sure you are right regarding the lack of URL-sorting. However, this can have another cause. If you consider Figure 4 in the paper which shows how "higher ranking rows" group documents together to allow faster intersection. URL sorting causes clusters in document-ids. Say, in the example in Fig. 4 there might be a cluster for that specific term for documents 0,1,2,3. This would mean the "higher ranking" row approach becomes worse (more false positives) when clustering occurs in the collection. So while URL-sorting helps PEF, it will most likely make BitFunnel worse.
There are much faster SA construction algorithms than skew (check out divsufsort). The O(n) algorithms using induced sorting are also likely much faster than this work. The constants of recent O(n) algorithms are very low.
Back then neural LMs were just beginning to emerge and we briefly experimented with using one of these unlimited N-gram models for pre-training but never got any results.
[1] https://aclanthology.org/D15-1288.pdf