Bradley-Terry and Elo scores are equivalent mathematical models! The fundamental presumption is the same Thurstone model - that an individual's skill in a particular game is a normally distributed random variable around their fundamental skill.
We did experiment with a Bradley-Terry loss function (https://hackmd.io/eOwlF7O_Q1K4hj7WZcYFiw), but we found that even better was to calculate Elo scores, do cross-query bias adjustment, and then MSE loss to predict the Elo score itself.
Yeah absolutely. In your link, it iterates on _ = ^{_}, until it finds the fixed point.
In our training pipeline, we had to convert the fixed point iteration to be on _ directly for numerical stability. I have a post on that here!: https://hackmd.io/x3_EkXGKRdeq-rNHo_RpZA
Bradley-Terry also very cleanly turns into a loss function that you can do gradient descent on, which will cause your model to efficiently learn Elo scores!
Our calculations are at: https://hackmd.io/eOwlF7O_Q1K4hj7WZcYFiw
tldr; wikipedia iterates on <e^elo>, but that can go to zero or infinity. Iterating on <elo> stays between -4 and 4 in all of our observed pairwise matrices, so it's very well-bounded.
We found that MSE after elo-adjustment worked equally well. And, MSE lets you shuffle (q, d) across the dataset which has good statistical properties (Versus contrastive, which makes you sample the same query many times within a single minibatch)
In this case "InfoNCE" isn't applicable because the reranker's output is a scalar, not a vector. So that's why we checked both bradley-terry and MSE.
Yeah that's exactly what we observed. Our goal was to create an absolute score that's completely independent from the Corpus, which is difficult because naturally all ELO distributions are inherently tied to the corpus itself!
When we were exploring the mathematical foundations, we considered ELO scoring against a "Universal Corpus" based on the natural entropy of human language (Obviously that's intractable, but sometimes this term cancels out like in the DPO proof).
But eventually we figured out a method using cross-query comparisons to assign an "ELO bias" to all document ELOs within a given query's candidate list. This normalizes it correctly such that when a candidate list is all bad, the ELOs shift low. And when the candidate list is all good, the ELOs shift high. Even when the relative ELOs are all the same.
I often see it rendered as "Elo" but I've always found it more natural to capitalize as "ELO", but perhaps I should swap to "Elo" given this. Pronouncing "ee-low" is certainly the way it's done in chess/esports though!
Hey! We actually did a lot of research into ELO consistency, i.e. to check whether or not the NxN pairwise matrix followed the ELO model. It was a long road that's probably grounds for an entirely separate blog post, but the TLDR is that we observe that:
For each document, there is a secret hidden score "s" which is the "fundamental relevance according to the LLM". Then, when we sample (q, d1, d2) from the LLM, the LLM follows the statistical property that:
- The "fundamental hidden preference" is `pref = s_{d1} - s_{d2}`, usually ranging between -4 and 4.
- The LLM will sample a normal distribution around the `pref` with stddev ~0.2, which is some "inner noise" that the LLM experiences before coming to a judgement.
- The preference will pass through the sigmoid to get a sampled_score \in [0, 1].
- There is an additional 2% noise. i.e., 0.98 * sampled_score + 0.02 * random.random()
When we use Maximum Likelihood Estimation to find the most likely predicted "hidden scores" \hat{s} associated with each document, then we go ahead and sample pairwise matrices according to `0.98 * sigmoid( \hat{s}_1 - \hat{s}_2 + N(0, 0.02) ) + Uniform(0.02)`, then we get a pairwise matrix with virtually identical statistical properties to the observed pairwise matrices.
Yes, a step where you do a structured extraction into a database column would be a potential solution. But, it requires a preprocessing step.
It all depends on the use-case, sometimes you get a query that you couldn't have predicted the filter beforehand. In those cases, usually what you have to do is open up a spreadsheet and then manually categorize every document by hand. LLMs and modern AI are great ways to automate this.
A really good solution might be to have a system that computes these filters on-the-fly, but also caches them for later reuse if a query asks for that filter again.
For long documents we have a rolling window strategy. So, we cut the document into 5,000 token groupings for use in inference. There's also a 400 token overlap, and we prefer the earlier chunk for overlap tokens.
For example, if Group #0 overlaps with Group #1 at index 5,200, then we use the logprob from Group #0, because it had more context. Group #1 gets the benefit of context for indices 5,000-5,400, even though we toss out the logprobs for that range.
No need to keep track of chunks that it's already made, we just want heatmap values and then we use those heatmaps to split at the hottest character that's around our target chunk length (Or use a threshold value and binary search the threshold for our target # Chunks or average chunk size).
It scored better than LlamaIndex's recursive character text splitter and that was including some custom regex work to improve it. If you put enough effort into the regex you could probably get there, but the whole point of the agentic chunking is for it to be automatic and contextual.
I've written a lot of RAG pipelines over the last year, and one consistent pain-point is writing regex to chunk the documents correctly.
Right now, the most common chunking algorithms are:
- Split every 1000 characters
- Split on whitespace
- Recursively split on: (many newlines, then one newline, then periods, then spaces)
The best is recursive character text splitter, but regex is super brittle and when it fails to match it ends up creating huge chunks. Worse, this solution also has the overhead of needing to maintain regexes for every single filetype.
Here we propose LlamaChunk, an inference-efficient method of LLM-powered chunking. Using this method, it only requires a single LLM inference over your document in order to provide the most optimal recursive character text splitting, without needing to hope that a bunch of hard-coded rules work on your unstructured data.
That's because the $50M were owned by a large percentage of the users of Ethereum, not a single person. It represented 15% of the total ETH in circulation, back when the currency was in a very nascent state and the flagship product that ETH provided that BTC didn't, was the DAO. When the DAO break happened ETH lost 60% of its value, so not only did 15% of people have their money stolen, but everyone across the board lost 60% of their ETH value. There was simply immense demand for people to get their money back, so people much preferred the fork that kept their money than the fork where the robber stole their money.
Simple as that, it's decentralized, that's the whole point, you fundamentally can't tell people which fork to believe, people use whatever fork they want. And the people mostly wanted the fork with their money in the DAO preserved. People who wanted the unaltered chain stayed there, no big deal.
Wait a second, am I missing out right now? I have an opportunity at Jane Street, but I haven't been exposed to any of the other small name firms. Where do I look to find the higher paying jobs?
This seems like a waste of time. It's not illegal to register a domain name, no matter how similar it is to your domain name. If you want those domain names, then buy them. My family runs a business and we bought the domain names that we thought were similar. We respect the fact that if someone buys our domain but with ".io" at the end instead, then that's their right. I would be very scared of a world where people can sue over names being "too similar to this large company with lots of money and power". These domain name companies serve one purpose: selling available domain names. It's not complicated. Let's not make it complicated.
I mean, I can be more lenient. Europe had good researchers. They were the Athens, between the Germans ones of the 20-50's, and Turing, there was a lot. The Russians got a lot done too. But again, all research. Nothing profitable. It was never self propagating. Rome was self propagating, they conquered. Athens, only sustained intellectualism until it bled away.
(I'm not counting the mathematicians of the 1700s that did amazing work, I'm referring to modern Europe and the US, 20th century and beyond)
It's kind of sad that a huge market for businessmen is taking an American business, and just doing the same thing in Europe or Latin America. It's, it's sad. There's uh, absolutely no foreign competition in most situations that we have to meet. The ratio is clearly wrong, and we compete with more companies than we can found so Europe should statistically be represented in the other direction. It's not even that profitable, American consumers are a gold mine they're so price insensitive. And the modern world is mostly what was invented in the roaring 20s in the US, and computing technology that was developed almost entirely in the US.
(The Japanese were trailing along in the 80s I guess, but they were mostly our China, they were just a factory line that we abandoned when China caught up and their GDP has stagnated for 20 years. They had a few great companies, and they still do. South Korea is the only actual one that goes toe-to-toe but they're too small to profit off 300 million people like we can, per capita I do think they are more entrepreneurial and inventive than us though)
There's not even VC money in Europe it's almost impossible to get funded, you need American or Chinese money which means they own most of the business anyway. Like in the US, take the money away, you can get started again. But in Europe? You better buy a plane ticket. No one will fund you. The non-tech companies are mostly incumbents. Or brands like WeWork that are much more expensive than other co-working spaces but they have the brand to hold them up.
Eh, I don't think so. There's too much interdependence, you can have anxiety over how many things your dependent on, especially in Tech. It's actually terrifying. If Azure closed down tomorrow I'd be fucked so hard, though any business will eventually try to be as independent as they can when they're big enough (Other than Netflix and AWS, I'm sure there's some under the table money making that happen as 50% margins apparently disappear from Netflix's wallet even though Netflix could just run it themselves). Too many companies and systems supply the whole world at a near monopoly, if epackets disappeared everyone would be fucked beyond belief. We all know that there's a lot of people involved, and we all do respect the workers too. Go to any area with businessmen, Grand Central Station or an international airport or something. And look at the section with books that are clearly targeted towards businessmen. At least 1/3rd of the selection is emotional support in response to having to fire people, and the emotional support of handling an economic downturn. People jumped out of buildings because of 1987 all over Downtown Manhattan, it was horrible. They know so many things need to be carefully in balance for them to continue.
You always have to be humble unless you're a billionaire because there's always someone vastly more powerful and rich than you are, by orders of magnitude. Your network's wealth and power as a normal distribution always ends up trailing behind you as you try to meet new people above you, but there's always the right tail end where you know a few people who could spend your net worth in a night because they want to.
Look at how humble the CEO of the Maverick's has to be as he meets Mark Cuban. It's somewhere in there, I'm not gonna find the time but it's in there.
I don't know if you've ever seen the Black Mirror dystopia about social media and kissing up to people. There's a lot of that. It sucks. That's why most business events involve drinking until tispy, it loosens everything up.
I'm not sure what you mean by "Quality of life is just a sham invented by politicians to keep the plebs from demanding... quality of life?". I don't think politicians invented anything like that? I remain confused.
This is kind of what I touch on at the end, with the getting the kids into business, there is a moral concern there. The first executives are intelligent, and caring. They're the engineers. Once you have MBAs flooding in, which HR doesn't know how else to hire people, it just becomes horrible. Feynmann famously ripped NASA a new one after asking the engineers "What's the probability of the Shuttle exploding?". They say: 1/100, 2/100. He asked the executives at NASA, they say 1/100000, 1/1000000. How, how is that even possible? You need the executives to be the engineer, or they fuck everything up. An engineer warned them about the o-ring, the executives ignored it because they were stupid. Same with the max:
Notably: "For the first time in my life, I’m sorry to say that I’m hesitant about putting my family on a Boeing airplane," Ed Pierson wrote to a company executive before the first tragedy.
This would never happen if the executives were engineers. Every time we, humanity, eventually gets stuck in a rut. The Romans invent the most unbelievably complex and well structured government of their time, while the gauls hunt like animals, and they conquer the world, they develop all of the technology (Well, the greeks were even better with their near idealistic Athens, but like the PhDs it never gets implemented or spreads). So the Romans do this, they win. Then...., nothing happens. It goes to shit. The Patricians murder the Plebians who protest, they become corrupt, the people have less and less control, it becomes awful. Then, dictatorship. Then, emperor. Who were the people who drafted the first setup of the roman government? They're 600 years dead, and those who replaced them made it shit. Took 1000 years to recover. The founding fathers. Compare George Washington, and his morality, to our modern politicians. No one had any reason to do anything other than create the perfect government, everyone helped, it was collaborative. Now they're cutthroat. It always goes to shit. Corporations are the new target. The founding fathers tried so hard to put every single failsafe they could, because they knew. They knew they knew. They knew those who would come after them would make it shit. While the founding fathers intentionally put failsafes - the entire article is a big ass fail safe you just read it (It just says exactly what the restrictions are - if people acted just like the founded fathers did they would never need restrictions in the first place), our modern politicians research every possible exploit to the system as if they're in infosec. Gerrymandering, oops! Jefferson forgot about that one, it took 300 years to discover the buffer overflow, but if it's discovered it's abused. Every metric becomes a target. The target of "leading the revolution" gave people who were intelligent and passionate about saving their country. The target of "becoming a politician" in 2020 is https://en.wikipedia.org/wiki/Roger_Stone I highly suggest you read just the paragraph about high school. It shows what you need to be, what the selection criterion is. The only selection criterion that works is: Coming up with something new, a new business, a new idea, a new government. After that, they hunt. They get their MBAs to overthrow you with credentials so that they can squeeze themselves in, even though they didn't found Boeing. The patricians bribe who they had to to squeeze even, even though they never came up with the original government. The politicians cheat what they must to squeeze into what the founding fathers tried to prevent. Someone has their eyes on the prize.
Mind you, the French are just as intelligent as the Italians, they're nearly identical at birth, only culture separates them. A french man who grew up in italy is an italian man. It's the system that is what separated the gauls from the romans, and it's survival of the fittest, you're the first to have an opposable thumb then you win. Whoever figures out how to create a self propagating government wins, a couple people did, got a few thousand followers, and it propagated. Business is the new self propagating system, until stability is reached. Only the new people have what it takes. It's not really too genetic either, it is in some way have 4 kids one of them will work, what of the other three? Kahn couldn't stop it, Alexander the great couldn't stop it, Romans worked for 300 years because of adoption, but it was still bleeding. Go through the history. Each half century involved more failures by the plebians and more success from the corrupt patricians, it was slow but sped up, more and more restrictions were put on the people by the incumbents. You always need fresh people. Carnegie? Vanderbilt? Compare to the modern CEO of BP...., of Boeing, stability kills. In hackernews, for people in IT, no one knows shitty bosses and executives more than they do.
We did experiment with a Bradley-Terry loss function (https://hackmd.io/eOwlF7O_Q1K4hj7WZcYFiw), but we found that even better was to calculate Elo scores, do cross-query bias adjustment, and then MSE loss to predict the Elo score itself.