Evaluation dataset designed to test the capabilities of Retrieval-Augmented Generation (RAG) systems. Paper with details and experiments is available on arXiv: https://arxiv.org/abs/2409.12941.
Dataset Overview
824 challenging multi-hop questions requiring information from 2-15 Wikipedia articles
Questions span diverse topics including history, sports, science, animals, health, etc.
Each question is labeled with reasoning types: numerical, tabular, multiple constraints, temporal, and post-processing
Gold answers and relevant Wikipedia articles provided for each question
Key Features
Tests end-to-end RAG capabilities in a unified framework
Requires integration of information from multiple sources
Incorporates complex reasoning and temporal disambiguation
Designed to be challenging for state-of-the-art language models
Usage
This dataset can be used to:
Evaluate RAG system performance
Benchmark language model factuality and reasoning
Develop and test multi-hop retrieval strategies
“Facebook users, randomized to deactivate their accounts for 4 weeks in exchange for $102, freed up an average of 60 minutes a day, spent more time socializing offline, became less politically polarized, and reported improved subjective well-being”
In my experience Facebook offers polarization and information overload on one side, or a barrage of memes and random nostalgia on the other. I’ve given up on trying to manage what is recommended to me there, although for a while I did purge my friends and likes carefully, to some effect.
I don’t go so as far as deleting Facebook but it’s definitely less a resource and more an entertainment / distraction vehicle
This headline strikes me as a shift burdening users themselves with finding bad actors. Even the statement from e.g. Twitter, make it seems so:
"We think it’s important for people to be aware that this exists out there and that they review the apps that they use to connect to their accounts,” said Lindsay McCallum, a Twitter spokeswoman."
But I would argue that users who fall for these nefarious services should not be the lookout. Instead the trust should be placed on teams at Facebook and Twitter that vet the bad actors, e.g. oneAudience.
I understand vulnerabilities abound and moderation is hard, and educating users is important. I'm just irked a bit that the accountability is shifted here.
My guess is that the most outspoken and principled activists among the Google workforce will find themselves singled out. If they navigate an exit wisely it could be great for them personally and the industry at large. The upside is that by managing an exit from adverse working environments, their passion experience and pedigree will likely fuel some great upstarts.
I really hope these people choose their battles wisely and use their force for good. Not to transform a firm, but rather to disrupt the status quo of an entire industry, without needing association to a name like Google or FB.
“So if you're sending one email that you wrote in 10 seconds to 20 people, you're not spending 10 seconds, but more like 20 minutes of resources: wouldn't it be better to work 5 minutes to find a solution yourself?”
I aim to be the person who will take 5 minutes to find a solution rather than seeking it via email or tweet. Evaluating my time vs others’ time and attention is habitual throughout my career. I pride myself in being capable and considerate. But many corporate, office, or otherwise networked cultures will drown out a person’s focused effort and wins among the outspoken or otherwise visible folk.
I appreciate your work in bringing to light email as a costly tools in the process of knowledge sharing and coordination.
Dark patterns are often justified as trying to gain more of a user’s/customer’s attention, as seen to be the case here. IMHO, this is Amazon shrewdly introducing limits to their charity.
Whether it be a functional constraint or a dark pattern, user’s attention is being exchanged for agency of charity on behalf of the consumer.
A quick search on Twitter leads me to believe there is a feedback cycle Amazon leveraged with notifications about the Smile program. Consumers buy, notifications of their charity are pushed, Amazon profits. Repeat cycle. Amazon profits.
Speakers list includes Ricardo Cabello, Creator of Three.js
on Creating VR (and AR) on the web.
Other interesting topics:
- Hooked on D3: Creating Animated Ch(art)s with D3 and React Hooks
- Building Distributed Systems with Node.js
- Designing and Building With Privacy In Mind
- Value Driven Development
Talk about the ethics of what we make and how we are contributing to a better and more interconnected world, being part of the change, building a future we all can be proud of.
- Simplify Web App Development with Svelte
Svelte circumvents including a runtime library, it compiles to bundled JavaScript that is very small compared to other approaches. Svelte components achieve "reactivity" without using a virtual DOM. Implementing components requires less code than popular web frameworks.
As far as I can tell this is a blanket generalization of Venezuelan Adobe account holders. Any student body, civil society NGO or independent media outlet that relies on registered copies of Photoshop, InDesign or Acrobat will be impacted.
France is planning to incorporate facial recognition technology into a mandatory digital identity for its citizens.
Alicem is the name of France’s face ID program. Facial recognition will be the only way for citizens in France to create a legal digital ID, through a one-time enrollment that compares a user’s passport photos with a selfie video taken on the Alicem app.
However, France’s government insists that, unlike China’s, its ID system won’t be used to monitor citizens, or integrated into identity databases. It says face scans will be deleted when the enrollment process is over.
This sad news is buried here on HN but surfaced on my Twitter timeline. Surprised there isn’t more empathy or anecdotes going around to bring light to issues surrounding this situation. There is a real need to humanize work in our industry and heed mental health. Work is hard, and this lost life speaks to how terrible reality becomes when work and identity has is unbalanced, too tightly coupled.
Dataset Overview 824 challenging multi-hop questions requiring information from 2-15 Wikipedia articles Questions span diverse topics including history, sports, science, animals, health, etc. Each question is labeled with reasoning types: numerical, tabular, multiple constraints, temporal, and post-processing Gold answers and relevant Wikipedia articles provided for each question
Key Features Tests end-to-end RAG capabilities in a unified framework Requires integration of information from multiple sources Incorporates complex reasoning and temporal disambiguation Designed to be challenging for state-of-the-art language models
Usage This dataset can be used to:
Evaluate RAG system performance Benchmark language model factuality and reasoning Develop and test multi-hop retrieval strategies