> Some people also use LLMs to generate flashcards. And of course, the result will be those impersonal, mediocre cards.
> I won’t say LLMs are useless for this. But from my trials, I get about 1 card that’s useful to me out of 10, and even that 1 card still needs rewriting.
I don't know the specifics of how the author tried to do so, but from what I've seen the majority of attempts are, let me drop a chapter of a textbook and say "make flashcards." If that is what we are talking about, then yes, LLMs are useless.
In my mind, though, this is sort of like looking at the very first GitHub Copilot LLM autocomplete from a couple of years ago and concluding, yeah it's nice for one-liners, but it cannot write an app.
If you create a framework around your card-creation AI so that it can use tools, and verify its work to ensure common card-creation pitfalls don't happen, you can get pretty high-quality cards. In my experience, you go from a 10~20% hit rate to a ~90% hit rate, which in my mind is good enough. I got to ~75% quality just from a two extra LLM calls that would assess a potential card against a standard set of rules (adapted from [0]). There are huge Pareto gains to be had here.
I've generated thousdands of cards over the last few months this way. I let the AI add it directly to Anki via AnkiConnect. Then, if when I go to review I find a card that my AI created and I don't like it, I just delete it.
Removing the limitation of card creation is really quite compelling, and I think the area is still highly under-invested in. Would be cool to see a generic framework evolve that one could use. For now, I've been using a personal fork of clanki [1].
It's a balance. Maybe a helpful analogy would be a book -- yes, reading a book is effortful, and yes, "almost everyone" does not read. Still, I think most wouldn't consider it crazy to say that reading can be "fun."
Based on the Intelligence vs. Cost graph, not clear to me why anyone would use Terra? Luna looks quite interesting though, happy to see OpenAI still serving the more budget-oriented side of the market (seems like Anthropic and Google have lost interest there).
This doesn't seem that big of a deal to me? I mean, in any other area where I want an assessment of a product, I'm not going to trust what the product producer says about it at face value -- obviously they're going to be biased. This is the whole raison d'etre for independent testing, like https://artificialanalysis.ai.
So interesting to read about AI perceptions from the other side. In my mind, the problems with these drafts were moreso that the writers using AI could not (or would not) actually engage and improve them when given feedback, not necessarily that their writing process did not match the traditional one.
I'm not sure I see any inherent problem with publishing books written with the help of AI. As with software, I don't really care much how it's made, I care bout my experience with the finished product.
"Is it worth the $?" is ultimately the question that will be asked of anything one pays for, regardless of how exactly it was produced.
Interesting perspective. In my experience the risk is actually that it results in alert fatigue, which means that drivers that would otherwise pay attention to such an alert no longer do.
I rarely make cards by hand anymore. I would recommend forking something like https://github.com/jasperket/clanki and editing it (perhaps with an agent) so that it works exactly to your liking.
I think you're measuring one method against a goal it wasn't set for. I assume you are referring to Mammoth direct air capture in your comment -- Stratos in Texas will soon be up, at ~15x the size of Mammoth. But DAC is only one method, and it's the hardest to scale. Just look at options outside of DAC, like Vaulted Deep, whose costs (financially and energy-wise) are far lower.
I do agree that it's unlikely that we can ignore reduction and just depend on purely scaling capture, especially if we care about avoiding more negative climate effects as the scaling goes on. But to say it is "completely infeasible" is not accurate.
It's fair to call out issues with the tool. But I think for individuals searching for jobs, using LLMs as the scapegoat for why it's hard to find a role is not terribly helpful.
In my experience, cold-applying has always worked essentially as a black hole, and LLMs haven't changed that much. The reality is that alternative avenues are always necessary to get the job you want. That could be a third-party recruiter; reaching out to a hiring manager on LinkedIn; or using your network to get referrals. Those continue to work whether the company is using a bone-headed tool like this or not.
Let's play this out further. How about high school, should there be grades there? Tests at all levels also typically involve a grade / metric -- are those included too?
Do you mean very well? Lots of proof out there that it works; Fin, Sierra and others already operate on a value based pricing model where they only get paid if the AI actually resolves issues.
If you can't concentrate while people are working on computers near you, I don't think you'll do well in any workplace that is based around knowledge work.
We use three classes of signals:
* Tool-calling success and reliability from real traffic
* Provider performance metrics such as throughput and latency
* Benchmark and evaluation data as it becomes available
Who's paying the $50k? I don't see how it makes sense to pay that much for a home-grown setup when I could pay <$5k/year total for both of the two best frontier models at effectively unlimited usage.