HackerTrans
TopNewTrendsCommentsPastAskShowJobs

stephendause

no profile record

Submissions

Six Principles for More Rigorous Evaluation of Cognitive Capacities

aiguide.substack.com
2 points·by stephendause·há 6 meses·0 comments

comments

stephendause
·há 6 meses·discuss
I don't know; I've never done something like that. If no one else answers, you can always ask Claude itself (or another chatbot). This kind of thing seems tricky to get right, so be careful!
stephendause
·há 6 meses·discuss
There is an example of [dis]allowing certain bash commands here: https://code.claude.com/docs/en/settings

As for queries, you might be able to achieve the same thing with usage of command-line tools if it's a `sqlite` database (I am not sure about other SQL DBs). If you want even more control than the settings.json allows, you can use the claude code SDK.
stephendause
·há 6 meses·discuss
Your story sounds similar to mine. There are some parts of programming at which I know I will never excel. I also don't have time in my life to spends lots of hours outside of work developing my skills. I think it's important to realize that the median software engineer is probably not doing these things either. Maybe the top 10% are? Something like that would be my guess. It's okay to not be in the top 10%!
stephendause
·há 6 meses·discuss
Beauty is in the eye of the beholder. My personal taste for the presentation of a piece of writing is that less is more. I usually find artwork that accompanies a text to be distracting. I love reading work that can stand on its own, invoking images in the mind. I also dislike animations that seem to be made for a certain scroll speed.

Having said all of that, I certainly don't think it's bad, nor is it a commentary on the arguments being made. It's just not my cup of tea.
stephendause
·há 7 meses·discuss
Why? To me, hosting previous versions of an article in a public git repo adds transparency. Or perhaps you are talking about GitHub specifically?
stephendause
·há 7 meses·discuss
> - Is the work faster? It sounds like it’s not faster.

The author didn't discuss the speed of the work very much. It is certainly true that LLMs can write code faster than humans, and sometimes that works well. What would be nice is an analysis of the productivity gains from LLM-assisted coding in terms of how long it took to do an entire project, start to finish.
stephendause
·há 9 meses·discuss
This is total speculation, but my guess is that human reviewers of AI-written text (whether code or natural language) are more likely to think that the text with emoji check marks, or dart-targets, or whatever, are correct. (My understanding is that many of these models are fine-tuned using humans who manually review their outputs.) In other words, LLMs were inadvertently trained to seem correct, and a little message that says "Boom! Task complete! How else may I help?" subconsciously leads you to think it's correct.
stephendause
·há 9 meses·discuss
This is a good insight, but do you know of better ways to measure machines' abilities to solve problems in the "messy real world"?
stephendause
·há 9 meses·discuss
I think it's not only the potential for self-improvement of AGI that is revolutionary. Even having an AGI that one could clone for a reasonable cost and have it work nonstop with its clones on any number of economically-valuable problems would be very revolutionary.
stephendause
·há 10 meses·discuss
This is a key question in my opinion. It's one of the things that make benchmarking the SWE capabilities of LLMs difficult. It's usually impossible to know whether the LLM has seen a problem before, and coming up with new, representative problem sets is time-consuming.
stephendause
·há 10 meses·discuss
Jonathan Haidt has a lot of good material on this. He is leading the charge in encouraging parents to delay giving their child a phone until high school and not allowing them to have social media accounts until age 16.

https://www.goodmorningamerica.com/family/story/author-sugge...