I think (related to the threads below) properly running evals in the state of the art models is likely outside the budget for most individuals. It's undoubtedly the right thing.
It would be very useful for companies to isolate interesting programming challenges in their past and publish evals on them (without revealing the actual codebase). In theory companies adopting these models should already be doing this to evaluate cost/benefit for each model, so it would be a matter of publishing them on a regular basis.
I do love the appeal to bread making. It's a wonderful example. If people haven't made french bread by hand, it's a humbling exercise.
Recipes of course have evolved too. Old roman recipes were merely a list of ingredients. Water, flour, salt, yeast.
Written steps came after, then photos, videos, gradually replacing hands on training / kneading.
There are now recipes as code running sour dough assembly lines. Certainly capturing much more detail in technique than even a well made video. But I bet there is still human QA at the end judging "is this bread what folks expect?"
I suspect that in order of complexity you'll get "can I attempt to follow each step", "can I follow the intention of each step and understand if I've failed to meet it" (mitigated by using more specific and detailed steps) "can I follow the intention of the recipe itself - can I add or modify steps that are missing to give the ideal form of sour dough" (maybe you show a machine what good bread looks like, moisture content, crunch?) Those mostly overlap with the 3 you've called out. But I'd add "why would anyone make bread?" Why the heck are we still mixing flour and water together. Why does this recipe exist? Great crusty sourdough requires them all.
I do think there's a chance open weight models have a bit of a moment with the costs of frontier models growing on business balance sheets. It's unfortunate from my "privacy loving" PoV that it's mostly Chinese models filling the gap. ( the top models on openrouter for instance ).
I have used Mistral models out of pure ideology for web agents and the like which aren't doing a lot of heavy lifting.
I tend to agree - making folks intending to interact with minors comply makes more sense.
That said, outside of the merits of this approach, I am dubious of any actual implementation given 2 points.
1) Protecting the youths will always be a leaky bucket. With disadvantaged youths possibly more at risk. Those exposed to non-compliant parents ("cool" parents who are ok with sharing unsuitable content) or lacking strong parental involvement, likely won't benefit a great deal from any implementation.
2) Anti privacy social networks stand to gain the most from targeting ads utilizing signals from most child safety acts. They also might be able to reduce some costs from moderation if they can make it someone else's problem. I'd argue the net social impact from these social networks is likely both more normalized and strongly negative for our youths than any smut.
On the balance I'd say we are better off investing our energy in other places.
If you walk through a forest there are billions of little things from creatures to bits of dna just looking to pass on their particular brand of biologic layout to another generation. They would love to involve you.
on a world swirling through the chaos of hard and ephemeral matter one big rock away from a new trajectory.
No, we in no way created the complexity. We have some baby complexities we've created sometimes for good reasons, sometimes not. We have complexities we've created to understand the world. Some to try and improve how we live. Some to mimic how we see existing systems or control others. It's all just a drop in the bucket.
I happen to subscribe to the general belief that we should aim to make life suck less for others in the future. I think we do that by learning more, not trying to back step into ignorance and forget how we got here. That is a dead end. Our present complexity of life is just the farthest we've got so far. Not very far at all.
I know magic has a nice Arthur C. Clarke ring to it, but I think arguing about magic obscures the actual argument.
It's about layers of abstraction, the need to understand them, modify them, know what is leaking etc.
I think people sometimes substitute magic when they mean "I suddenly need to learn a lower layer I assumed was much less complex ". I don't think anyone is calling the linux kernal magic. Everyone assumes it's complex.
Another use of "magic" is when you find yourself debugging a lower layer because the abstraction breaks in some way. If it's highly abstracted and the inner loop gives you few starting points ( while (???) pickupWorkFromAnyWhere() )). It can feel kafkaesque.
I sleep just fine not knowing how much software I use exactly works. It's the layers closest to application code that I wish were more friendly to the casual debugger.
I think the context matters here - for SEO heavy marketing pages I still see google only executing a full browser based crawl for a subset of pages. So SSR matters for the remainder.
You have many great replies about specific methods - but I found the most important tip wasn't where I was looking. Tools, software, books, methods all can come later. The most important part for me was creating the time for cleanup and organization. Physically and mentally.
Jumping from thing to thing without time set aside for "stop, reflect, adjust" makes it very challenging to make
changes. I realized that you don't become organized if you don't spend time on it. Picking up physical messes. Thinking about what was important in your day vs what you got done. How the week went. Writing it down.
I found it was only after I started consistently putting time aside to catchup, think and adjust that I started being able to consider if any particular methods would be helpful. Parts of GTD have helped me (capture first) - but the aha moment really
came before that.
If you want to be organized, put time into
reflecting and adjusting (eg. organizing) the critical parts of your life. Once a day, every day. Maybe more than once a day. Then use one of those to reflect on the week. Not reading about it or endlessly sorting the books on your shelf, but focusing intently on stuff you'll remember 20 years from now.
If there is any truth to OpenAi having filters for the Rothschilds, I'd guess that OpenAI wants to stay clear of repeating or even hallucinating additions to conspiracy theories. I would hope at least.
Always interesting to compare how things work across industries, but comparing tech with sports I think is both common and problematic.
Problematic because it leads ICs to think of themselves as the quarterback throwing or wide receiver making the game winning catch with 1 minute left, getting lifted up into the air, renegotiating a contract for millions more, retiring early, getting inducted to the hall of fame etc.
Why is this so problematic? I think it leads engineers to overvalue the short term wins (getting a particularly tricky implementation correct) sometimes at the cost of their health and wellbeing if they work nights and weekends to get it done. And no one is watching it live at the edge of their seats. Even more distressing is that having more junior ICs have to pull heroics to keep the business alive is a tremendous anti-pattern. The point of management is make the right decisions so the organization steers clear of asking their least experienced contributors to damage their health on a regular basis (...really at all!).
A well run org couples decision making and seniority. If an IC is making a lot of org impacting decisions they are probably as senior or more-so than most line managers (or should be promoted to be if this happens regularly) and are comped the same way. Now it would be nice if decision making was as transparent as coaches making the right/wrong play (the offensive coordinator calling a running play no one watching thought was a good idea). In an ideal org failures or inaction would be more transparent. If a manager is basically not making any decisions (adding no value) there are certainly management failures at several levels (maybe up to the top) preventing folks from getting upset about it. Again ideally audibles are a very occasional exception.
Where does money comes from? In sports, teams competing are the product. Players are kinda like features and marketing all built into one. Companies spend a significant portion of all their money on building product and marketing it. I think IC engineers are probably closer to the support staff than the players on the field...the comparison is problematic.
And which hall of fame? - the engineers in our hall of fame are the folks doing things for the first time, trailblazing, largely researchers (Turing awards etc). There is no "got 4 hours of sleep for months delivering a poorly planned product and got an autoimmune disease" hall of fame.
The convo has been a bit light on examples - I think a canonical example of how to achieve this can be found in ACID databases and high uptime mainframes. Definitely doable, but if performance and large scale data is also a concern, doing it from scratch (eg. starting with file handles) is probably a 5-10+ year process to write your own stable db.
There is a lot to be said for getting outside the four walls of a business (or org) to evaluate things. If it's not visible outside those walls (software buggy enough to lose customers) and doesn't introduce significant future risk to the business (competition can move faster than you) it's probably good enough. The real trick of course is predicting and communicating why you think one of these is true. It's an essential problem of commercial software dev.
Not to pick on you specifically, but I tend to agree with other posters that testing (automated or otherwise) is just an element of programming. Like all elements it needs to be done to taste, but it's pretty essential.
A line of questioning: Do you have time to write clear code? Time for comments? Time to manually test your changes hundreds of times? Time to refactor existing code when adding new code? Time to remove dead code? Time to automate the testing while you still have the little state machine you are working on in your head? Time to add observability to spot performance issues? Time to consider your rollout plan? Time to keep on top of changes which are in use? etc...
Automated tests (including unit tests) are just one part of writing correct code. If you are asked, "How long is it going to take?" that implies finishing all aspects of coding required to get something correct into use. You prioritize that, not anyone else.
Take it with a grain of salt, since I don't own the hardware - but did have two points of feedback 1) you may want a more complex/interesting thumbnail- I almost scrolled by since it looked fake/computer generated. The 3rd photo on the lander (red-ish foreground, blue-ish background) looks more engaging to me. 2) The lander would be a good bit more engaging if I could click to play the videos.
+1 to this. Focusing on usecases would be great. ex: I wanted to see what features were available for your sheets module.
Was willing to spend 5 minutes on a walk. Tried the web app - ios safari is not supported :( downloaded the ios app and registered. Got a totally blank app - no onboarding, no template / samples, no obvious way to import from my existing google sheets to see how things scaled. I added a datasource and a few fields (which felt confusing) and my walk was over.
It would be very useful for companies to isolate interesting programming challenges in their past and publish evals on them (without revealing the actual codebase). In theory companies adopting these models should already be doing this to evaluate cost/benefit for each model, so it would be a matter of publishing them on a regular basis.