Highly recommend the extremely good multipart documentary All Watched Over By Machines Of Loving Grace by Adam Curtis for a fun and wide ranging if slightly silly look at the nexus of Greenspan, Ayn Rand, Silicon Valley, computer technology etc
Being able to cycle to and from work across a nice university campus, through fields and trees and in good weather, makes me feel as rich as any human who has ever lived
The training process literally ingests the majority of text on the internet, including a huge volume of SEO garbage, and seeks to create a self-consistent compressed model of that. This is totally imperfect of course but is also likely more truthful than the median Google result, because of the incentive for self-consistency and coherence that is created by the reward function as well as during RL.
Imagine that you had 1,000 years to read every Google result on a particular topic, and literally infinite patience. You would read a lot of rubbish but ultimately you are a smart person, you would figure out the underlying truth and likely produce something that is more valuable than the average or even the sum of the parts.
Be careful about how you interpret that paper. It looks really impressive -- real neurons in a petri dish seem to successfully (if amateurishly) murk a few imps.
So there is an entire pytorch stack wrapped around the mysterious little blob of neurons -- they aren't just wired straight into WASD. There is a conventional convnet-based encoder, running on a GPU, in the critical path. The README tries to argue that the "neurons are doing the learning" but to my dilettante, critical eye it really looks as though there is a hell of a lot of learning happening in the convnet also.
Are the neurons learning to play doom, or are they learning to inject ever so slightly more effective noise into the critical path? Would this work just as well if we replaced the neurons with some other non-markovian sludge? The authors do ablation experiments to try to get to the bottom of this but I can't really tell how compelling the results are (due to my own ignorance/stupidity of course)
So far the wonders of claude/codex have been mostly constrained to applications that are built within the boundary conditions of existing libraries -- the models make direct use of the good work that humans have done to date to build Python, `requests`, `ffmpeg`, you name it.
But I'm excited for the (I think inevitable) stage where the shoggoth starts to reach outside those constraints -- rewriting, patching, renaming, rebuilding libraries, DLLs, binaries -- and we move into a regime where the libraries dissolve, the application floats on top of the shifting sands of an ever more efficient, secure, unified and totally inhuman technology stack.
Obviously this is a horrifying idea in some ways (interpretability, security etc), but it's also not obvious to me that it can't work, especially if there are dedicated, centralized efforts to do this. it's also not clear that interpretability is necessarily mutually exclusive with full slopification/machine rewrite of decades of foundational, incremental development
Journalists are so funny man. "Last week we told you that AI is fake and fraud. But we just learned something fascinating. A few months after everyone else was talking about it, we uncovered an amazing scoop: the companies which raised a lot of capital, are also doing tens of billions of dollars of revenue. So now we're starting to thing it might not all be a scam! Tune in next week for when we say it's all 100% fraudulent again."
I think that was true when you could rely on good old Moore’s law to make the heavy iron quickly obsolete but I also think those days are coming to an end
Continue to believe that Cerebras is one of the most underrated companies of our time. It's a dinner-plate sized chip. It actually works. It's actually much faster than anything else for real workloads. Amazing
I love this… have been thinking about exactly this technology for years but combined with phased array directional loudspeaker and shotgun mic. Deploy during major political speech, instantly shut down brain of speaker, would appear to be an internal malfunction
Google Maps is the mind killer. We all worry about social media controlling the way we think, feel, vote etc. but Google Maps literally manipulates where people physically go in real life, what they do on holiday, where they hang out, what they eat etc. I got so sick of feeling like a four point five star Google Maps automaton I had to mostly stop with it. In addition to OSM, personal recommendations etc. the best substitute for me for a 4.5 star review is my nose, eyes and ears