The best part of this workflow - which I see often - is that by having someone build custom software to automate some process they often step back away from the process being their job. That eventually translates into them understanding that some (or sometimes most or all) of that process is not needed. There are so many corporate processes that were implemented and then become the way... and if there are people who identify that process as being their job those people resist attempts to optimize that process.
I have seem several people use AI to write apps to automate a process and along they way finally ask the question 'do we even need this process?'.
What I was commenting on was the concept that a small model at home is somehow more efficient. To make a reasonable and fair comparison you would compare many people running a small model at home vs those same people using what would likely be a shared resource in a datacenter.
The core concept is that tokens/watt is tokens/watt ( for a given model of course ). A computer at home is actually less efficient overall because most of the time it is not doing tokens but still using a small footprint of power.
The revenue pressure is an interesting problem , but I suspect the actual demand math will be much more complicated.
I find local models interesting for sure, and run several on my own personal DGX cluster. I am however most certainly not power efficient!
"But there’s another challenge: local LLMs. It’s already possible to run LLMs on local hardware, and that’s only going to get easier in the future. Apple’s M-series chips are extremely good at doing this today. Open weight (read: free) models are widely available and good enough that most people probably couldn’t tell the difference. They also have the benefits of running on hardware that’s sipping power most of the time, rather than slurping it down in massive data centres."
This is such an odd and illogical conclusion. If a smaller model can be sufficient (which is not something I would have said), that smaller model can be ran in a datacenter. The idea that a small model running at home is 'sipping' while that same small model in a datacenter is 'slurping' is absurd. The datacenter will have much greater overall efficiency in both power usage and total cost to implement. Of course if you compare a small home model to a DC frontier model the power usage is different, but so is the output.
That brings back memories! Riding in the Transport to pizza hut with tanks of refrigerant banging around in the back. I remember when Ardent appeared! It was such a cool machine, and to this day I occasionally scan e-bay to see if one comes up.
Indeed what made George special was that he was a broad 'engineer' first, and a specialist later. Everything was an engineering problem that could be solved. Perhaps most of all he believed in the students, hiring as many as he could get approved.
I found a 'bug' in the debugger on the NP1s that gave me a root escalation, and when he found out he said I should come work for him and that was that! About a year in we scrapped another older system and he had a spare disk that we put in the EE NP1 (en.ecn.purdue.edu) and mounted it as '/hogs' because I was always hogging disk space with stuff I downloaded. Funny enough I still have an exabyte tape backup of that drive.
That is awesome. The NP-1s were great. I spent lots of time working on en.ecn.purdue.edu - Some tape drivers, some maintenance, and lots of software projects - it was really cool that in those days everyone was on the same machine, working from terminals. Good times!
George was really into video stuff - he had stacks of 8mm video tapes in his office, and of course stacks of exabyte drives. He had many different cameras and was always trying new ones out. He was also a really early adopter of laserdiscs, and I have a few discs he gave me when I graduated.
Sad to hear! I worked for George for all of my undergraduate time at Purdue. He was an amazing boss with such a passion for all things unix. For a while he had the UNIX license plate on his minivan.
I am similar in that all of my interactions are with my real name and it is unique enough that just putting it into google will instantly identify me. There is one other 'jeff sponaugle' but I think he is far more annoyed with my presence than I would be with him.
On the plus side, someone will sometimes say while talking to me - oh your are that Subaru guy, or that youtube guy, or whatever and that is fun connection.
A fantastic read, and really interesting to see the use of Forth. I remember Forth having a bit of popularity in the 80s. This was such an amazing game, especially in that you felt like the world was huge with the encouragement to just explore.
The other game this reminds me of is a game for the TI99/4a called Tunnels of Doom. It was a cartridge game that also had a floppy or cassette data load. It had a dynamic dungeon creation so every time you played the game you got a new unique experience. That would be an equally challenging one to reverse engineer due to the oddity of the GROM/GPL architecture in the TI99/4a.
The study looks at a wide range of different tests spanning many different areas of expertise and output types. Some of the tests, like the web vis tasks used Sonnet not Opus (which was not out at the time). It is similar to testing a car to do many different things, but only one of the tests is the actual driving somehwhere and many of the others are based of the fabric used in the interior. This gives a very broad "96% failure" while missing the observation of the successes. Of course AI can't do everything, and nor can I.
One of the most interesting observations about AI is the timescale at which the favorite model and favorite task changes. Before November I found Sonnet to be interesting, but not moving that much of the needle. Once Opus came out it was clear the needle was not only moving, but moving fast.
Yea, that makes sense as you would need quite a bit of information across a reasonable temporal range if the identifying qualities are movement related. Very interesting.
This is a VERY controlled environment - and they used 20 passes of each person walking with direct knowledge of each person to train for identity. They did no tests with multiple people walking at the same time, or with any other external moving distortion effects (doors opening, etc) . This is very far from actual 'identification' of people in real public settings - and no doubt the cell phone everyone is carrying with them offers many orders of magnitude better opportunity. In a real crowded environment this would be nearly worthless.
The devices that reported BFI information were also stationary, and there were no extra devices transmitting information that would be conflicting.
Oh they are both loud and eat a ton of power. I think the 9305 is at least 800 watts at idle. That is the biggest downside of the retired datacenter gear... yo u really need a dedicated room with power, cooling, and sound isolation.
Agreed - the big thing is 100g is much much cheaper now as so much 100g gear is coming out of datacenters. So many of those older ConnectX4s and 5s, plus lots of switches and optics. 100g really is the new 10g for homelabs.
I'm using ConnectX5s for most, and some ConnectX4s in the older servers. Both of those cards have really come down in price in the used/ebay market. I have been playing around with some different optics - I have a bunch of CWDM4s which are very inexpensive and use a single SingleMode pair.... but of course they run hot so if you have them in servers without good air flow you might have problems.
I'm using mostly fiber just because the servers are connected to Cisco 9305 with 72 100g ports.
I have a 3 node proxmox setup on MS-01s using a 25G Thunderbolt ring for Ceph, and indeed it took a lot of hoops to get it working correctly and reliably. I did manage to get it such that nodes can go up and down without needing to unplug anything, and the dynamic routing works if a node disappears. Performance is pretty good, with a more realistic 20ish gbit/sec.
"because I can, and it is fun." The best answer! I am most of the way done with upgrading most of my homelab to 100G from 10G, but there really isn't a practical reason for it. 100G has dropped in price so much as datacenters are all about 400/800G now.
I have seem several people use AI to write apps to automate a process and along they way finally ask the question 'do we even need this process?'.
Regrettably this does not happen everywhere.