HackerTrans
TopNewTrendsCommentsPastAskShowJobs

gpjt

1,787 karmajoined 18 years ago
https://www.gilesthomas.com/

Submissions

Building intuition about LLM parameter counts

gilesthomas.com
2 points·by gpjt·22 hours ago·0 comments

Poppy the training box, part 1: the beginnings

gilesthomas.com
3 points·by gpjt·3 days ago·0 comments

From bigrams to GPT-2, one component at a time (in Jax)

gilesthomas.com
1 points·by gpjt·3 days ago·0 comments

Building a Jax training loop for an LLM training run

gilesthomas.com
2 points·by gpjt·11 days ago·0 comments

Thoughts on Role Confusion

gilesthomas.com
3 points·by gpjt·17 days ago·0 comments

Flax debugging: making a hash of things

gilesthomas.com
2 points·by gpjt·25 days ago·0 comments

10Gb/s Ethernet: switching to a Broadcom SFP+ module

gilesthomas.com
195 points·by gpjt·25 days ago·170 comments

Jax: Commitment Issues

gilesthomas.com
4 points·by gpjt·26 days ago·0 comments

Jax Back Ends and Devices

gilesthomas.com
2 points·by gpjt·last month·0 comments

Using Safetensors with Flax

gilesthomas.com
2 points·by gpjt·last month·0 comments

First Looking into Jax

gilesthomas.com
3 points·by gpjt·last month·0 comments

10Gb/s Ethernet: using mini-heatsinks with a 10GBASE-T SFP+ module

gilesthomas.com
3 points·by gpjt·2 months ago·0 comments

10Gb/s Ethernet: what I did to get it working in my home

gilesthomas.com
232 points·by gpjt·2 months ago·177 comments

10Gb Ethernet: what I had to (re)learn

gilesthomas.com
1 points·by gpjt·2 months ago·1 comments

LLM from scratch, part 33 – what I learned from the appendices

gilesthomas.com
5 points·by gpjt·3 months ago·0 comments

LLM from scratch (32l) – Interventions: updated instruction fine-tuning results

gilesthomas.com
1 points·by gpjt·3 months ago·0 comments

How an LLM becomes more coherent as we train it

gilesthomas.com
3 points·by gpjt·3 months ago·0 comments

LLM from scratch, part 32k – Interventions: gradient accumulation

gilesthomas.com
2 points·by gpjt·3 months ago·0 comments

Provision: LLM-powered server setup from Markdown

provision.sh
2 points·by gpjt·3 months ago·0 comments

LLM from scratch, part 32j – trying to train a better model in the cloud

gilesthomas.com
2 points·by gpjt·3 months ago·0 comments

comments

gpjt
·18 days ago·discuss
In the EU, at least one of the problems is regulatory/tax for digital services. They need to charge you the VAT rate for the country where you are based. For that, they need two pieces of evidence about your location. There are various things they can use for that -- telephone number, IP address, card billing address, and so on. If they can collect two that indicate the same country, they're safe -- but if all of them point in different directions then they could get in trouble during a tax audit.

Of course, for larger transactions you'd expect that a human in the loop could work with you to get the right info so that they would be covered. But I guess for Microsoft, their definition of "larger" might be more than a few grand...
gpjt
·25 days ago·discuss
OP here -- yes indeed. I ever do a new series of posts on upgrading my network to anything faster, the first one will probably be titled something like "25Gb/s Ethernet: how much it costs to rip out your CAT-6A and replace it with fibre".
gpjt
·25 days ago·discuss
OP here -- yes, 100%! Within my study it's DACs all the way. I only use 10GBASE-T where it's the only option: through the wall cabling, and from the connector on my ISP's crappy router to the mini-PC that acts as the real router.
gpjt
·25 days ago·discuss
Thanks for making the comment! It saved me having to bolt a USB fan over the old, misbehaving SFP+ module or something similarly silly...
gpjt
·28 days ago·discuss
Doesn't say anything about citizenship though. There are plenty of US residents who are not citizens. And a lot of people abroad appear to use US billing address credit cards -- in my last company we had hundreds of people with the same US billing address who appeared to be managing Africa-focused businesses and used IPs that matched that.
gpjt
·last month·discuss
Excellent points. It's also worth noting that many people don't wind up working in the field they studied at university. CS grads have probably been the exception in recent years, because the industry has been booming, but the two most successful entrepreneurs I know studied philosophy and art history. A friend who is very senior in recruitment studied economics.
gpjt
·last month·discuss
That is an excellent point so long as you don't take it too far! Lisp/Haskell/Erlang yes. INTERCAL/Brainfuck less so...
gpjt
·2 months ago·discuss
Hmmm. "Comforting haze" seems dubious. My grandmother had Alzheimer's and at least from the outside it seemed like a bad LSD trip that never ended. She didn't understand what was happening and was scared.

Sample size of one and anecdotal of course, but...
gpjt
·2 months ago·discuss
Love the verbification!
gpjt
·2 months ago·discuss
I was thinking the same. It's a simple idea, heavily over-explained. The code is similar, massively overengineered for such a simple test.
gpjt
·2 months ago·discuss
I guess if it's on the inside of a water-cooling loop, you should be OK if the water is pure enough. I don't know how hard that "enough" would be, though.
gpjt
·3 months ago·discuss
Yup, agreed -- it's amazing how close they are getting! I was just wondering if there was some true frontier non-US model that I'd missed.
gpjt
·3 months ago·discuss
Thanks for the link -- I read that when it was published, then the other day I wanted to send it to someone but I'd forgotten where it was.
gpjt
·3 months ago·discuss
Which are not? There are Chinese models that are only months behind, which is impressive -- but they are still behind.
gpjt
·3 months ago·discuss
I have it running on a Proxmox VM. It basically just sends me summaries -- it reads a bunch of RSS feeds I pointed it to (news, tech, etc) and gives me a daily summary, along with an image of the day based on that. It also sends me recommendations for times to go for a run based on the weather and my Strava activity, daily recommendations for stargazing (what's visible and when, weather, etc), and a couple of daily reminders for things I tend to forget.

I'm using Claude as the model, though, so it's smart but pricey. Should configure it to use different models for different things, but it's trickier than I would have expected to do that.
gpjt
·3 months ago·discuss
Wait, that's it? Seven paragraphs, all short? Two quotes, one from some anonymous MS exec? Is the site sending some minimal version of the article to me because I'm using Brave, or is this the lowest-content article I've seen in weeks (and I'm on Twitter)?
gpjt
·4 months ago·discuss
It's a bit of a double edged sword. As someone who smoked and found it impossible to quit for decades, I'm very happy to have been able to switch to (reuasable) vaping. It's probably added years to my life expectancy.

OTOH the upsurge in nicotine use amongst young people feels suboptimal, and disposable vapes are a scourge.
gpjt
·4 months ago·discuss
Looked to me like it was trying to work out whether it was edible. Sensible behaviour for an animal in a world where unfamiliar edible things appear from time to time.
gpjt
·7 months ago·discuss
Not sure that every browser advertises English, but mine certainly does. However, as I'm in Portugal, many websites ignore what my browser says and send me to translated versions, I assume based on my IP. That causes problems because the translations are often quite bad, and they do it with redirects to PT URLs so I can't share links with people who don't speak the language.
gpjt
·7 months ago·discuss
Awesome, thanks! I'm still doing trains on the big machines right now (hopefully will write up over xmas) but I think once I've worked out the sweet spot for memgatokens per dollar for this model, it's time to start tweaking the other controls -- LR and cosine variation of it, as you said, and also dropout, bias, weight tying, and definitely gradient clipping (which should at least get better bang for the buck from time/$ spent). I'll leave it to Google to follow up Chinchilla with a "best batch size across a thousand trained models" paper ;-)