HackerTrans
TopNewTrendsCommentsPastAskShowJobs

gpjt

1,787 karmajoined 18 anni fa
https://www.gilesthomas.com/

Submissions

Building intuition about LLM parameter counts

gilesthomas.com
2 points·by gpjt·17 ore fa·0 comments

Poppy the training box, part 1: the beginnings

gilesthomas.com
3 points·by gpjt·3 giorni fa·0 comments

From bigrams to GPT-2, one component at a time (in Jax)

gilesthomas.com
1 points·by gpjt·3 giorni fa·0 comments

Building a Jax training loop for an LLM training run

gilesthomas.com
2 points·by gpjt·11 giorni fa·0 comments

Thoughts on Role Confusion

gilesthomas.com
3 points·by gpjt·17 giorni fa·0 comments

Flax debugging: making a hash of things

gilesthomas.com
2 points·by gpjt·24 giorni fa·0 comments

10Gb/s Ethernet: switching to a Broadcom SFP+ module

gilesthomas.com
195 points·by gpjt·25 giorni fa·170 comments

Jax: Commitment Issues

gilesthomas.com
4 points·by gpjt·26 giorni fa·0 comments

Jax Back Ends and Devices

gilesthomas.com
2 points·by gpjt·mese scorso·0 comments

Using Safetensors with Flax

gilesthomas.com
2 points·by gpjt·mese scorso·0 comments

First Looking into Jax

gilesthomas.com
3 points·by gpjt·mese scorso·0 comments

10Gb/s Ethernet: using mini-heatsinks with a 10GBASE-T SFP+ module

gilesthomas.com
3 points·by gpjt·2 mesi fa·0 comments

10Gb/s Ethernet: what I did to get it working in my home

gilesthomas.com
232 points·by gpjt·2 mesi fa·177 comments

10Gb Ethernet: what I had to (re)learn

gilesthomas.com
1 points·by gpjt·2 mesi fa·1 comments

LLM from scratch, part 33 – what I learned from the appendices

gilesthomas.com
5 points·by gpjt·3 mesi fa·0 comments

LLM from scratch (32l) – Interventions: updated instruction fine-tuning results

gilesthomas.com
1 points·by gpjt·3 mesi fa·0 comments

How an LLM becomes more coherent as we train it

gilesthomas.com
3 points·by gpjt·3 mesi fa·0 comments

LLM from scratch, part 32k – Interventions: gradient accumulation

gilesthomas.com
2 points·by gpjt·3 mesi fa·0 comments

Provision: LLM-powered server setup from Markdown

provision.sh
2 points·by gpjt·3 mesi fa·0 comments

LLM from scratch, part 32j – trying to train a better model in the cloud

gilesthomas.com
2 points·by gpjt·3 mesi fa·0 comments

comments

gpjt
·18 giorni fa·discuss
In the EU, at least one of the problems is regulatory/tax for digital services. They need to charge you the VAT rate for the country where you are based. For that, they need two pieces of evidence about your location. There are various things they can use for that -- telephone number, IP address, card billing address, and so on. If they can collect two that indicate the same country, they're safe -- but if all of them point in different directions then they could get in trouble during a tax audit.

Of course, for larger transactions you'd expect that a human in the loop could work with you to get the right info so that they would be covered. But I guess for Microsoft, their definition of "larger" might be more than a few grand...
gpjt
·24 giorni fa·discuss
OP here -- yes indeed. I ever do a new series of posts on upgrading my network to anything faster, the first one will probably be titled something like "25Gb/s Ethernet: how much it costs to rip out your CAT-6A and replace it with fibre".
gpjt
·25 giorni fa·discuss
OP here -- yes, 100%! Within my study it's DACs all the way. I only use 10GBASE-T where it's the only option: through the wall cabling, and from the connector on my ISP's crappy router to the mini-PC that acts as the real router.
gpjt
·25 giorni fa·discuss
Thanks for making the comment! It saved me having to bolt a USB fan over the old, misbehaving SFP+ module or something similarly silly...
gpjt
·28 giorni fa·discuss
Doesn't say anything about citizenship though. There are plenty of US residents who are not citizens. And a lot of people abroad appear to use US billing address credit cards -- in my last company we had hundreds of people with the same US billing address who appeared to be managing Africa-focused businesses and used IPs that matched that.
gpjt
·mese scorso·discuss
Excellent points. It's also worth noting that many people don't wind up working in the field they studied at university. CS grads have probably been the exception in recent years, because the industry has been booming, but the two most successful entrepreneurs I know studied philosophy and art history. A friend who is very senior in recruitment studied economics.
gpjt
·mese scorso·discuss
That is an excellent point so long as you don't take it too far! Lisp/Haskell/Erlang yes. INTERCAL/Brainfuck less so...
gpjt
·mese scorso·discuss
Hmmm. "Comforting haze" seems dubious. My grandmother had Alzheimer's and at least from the outside it seemed like a bad LSD trip that never ended. She didn't understand what was happening and was scared.

Sample size of one and anecdotal of course, but...
gpjt
·mese scorso·discuss
Love the verbification!
gpjt
·2 mesi fa·discuss
I was thinking the same. It's a simple idea, heavily over-explained. The code is similar, massively overengineered for such a simple test.
gpjt
·2 mesi fa·discuss
I guess if it's on the inside of a water-cooling loop, you should be OK if the water is pure enough. I don't know how hard that "enough" would be, though.
gpjt
·3 mesi fa·discuss
Yup, agreed -- it's amazing how close they are getting! I was just wondering if there was some true frontier non-US model that I'd missed.
gpjt
·3 mesi fa·discuss
Thanks for the link -- I read that when it was published, then the other day I wanted to send it to someone but I'd forgotten where it was.
gpjt
·3 mesi fa·discuss
Which are not? There are Chinese models that are only months behind, which is impressive -- but they are still behind.
gpjt
·3 mesi fa·discuss
I have it running on a Proxmox VM. It basically just sends me summaries -- it reads a bunch of RSS feeds I pointed it to (news, tech, etc) and gives me a daily summary, along with an image of the day based on that. It also sends me recommendations for times to go for a run based on the weather and my Strava activity, daily recommendations for stargazing (what's visible and when, weather, etc), and a couple of daily reminders for things I tend to forget.

I'm using Claude as the model, though, so it's smart but pricey. Should configure it to use different models for different things, but it's trickier than I would have expected to do that.
gpjt
·3 mesi fa·discuss
Wait, that's it? Seven paragraphs, all short? Two quotes, one from some anonymous MS exec? Is the site sending some minimal version of the article to me because I'm using Brave, or is this the lowest-content article I've seen in weeks (and I'm on Twitter)?
gpjt
·4 mesi fa·discuss
It's a bit of a double edged sword. As someone who smoked and found it impossible to quit for decades, I'm very happy to have been able to switch to (reuasable) vaping. It's probably added years to my life expectancy.

OTOH the upsurge in nicotine use amongst young people feels suboptimal, and disposable vapes are a scourge.
gpjt
·4 mesi fa·discuss
Looked to me like it was trying to work out whether it was edible. Sensible behaviour for an animal in a world where unfamiliar edible things appear from time to time.
gpjt
·7 mesi fa·discuss
Not sure that every browser advertises English, but mine certainly does. However, as I'm in Portugal, many websites ignore what my browser says and send me to translated versions, I assume based on my IP. That causes problems because the translations are often quite bad, and they do it with redirects to PT URLs so I can't share links with people who don't speak the language.
gpjt
·7 mesi fa·discuss
Awesome, thanks! I'm still doing trains on the big machines right now (hopefully will write up over xmas) but I think once I've worked out the sweet spot for memgatokens per dollar for this model, it's time to start tweaking the other controls -- LR and cosine variation of it, as you said, and also dropout, bias, weight tying, and definitely gradient clipping (which should at least get better bang for the buck from time/$ spent). I'll leave it to Google to follow up Chinchilla with a "best batch size across a thousand trained models" paper ;-)