HackerTrans
TopNewTrendsCommentsPastAskShowJobs

gpjt

1,787 karmajoined 18 yıl önce
https://www.gilesthomas.com/

Submissions

Building intuition about LLM parameter counts

gilesthomas.com
2 points·by gpjt·16 saat önce·0 comments

Poppy the training box, part 1: the beginnings

gilesthomas.com
3 points·by gpjt·3 gün önce·0 comments

From bigrams to GPT-2, one component at a time (in Jax)

gilesthomas.com
1 points·by gpjt·3 gün önce·0 comments

Building a Jax training loop for an LLM training run

gilesthomas.com
2 points·by gpjt·11 gün önce·0 comments

Thoughts on Role Confusion

gilesthomas.com
3 points·by gpjt·17 gün önce·0 comments

Flax debugging: making a hash of things

gilesthomas.com
2 points·by gpjt·24 gün önce·0 comments

10Gb/s Ethernet: switching to a Broadcom SFP+ module

gilesthomas.com
195 points·by gpjt·25 gün önce·170 comments

Jax: Commitment Issues

gilesthomas.com
4 points·by gpjt·26 gün önce·0 comments

Jax Back Ends and Devices

gilesthomas.com
2 points·by gpjt·geçen ay·0 comments

Using Safetensors with Flax

gilesthomas.com
2 points·by gpjt·geçen ay·0 comments

First Looking into Jax

gilesthomas.com
3 points·by gpjt·geçen ay·0 comments

10Gb/s Ethernet: using mini-heatsinks with a 10GBASE-T SFP+ module

gilesthomas.com
3 points·by gpjt·2 ay önce·0 comments

10Gb/s Ethernet: what I did to get it working in my home

gilesthomas.com
232 points·by gpjt·2 ay önce·177 comments

10Gb Ethernet: what I had to (re)learn

gilesthomas.com
1 points·by gpjt·2 ay önce·1 comments

LLM from scratch, part 33 – what I learned from the appendices

gilesthomas.com
5 points·by gpjt·3 ay önce·0 comments

LLM from scratch (32l) – Interventions: updated instruction fine-tuning results

gilesthomas.com
1 points·by gpjt·3 ay önce·0 comments

How an LLM becomes more coherent as we train it

gilesthomas.com
3 points·by gpjt·3 ay önce·0 comments

LLM from scratch, part 32k – Interventions: gradient accumulation

gilesthomas.com
2 points·by gpjt·3 ay önce·0 comments

Provision: LLM-powered server setup from Markdown

provision.sh
2 points·by gpjt·3 ay önce·0 comments

LLM from scratch, part 32j – trying to train a better model in the cloud

gilesthomas.com
2 points·by gpjt·3 ay önce·0 comments

comments

gpjt
·18 gün önce·discuss
In the EU, at least one of the problems is regulatory/tax for digital services. They need to charge you the VAT rate for the country where you are based. For that, they need two pieces of evidence about your location. There are various things they can use for that -- telephone number, IP address, card billing address, and so on. If they can collect two that indicate the same country, they're safe -- but if all of them point in different directions then they could get in trouble during a tax audit.

Of course, for larger transactions you'd expect that a human in the loop could work with you to get the right info so that they would be covered. But I guess for Microsoft, their definition of "larger" might be more than a few grand...
gpjt
·24 gün önce·discuss
OP here -- yes indeed. I ever do a new series of posts on upgrading my network to anything faster, the first one will probably be titled something like "25Gb/s Ethernet: how much it costs to rip out your CAT-6A and replace it with fibre".
gpjt
·25 gün önce·discuss
OP here -- yes, 100%! Within my study it's DACs all the way. I only use 10GBASE-T where it's the only option: through the wall cabling, and from the connector on my ISP's crappy router to the mini-PC that acts as the real router.
gpjt
·25 gün önce·discuss
Thanks for making the comment! It saved me having to bolt a USB fan over the old, misbehaving SFP+ module or something similarly silly...
gpjt
·28 gün önce·discuss
Doesn't say anything about citizenship though. There are plenty of US residents who are not citizens. And a lot of people abroad appear to use US billing address credit cards -- in my last company we had hundreds of people with the same US billing address who appeared to be managing Africa-focused businesses and used IPs that matched that.
gpjt
·geçen ay·discuss
Excellent points. It's also worth noting that many people don't wind up working in the field they studied at university. CS grads have probably been the exception in recent years, because the industry has been booming, but the two most successful entrepreneurs I know studied philosophy and art history. A friend who is very senior in recruitment studied economics.
gpjt
·geçen ay·discuss
That is an excellent point so long as you don't take it too far! Lisp/Haskell/Erlang yes. INTERCAL/Brainfuck less so...
gpjt
·geçen ay·discuss
Hmmm. "Comforting haze" seems dubious. My grandmother had Alzheimer's and at least from the outside it seemed like a bad LSD trip that never ended. She didn't understand what was happening and was scared.

Sample size of one and anecdotal of course, but...
gpjt
·geçen ay·discuss
Love the verbification!
gpjt
·2 ay önce·discuss
I was thinking the same. It's a simple idea, heavily over-explained. The code is similar, massively overengineered for such a simple test.
gpjt
·2 ay önce·discuss
I guess if it's on the inside of a water-cooling loop, you should be OK if the water is pure enough. I don't know how hard that "enough" would be, though.
gpjt
·3 ay önce·discuss
Yup, agreed -- it's amazing how close they are getting! I was just wondering if there was some true frontier non-US model that I'd missed.
gpjt
·3 ay önce·discuss
Thanks for the link -- I read that when it was published, then the other day I wanted to send it to someone but I'd forgotten where it was.
gpjt
·3 ay önce·discuss
Which are not? There are Chinese models that are only months behind, which is impressive -- but they are still behind.
gpjt
·3 ay önce·discuss
I have it running on a Proxmox VM. It basically just sends me summaries -- it reads a bunch of RSS feeds I pointed it to (news, tech, etc) and gives me a daily summary, along with an image of the day based on that. It also sends me recommendations for times to go for a run based on the weather and my Strava activity, daily recommendations for stargazing (what's visible and when, weather, etc), and a couple of daily reminders for things I tend to forget.

I'm using Claude as the model, though, so it's smart but pricey. Should configure it to use different models for different things, but it's trickier than I would have expected to do that.
gpjt
·3 ay önce·discuss
Wait, that's it? Seven paragraphs, all short? Two quotes, one from some anonymous MS exec? Is the site sending some minimal version of the article to me because I'm using Brave, or is this the lowest-content article I've seen in weeks (and I'm on Twitter)?
gpjt
·4 ay önce·discuss
It's a bit of a double edged sword. As someone who smoked and found it impossible to quit for decades, I'm very happy to have been able to switch to (reuasable) vaping. It's probably added years to my life expectancy.

OTOH the upsurge in nicotine use amongst young people feels suboptimal, and disposable vapes are a scourge.
gpjt
·4 ay önce·discuss
Looked to me like it was trying to work out whether it was edible. Sensible behaviour for an animal in a world where unfamiliar edible things appear from time to time.
gpjt
·7 ay önce·discuss
Not sure that every browser advertises English, but mine certainly does. However, as I'm in Portugal, many websites ignore what my browser says and send me to translated versions, I assume based on my IP. That causes problems because the translations are often quite bad, and they do it with redirects to PT URLs so I can't share links with people who don't speak the language.
gpjt
·7 ay önce·discuss
Awesome, thanks! I'm still doing trains on the big machines right now (hopefully will write up over xmas) but I think once I've worked out the sweet spot for memgatokens per dollar for this model, it's time to start tweaking the other controls -- LR and cosine variation of it, as you said, and also dropout, bias, weight tying, and definitely gradient clipping (which should at least get better bang for the buck from time/$ spent). I'll leave it to Google to follow up Chinchilla with a "best batch size across a thousand trained models" paper ;-)