HackerTrans
TopNewTrendsCommentsPastAskShowJobs

gpjt

1,787 karmajoined 18 tahun yang lalu
https://www.gilesthomas.com/

Submissions

Building intuition about LLM parameter counts

gilesthomas.com
2 points·by gpjt·18 jam yang lalu·0 comments

Poppy the training box, part 1: the beginnings

gilesthomas.com
3 points·by gpjt·3 hari yang lalu·0 comments

From bigrams to GPT-2, one component at a time (in Jax)

gilesthomas.com
1 points·by gpjt·3 hari yang lalu·0 comments

Building a Jax training loop for an LLM training run

gilesthomas.com
2 points·by gpjt·11 hari yang lalu·0 comments

Thoughts on Role Confusion

gilesthomas.com
3 points·by gpjt·17 hari yang lalu·0 comments

Flax debugging: making a hash of things

gilesthomas.com
2 points·by gpjt·25 hari yang lalu·0 comments

10Gb/s Ethernet: switching to a Broadcom SFP+ module

gilesthomas.com
195 points·by gpjt·25 hari yang lalu·170 comments

Jax: Commitment Issues

gilesthomas.com
4 points·by gpjt·26 hari yang lalu·0 comments

Jax Back Ends and Devices

gilesthomas.com
2 points·by gpjt·bulan lalu·0 comments

Using Safetensors with Flax

gilesthomas.com
2 points·by gpjt·bulan lalu·0 comments

First Looking into Jax

gilesthomas.com
3 points·by gpjt·bulan lalu·0 comments

10Gb/s Ethernet: using mini-heatsinks with a 10GBASE-T SFP+ module

gilesthomas.com
3 points·by gpjt·2 bulan yang lalu·0 comments

10Gb/s Ethernet: what I did to get it working in my home

gilesthomas.com
232 points·by gpjt·2 bulan yang lalu·177 comments

10Gb Ethernet: what I had to (re)learn

gilesthomas.com
1 points·by gpjt·2 bulan yang lalu·1 comments

LLM from scratch, part 33 – what I learned from the appendices

gilesthomas.com
5 points·by gpjt·3 bulan yang lalu·0 comments

LLM from scratch (32l) – Interventions: updated instruction fine-tuning results

gilesthomas.com
1 points·by gpjt·3 bulan yang lalu·0 comments

How an LLM becomes more coherent as we train it

gilesthomas.com
3 points·by gpjt·3 bulan yang lalu·0 comments

LLM from scratch, part 32k – Interventions: gradient accumulation

gilesthomas.com
2 points·by gpjt·3 bulan yang lalu·0 comments

Provision: LLM-powered server setup from Markdown

provision.sh
2 points·by gpjt·3 bulan yang lalu·0 comments

LLM from scratch, part 32j – trying to train a better model in the cloud

gilesthomas.com
2 points·by gpjt·3 bulan yang lalu·0 comments

comments

gpjt
·18 hari yang lalu·discuss
In the EU, at least one of the problems is regulatory/tax for digital services. They need to charge you the VAT rate for the country where you are based. For that, they need two pieces of evidence about your location. There are various things they can use for that -- telephone number, IP address, card billing address, and so on. If they can collect two that indicate the same country, they're safe -- but if all of them point in different directions then they could get in trouble during a tax audit.

Of course, for larger transactions you'd expect that a human in the loop could work with you to get the right info so that they would be covered. But I guess for Microsoft, their definition of "larger" might be more than a few grand...
gpjt
·24 hari yang lalu·discuss
OP here -- yes indeed. I ever do a new series of posts on upgrading my network to anything faster, the first one will probably be titled something like "25Gb/s Ethernet: how much it costs to rip out your CAT-6A and replace it with fibre".
gpjt
·25 hari yang lalu·discuss
OP here -- yes, 100%! Within my study it's DACs all the way. I only use 10GBASE-T where it's the only option: through the wall cabling, and from the connector on my ISP's crappy router to the mini-PC that acts as the real router.
gpjt
·25 hari yang lalu·discuss
Thanks for making the comment! It saved me having to bolt a USB fan over the old, misbehaving SFP+ module or something similarly silly...
gpjt
·28 hari yang lalu·discuss
Doesn't say anything about citizenship though. There are plenty of US residents who are not citizens. And a lot of people abroad appear to use US billing address credit cards -- in my last company we had hundreds of people with the same US billing address who appeared to be managing Africa-focused businesses and used IPs that matched that.
gpjt
·bulan lalu·discuss
Excellent points. It's also worth noting that many people don't wind up working in the field they studied at university. CS grads have probably been the exception in recent years, because the industry has been booming, but the two most successful entrepreneurs I know studied philosophy and art history. A friend who is very senior in recruitment studied economics.
gpjt
·bulan lalu·discuss
That is an excellent point so long as you don't take it too far! Lisp/Haskell/Erlang yes. INTERCAL/Brainfuck less so...
gpjt
·bulan lalu·discuss
Hmmm. "Comforting haze" seems dubious. My grandmother had Alzheimer's and at least from the outside it seemed like a bad LSD trip that never ended. She didn't understand what was happening and was scared.

Sample size of one and anecdotal of course, but...
gpjt
·bulan lalu·discuss
Love the verbification!
gpjt
·2 bulan yang lalu·discuss
I was thinking the same. It's a simple idea, heavily over-explained. The code is similar, massively overengineered for such a simple test.
gpjt
·2 bulan yang lalu·discuss
I guess if it's on the inside of a water-cooling loop, you should be OK if the water is pure enough. I don't know how hard that "enough" would be, though.
gpjt
·3 bulan yang lalu·discuss
Yup, agreed -- it's amazing how close they are getting! I was just wondering if there was some true frontier non-US model that I'd missed.
gpjt
·3 bulan yang lalu·discuss
Thanks for the link -- I read that when it was published, then the other day I wanted to send it to someone but I'd forgotten where it was.
gpjt
·3 bulan yang lalu·discuss
Which are not? There are Chinese models that are only months behind, which is impressive -- but they are still behind.
gpjt
·3 bulan yang lalu·discuss
I have it running on a Proxmox VM. It basically just sends me summaries -- it reads a bunch of RSS feeds I pointed it to (news, tech, etc) and gives me a daily summary, along with an image of the day based on that. It also sends me recommendations for times to go for a run based on the weather and my Strava activity, daily recommendations for stargazing (what's visible and when, weather, etc), and a couple of daily reminders for things I tend to forget.

I'm using Claude as the model, though, so it's smart but pricey. Should configure it to use different models for different things, but it's trickier than I would have expected to do that.
gpjt
·3 bulan yang lalu·discuss
Wait, that's it? Seven paragraphs, all short? Two quotes, one from some anonymous MS exec? Is the site sending some minimal version of the article to me because I'm using Brave, or is this the lowest-content article I've seen in weeks (and I'm on Twitter)?
gpjt
·4 bulan yang lalu·discuss
It's a bit of a double edged sword. As someone who smoked and found it impossible to quit for decades, I'm very happy to have been able to switch to (reuasable) vaping. It's probably added years to my life expectancy.

OTOH the upsurge in nicotine use amongst young people feels suboptimal, and disposable vapes are a scourge.
gpjt
·4 bulan yang lalu·discuss
Looked to me like it was trying to work out whether it was edible. Sensible behaviour for an animal in a world where unfamiliar edible things appear from time to time.
gpjt
·7 bulan yang lalu·discuss
Not sure that every browser advertises English, but mine certainly does. However, as I'm in Portugal, many websites ignore what my browser says and send me to translated versions, I assume based on my IP. That causes problems because the translations are often quite bad, and they do it with redirects to PT URLs so I can't share links with people who don't speak the language.
gpjt
·7 bulan yang lalu·discuss
Awesome, thanks! I'm still doing trains on the big machines right now (hopefully will write up over xmas) but I think once I've worked out the sweet spot for memgatokens per dollar for this model, it's time to start tweaking the other controls -- LR and cosine variation of it, as you said, and also dropout, bias, weight tying, and definitely gradient clipping (which should at least get better bang for the buck from time/$ spent). I'll leave it to Google to follow up Chinchilla with a "best batch size across a thousand trained models" paper ;-)