HackerTrans
トップ新着トレンドコメント過去質問紹介求人

gpjt

1,787 カルマ登録 18 年前
https://www.gilesthomas.com/

投稿

Building intuition about LLM parameter counts

gilesthomas.com
2 ポイント·投稿者 gpjt·15 時間前·0 コメント

トレーニング ボックスのポピー、パート 1: 始まり

gilesthomas.com
3 ポイント·投稿者 gpjt·一昨日·0 コメント

From bigrams to GPT-2, one component at a time (in Jax)

gilesthomas.com
1 ポイント·投稿者 gpjt·3 日前·0 コメント

Building a Jax training loop for an LLM training run

gilesthomas.com
2 ポイント·投稿者 gpjt·11 日前·0 コメント

Thoughts on Role Confusion

gilesthomas.com
3 ポイント·投稿者 gpjt·17 日前·0 コメント

Flax debugging: making a hash of things

gilesthomas.com
2 ポイント·投稿者 gpjt·24 日前·0 コメント

10Gb/s Ethernet: switching to a Broadcom SFP+ module

gilesthomas.com
195 ポイント·投稿者 gpjt·25 日前·170 コメント

Jax: Commitment Issues

gilesthomas.com
4 ポイント·投稿者 gpjt·26 日前·0 コメント

Jax Back Ends and Devices

gilesthomas.com
2 ポイント·投稿者 gpjt·先月·0 コメント

Using Safetensors with Flax

gilesthomas.com
2 ポイント·投稿者 gpjt·先月·0 コメント

First Looking into Jax

gilesthomas.com
3 ポイント·投稿者 gpjt·先月·0 コメント

10Gb/s Ethernet: using mini-heatsinks with a 10GBASE-T SFP+ module

gilesthomas.com
3 ポイント·投稿者 gpjt·2 か月前·0 コメント

10Gb/s Ethernet: what I did to get it working in my home

gilesthomas.com
232 ポイント·投稿者 gpjt·2 か月前·177 コメント

10Gb Ethernet: what I had to (re)learn

gilesthomas.com
1 ポイント·投稿者 gpjt·2 か月前·1 コメント

LLM from scratch, part 33 – what I learned from the appendices

gilesthomas.com
5 ポイント·投稿者 gpjt·3 か月前·0 コメント

LLM from scratch (32l) – Interventions: updated instruction fine-tuning results

gilesthomas.com
1 ポイント·投稿者 gpjt·3 か月前·0 コメント

How an LLM becomes more coherent as we train it

gilesthomas.com
3 ポイント·投稿者 gpjt·3 か月前·0 コメント

LLM from scratch, part 32k – Interventions: gradient accumulation

gilesthomas.com
2 ポイント·投稿者 gpjt·3 か月前·0 コメント

Provision: LLM-powered server setup from Markdown

provision.sh
2 ポイント·投稿者 gpjt·3 か月前·0 コメント

LLM from scratch, part 32j – trying to train a better model in the cloud

gilesthomas.com
2 ポイント·投稿者 gpjt·3 か月前·0 コメント

コメント

gpjt
·18 日前·議論
In the EU, at least one of the problems is regulatory/tax for digital services. They need to charge you the VAT rate for the country where you are based. For that, they need two pieces of evidence about your location. There are various things they can use for that -- telephone number, IP address, card billing address, and so on. If they can collect two that indicate the same country, they're safe -- but if all of them point in different directions then they could get in trouble during a tax audit.

Of course, for larger transactions you'd expect that a human in the loop could work with you to get the right info so that they would be covered. But I guess for Microsoft, their definition of "larger" might be more than a few grand...
gpjt
·24 日前·議論
OP here -- yes indeed. I ever do a new series of posts on upgrading my network to anything faster, the first one will probably be titled something like "25Gb/s Ethernet: how much it costs to rip out your CAT-6A and replace it with fibre".
gpjt
·24 日前·議論
OP here -- yes, 100%! Within my study it's DACs all the way. I only use 10GBASE-T where it's the only option: through the wall cabling, and from the connector on my ISP's crappy router to the mini-PC that acts as the real router.
gpjt
·24 日前·議論
Thanks for making the comment! It saved me having to bolt a USB fan over the old, misbehaving SFP+ module or something similarly silly...
gpjt
·28 日前·議論
Doesn't say anything about citizenship though. There are plenty of US residents who are not citizens. And a lot of people abroad appear to use US billing address credit cards -- in my last company we had hundreds of people with the same US billing address who appeared to be managing Africa-focused businesses and used IPs that matched that.
gpjt
·先月·議論
Excellent points. It's also worth noting that many people don't wind up working in the field they studied at university. CS grads have probably been the exception in recent years, because the industry has been booming, but the two most successful entrepreneurs I know studied philosophy and art history. A friend who is very senior in recruitment studied economics.
gpjt
·先月·議論
That is an excellent point so long as you don't take it too far! Lisp/Haskell/Erlang yes. INTERCAL/Brainfuck less so...
gpjt
·先月·議論
Hmmm. "Comforting haze" seems dubious. My grandmother had Alzheimer's and at least from the outside it seemed like a bad LSD trip that never ended. She didn't understand what was happening and was scared.

Sample size of one and anecdotal of course, but...
gpjt
·先月·議論
Love the verbification!
gpjt
·2 か月前·議論
I was thinking the same. It's a simple idea, heavily over-explained. The code is similar, massively overengineered for such a simple test.
gpjt
·2 か月前·議論
I guess if it's on the inside of a water-cooling loop, you should be OK if the water is pure enough. I don't know how hard that "enough" would be, though.
gpjt
·3 か月前·議論
Yup, agreed -- it's amazing how close they are getting! I was just wondering if there was some true frontier non-US model that I'd missed.
gpjt
·3 か月前·議論
Thanks for the link -- I read that when it was published, then the other day I wanted to send it to someone but I'd forgotten where it was.
gpjt
·3 か月前·議論
Which are not? There are Chinese models that are only months behind, which is impressive -- but they are still behind.
gpjt
·3 か月前·議論
I have it running on a Proxmox VM. It basically just sends me summaries -- it reads a bunch of RSS feeds I pointed it to (news, tech, etc) and gives me a daily summary, along with an image of the day based on that. It also sends me recommendations for times to go for a run based on the weather and my Strava activity, daily recommendations for stargazing (what's visible and when, weather, etc), and a couple of daily reminders for things I tend to forget.

I'm using Claude as the model, though, so it's smart but pricey. Should configure it to use different models for different things, but it's trickier than I would have expected to do that.
gpjt
·3 か月前·議論
Wait, that's it? Seven paragraphs, all short? Two quotes, one from some anonymous MS exec? Is the site sending some minimal version of the article to me because I'm using Brave, or is this the lowest-content article I've seen in weeks (and I'm on Twitter)?
gpjt
·4 か月前·議論
It's a bit of a double edged sword. As someone who smoked and found it impossible to quit for decades, I'm very happy to have been able to switch to (reuasable) vaping. It's probably added years to my life expectancy.

OTOH the upsurge in nicotine use amongst young people feels suboptimal, and disposable vapes are a scourge.
gpjt
·4 か月前·議論
Looked to me like it was trying to work out whether it was edible. Sensible behaviour for an animal in a world where unfamiliar edible things appear from time to time.
gpjt
·7 か月前·議論
Not sure that every browser advertises English, but mine certainly does. However, as I'm in Portugal, many websites ignore what my browser says and send me to translated versions, I assume based on my IP. That causes problems because the translations are often quite bad, and they do it with redirects to PT URLs so I can't share links with people who don't speak the language.
gpjt
·7 か月前·議論
Awesome, thanks! I'm still doing trains on the big machines right now (hopefully will write up over xmas) but I think once I've worked out the sweet spot for memgatokens per dollar for this model, it's time to start tweaking the other controls -- LR and cosine variation of it, as you said, and also dropout, bias, weight tying, and definitely gradient clipping (which should at least get better bang for the buck from time/$ spent). I'll leave it to Google to follow up Chinchilla with a "best batch size across a thousand trained models" paper ;-)