xmddmx

23 days agodiscuss

Another way (which happens to be correct) to say tenth of a second is 100 000 000 (one hundred million) nanoseconds. You were off by a factor of 1000!

xmddmx

last monthdiscuss

There's a meta-level of irony here that's important to note.

TFA is defending the use of AI, and it very clearly (to me) used AI to analyze the data and present the results.

In doing so, the author used statistics in a way they do not appear to understand, and ended up making numerous false claims (you can see the thread discussing these here https://news.ycombinator.com/item?id=48417626 )

In short, the study doesn't have sufficient statistical power, and is making "no difference" claims that aren't justified.

The meta-irony is this: the author used an LLM to interpret data in this study, and seems to have made the same category of mistake (confidently asserting falsehoods) that the study was supposed to be investigating (confidently submitting bad commits to the rsync project).

xmddmx

last monthdiscuss

You say "the underpowered study did not find evidence". Not true, it found quite a bit of evidence - many statistics were presented. There is no absence of evidence. The author wrote about the evidence, presenting P values and other statistics.

Of course the critical part is not the numbers, but what they mean.

So, what does the evidence mean?

The author interprets it to mean that there is no difference. They state this several times:

"46% EXACT PERMUTATION TEST P-VALUE (ONE-SIDED, H₁: CLAUDE MEAN > HISTORICAL)[...] What this p-value tells us is There's nothing unusual about the Claude group."

"74% ONE-SIDED P-VALUE (H₁: CLAUDE MORE LIKELY ABOVE MEDIAN) Fisher's exact test asks: if we split all releases at the historical median (0.74 sev/10c), are these Claude releases significantly buggy than previous releases (more likely to land above the median)? With a p-value of 74%, the answer is a decisive no. "

In an under-powered study, when a P value is above your alpha level cutoff (.05, .01, whatever was chosen) you can't distinguish between "no effect" and "could be an effect, but I didn't see one".

xmddmx

last monthdiscuss

Perhaps in an “everyday language” way, but not in the technical, statistical sense.

In an underpowered statistical study, a claim that two experimental conditions did not differ are not persuasive.

xmddmx

last monthdiscuss

The concept you need here is "Statistical Power".

The ELI5 version is that there are two mistakes you can make when looking at a P value:

Type I error, where your P value is falsely low. In the experiment being discussed here, it would lead one to conclude that AI code is worse. Otherwise known as a false positive.

Type II error, where your P value is falsely high, leading you to conclude that AI code is no different. Otherwise known as a false negative.

https://en.wikipedia.org/wiki/Power_(statistics)

One can calculate statistical power for a given experimental protocol.

My hunch is that if you did this, you would find this experiment is grossly under-powered.

This means you can't make the "absence of evidence" claim.

xmddmx

3 months agodiscuss

On a M4 Pro MacBook Pro with 48GB RAM I did this test:

ollama run $model "calculate fibonacci numbers in a one-line bash script" --verbose

  Model                         PromptEvalRate EvalRate
  ------------------------------------------------------
  qwen3.5:35b-a3b-q4_K_M         6.6            30.0
  qwen3.5:35b-a3b-nvfp4         13.2            66.5
  qwen3.5:35b-a3b-int4          59.4            84.4

I can't comment on the quality differences (if any) between these three.

xmddmx

4 months agodiscuss

See my other note [1] about bugs in Ollama with Qwen3.5.

I just tried this (Ollama macOS 0.17.4, qwen3.5:35b-a3b-q4_K_M) on a M4 Pro, and it did fine:

[Thought for 50.0 seconds]

1. potato 2. potato [...] 100. potato

In other words, it did great.

I think 50 seconds of thinking beforehand was perhaps excessive?

[1] https://news.ycombinator.com/item?id=47202082

xmddmx

4 months agodiscuss

See my other note about bugs in Ollama with Qwen3.5.

I just tried this (Ollama macOS 0.17.4, qwen3.5:35b-a3b-q4_K_M) on a M4 Pro, and it did fine:

[Thought for 50.0 seconds]

1. potato 2. potato [...] 100. potato

In other words, it did great.

I think 50 seconds of thinking beforehand was perhaps excessive?

xmddmx

4 months agodiscuss

Ollama users: there are notable bugs with ollama and Qwen3.5 so don't let your first impression be the last.

Theory is that some of the model parameters aren't set properly and this encourages endless looping behavior when run under ollama:

https://github.com/ollama/ollama/issues?q=is%3Aissue%20state... (a bunch of them)

xmddmx

5 months agodiscuss

> Sheldon Brown, a beloved iconoclast bicycle tech guru, died Sunday from a heart attack. He was 68 63.

Curious, what does "He was 68 63" mean. Is it a bicycle gear joke about his age at death?

xmddmx

6 months agodiscuss

On Mac Safari, holding shift and using the magic mouse to scroll up or down reverses the zoom direction.

This is both right (Shift-X is the reverse of X due to convention) But is also wrong (Shift-Scroll is the macOS gesture for scrolling on maps where Scroll alone doesn't zoom in or out).

TLDR: I really wish Apple would adopt the "scroll up to zoom in" convention used by the rest of the free world.

xmddmx

6 months agodiscuss

My memory is that it was named "Aero Glass" which heightens the irony of "Liquid Glass" sucking.

But I see many references to it being called just "Aero", but some call it "Aero Glass" [1]

Does anyone know the truth?

[1] https://www.pcmag.com/archive/rip-aero-glass-windows-8-stick...

xmddmx

6 months agodiscuss

Right, this annoyed me too - it was stated w/o attribution as if novel.

What is the name of the law when someone writes a think piece of "stuff I've learned" and fails to cite any of it to existing knowledge?

Makes me wonder if (A) they do know it's not their idea, but they are just cool with plagiarism or (B) they don't know it's not their idea.

xmddmx

6 months agodiscuss

You mean 1990. Someone graduating college in 1990 would have been about 21. That was 35 years ago, so they would be about 56 in 2025.

Math is hard.

xmddmx

6 months agodiscuss

The CFR is pretty clear, and I have experience with this (being both an IRB reviewer, faculty member, and researcher). When it says "is exempt" it means "is exempt".

Imagine otherwise: a teacher who wants change their final exam from a 50 item Scantron using A-D choices, to a 50 item Scantron using A-E choices, because they think having 5 choices per item is better than 4, would need to ask for IRB approval. That's not feasible, and is not what happens in the real world of academia.

It is true that local IRBs may try to add additional rules, but the NU policy you quote talks about "studies". Most IRBs would disagree that "professor playing around with grading procedures and policies" constitutes a "study".

It would be presumed exempted.

Are you a teacher or a student? If you are a teacher, you have wide latitude that a student researcher does not.

Also, if you are a teacher, doing "research about your teaching style", that's exempted.

By contrast, if you are a student, or a teacher "doing research" that's probably not exempt and must go through IRB.

xmddmx

6 months agodiscuss

Turns out that under the USA Code of Federal Regulations, there's a pretty big exemption to IRB for research on pedagogy:

CFR 46.104 (Exempt Research):

46.104.d.1 "Research, conducted in established or commonly accepted educational settings, that specifically involves normal educational practices that are not likely to adversely impact students' opportunity to learn required educational content or the assessment of educators who provide instruction. This includes most research on regular and special education instructional strategies, and research on the effectiveness of or the comparison among instructional techniques, curricula, or classroom management methods."

https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-...

So while this may have been a dick move by the instructors, it was probably legal.

xmddmx

7 months agodiscuss

I was impressed by the lack of dominance of Thunderbolt:

"Next I tested llama.cpp running AI models over 2.5 gigabit Ethernet versus Thunderbolt 5"

Results from that graph showed only a ~10% benefit from TB5 vs. Ethernet.

Note: The M3 studios support 10Gbps ethernet, but that wasn't tested. Instead it was tested using 2.5Gbps ethernet.

If 2.5G ethernet was only 10% slower than TB, how would 10G Ethernet have fared?

Also, TB5 has to be wired so that every CPU is connected to every other over TB, limiting you to 4 macs.

By comparison, with Ethernet, you could use a hub & spoke configuration with a Ethernet switch, theoretically letting you use more than 4 CPUs.

xmddmx

7 months agodiscuss

I really hope the user was running Time Machine - in default settings, Time Machine does hourly snapshot backups of your whole Mac. Restoring is super easy.

xmddmx

8 months agodiscuss

There is something wrong with this article, possibly just copyediting mistakes but it makes me question the whole thing.

For example, check out this mess:

> “Unfortunately, there is one significant issue with the aforementioned data: schooling. Seeing as the majority of work to date includes only aggregate data, it is impossible to account. The first concerns small N: seeing as most publish studies only include a handful of TRA data, there is a lot of room for error and over.

Unfortunately, there is a largely unaccounted for confound in this aggregate data which may make generalized analysis questionable: schooling.”

xmddmx

8 months agodiscuss

Why not do an empirical A/B test: Set up two honeypots (or perhaps 2000 for statistical significance). A gets zero updates, B gets all updates immediately. See which ones get pwned faster.

Submissions

What's the deal with distributed SYN DOS attacks

comments