Do you know which method is this? Euler's by any chance? And do you have an idea how one would prove that it creates a magic square? It's actually one of my inspirations for writing this, the relationship between the code that does something and the proof that the code actually does what it claims. I'd argue an LLM would find the proof helpful if it were asked to generalize an existing function in some way
It did go back on itself 3 times, no? "Actually, let’s trace for x=3:" (it had just computed for x=3 the first time); then "Better to check actual q output:" -- did it actually run it in a q session, or just pretended? And another one "That doesn’t seem to align. Let’s do it step by step:"
Author here. Majromax challenged me to test `i = 1 + i`, which broke my
theoretical framework. While setting up that experiment, I realized I hadn't
used chat templates in my original measurements (rookie mistake with an
Instruct model!).
Re-running with proper methodology completely flips the results - the terse
version actually wins. I'll add a correction note to the article once AWS/Medium
comes back online and will write a follow-up with the corrected experiments.
This is open science working as intended - community scrutiny improves the work.
Thank you all for the engagement, and especially to Majromax for the challenge
that led to discovering this!
Thanks so much for this challenge! I just ran the experiment with i = 1 + i
and you're absolutely right - it breaks my theoretical framework (same semantic
information, but much higher perplexity).
While setting this up, I realized I hadn't used chat templates in my original
measurements (rookie mistake with an Instruct model!). Re-running with proper
methodology completely flips the results - the terse version actually wins.
I'll add a correction note to the article once AWS/Medium comes back online,
and will write a proper follow-up with all the corrected experiments. Your
comment literally made the research better - thank you!
I agree with you, though in the q world people tend to take it to the extreme, like packing a whole function into a single line rather than a single screen. Here's a ticker plant standard script from KX themselves; I personally find this density makes it harder to read, and when reading it I put it into my text editor and split semicolon-separated statements onto different lines: https://github.com/KxSystems/kdb-tick/blob/master/tick.q
E.g. one challenge I've had was generating a magic square on a single line; for odd-size only, I wrote: ms:{{[(m;r;c);i]((.[m;(r;c);:;i],:),$[m[s:(r-1)mod n;d:(c+1) mod n:#:[m]];((r+1)mod n;c);(s;d)])}/[((x;x)#0;0;x div 2);1+!:[x*x]]0}; / but I don't think that's helping anyone