n=56 doesn't give you much information regarding the margin of error, unless you practice Tarot Reader's cat science. The standard deviation of outcomes and the difference between both outcomes matter just as much.
If I flip a coin 56 times and it always falls on head, I can be pretty much certain that it's not a fair coin. I wouldn't need to flip it 1000 times. We are all someone else's "know-nothing hack"...
Can we know for sure that the price drop is accompanied by a change in the model such as quantization?
On twitter, some people say that some models perform better at night when there is a less demand which allows them to serve a non-quantized model.
Since the models are only available through API and there is no test to check which version of the model is served, it's hard to know what we're buying...
If I flip a coin 56 times and it always falls on head, I can be pretty much certain that it's not a fair coin. I wouldn't need to flip it 1000 times. We are all someone else's "know-nothing hack"...