Thank you for this due diligence, I was just reading through the technical report and couldn’t find any references to the software stack or hardware mentioning Huawei either and came back here wondering about this comment that I had read earlier.
What a find! It's on page 106 but I didn't immediately do a control-f to find it or look at the table of contents. My gosh, all the stuff I flipped through before that... some things haven't changed (e.g. Digikey and National Instruments ads).
Great. Do you have any details on how you produced this? The "reproducible code" isn't really reproducible. The "hierarchical topic model" that you mentioned - which model was used?
I hear you. Title is an exact quote from him. A couple of other accurate direct quotes from him in there. Overall a very neutral take on him.
I'd imagine it's possible bots that have taken root on this site lately - approximately the last year possibly year and a half - seemingly to suppress what could be considered, by sentiment analysis, "negative" news of anything or anyone conservative.
Or those who feel more emboldened to downvote than in previous times on this site; possibly to its growth reaching new demographics. However it feels fairly automated though given how quickly it occurs and the frequency, and how it seemed also to coincide with this last election cycle.
The guy got arrested, lost his job and had to hire a lawyer. Almost got charged with a felony assault, but the jury decided that he shouldn't be charged. Instead, later charged with a misdemeanor. I'd be super stressed about all of that, plus the trial and then having to wait 7 hours to find out I'm not guilty, but also incredibly happy after. However, all that and his name is forever associated with this incident, so despite no criminal history he might find future employment more challenging. He has probably been through quite a bit despite not being found guilty of a crime. I think that any person who is somehow inspired by doing what he did because he got away with it was always willing to do it anyway. Surely any reasonable person would realize how much he already had to go through which is enough of a deterrent. It was some degree of luck as well, even with similar or same circumstances it might up that another individual is charged and found guilty.
He's fairly lucky he doesn't have a criminal record, but it didn't come without consequences. I think the fact that the sandwich was still wrapped on the ground, hit the officer's shoulder, that the other police at the time were visibly amused during the incident, and clearly joking about it for several days after as well with the officer who had it happen to him, showed that the incident wasn't serious enough to ruin anyone's life over. A formal criminal conviction in the US would've made it hard for him to get employment for some time, if not the rest of his life.
Would really be great to move windows or tab groups between different profiles; Edge offers this. Sometimes you don't realize what profile you're in and you start doing stuff in your personal profile that doesn't fit there. With firefox right now there is no way to select and move any tabs, tab groups or windows to the profile that is best suited for those links.
There is inherent non-determinism in all machine learning models unless you explicitly configure pytorch or other frameworks to do determinism (https://docs.pytorch.org/docs/stable/notes/randomness.html). However, this is very unlikely to be done in models that are being run in production due to performance and other issues.
Hey Simon, do you have any posts diving into how one might be able to deal with evaluating LLMs or Machine Learning models in general when reproducibility is so difficult given non-determinism? Pytorch has an article on it https://docs.pytorch.org/docs/stable/notes/randomness.html but then doesn't really go into how one would then take this deterministic result, and evaluate a model that is in production (which would very likely need for performance reasons the non-determinism features enabled).
While this affects all models it seems, I think the case gets worse for in particular LLMs because I would imagine all backends, including proprietary ones, are batching users prompts. Other concurrent requests seem to change the output of your request, and then if there is even a one token change to the input or output token, especially on large inputs or outputs, the divergence can compound. Also vLLM's documentation mentions this:
https://docs.vllm.ai/en/latest/usage/faq.html
So how does one do benchmarking of AI/ML models and LLMs reliably (lets ignore arguing over the flaws of the metrics themselves, and just the fact that the output for any particular input can diverge given the above). You'd also want to redo evals as soon as any hardware or software stack changes are made to the production environment.
Seems like one needs to setup a highly deterministic backend, by forcing non-deterministic behavior in pytorch and using a backend which doesn't do batching for an initial eval that would allow for troubleshooting and non-variation in output to get a better sense of how consistent the model without the noise of batching and non-deterministic GPU calculations/kernels etc.
However then, for production, when determinism isn't guaranteed because you'd need batching and non-determism for performance, I would think that one would want to do multiple runs in various real-world situations (such as multiple users doing all sorts of different queries at the same time) and do some sort of averaging of the results. But I'm not entirely sure, because I would imagine the types of queries other users are making would then change the results fairly significantly. I'm not sure how much the batching that vLLM does would change the results of the output; but vLLM does say that batching does influence changes in the outputs.
And what paradigm is that which you decided to leave out ?