I've found "T/F" vals are not always portable between languages ("True" in Python vs "true" in JS) whereas integer comparisons are always interpreted correctly.
LLM hype is everywhere these days, but actual usage data is pretty hard to find. I see tons of companies pushing AI tools at me but I have almost no idea what user behaviour/preferences are.
I built a TaxGPT.ca, an AI chatbot for answering people's (Canadian) tax questions and ran a bunch of numbers over the questions submitted between March 4 to April 31, 2024.
Highlights include:
- Over 8,500 questions answered
- Users frequently asked about basic tax info and business expenses
- Feedback is not very common but surprisingly positive
- Experiments in tagging questions in relevant and for 'complexity'
My verdict here is that these bots have potential but you have to have really quick feedback loops and pretty good product chops — you can't just throw AI at some problem and assume users will find their own ways to make it useful.
> How often do they correctly predict the weather?
I mentioned this in another comment, but accuracy data is basically impossible to get. You have to get weather data year by year for different regions of the country and then compare them to historical ranges. Means a pretty huge amount of trawling through data, not a lot of payoff.
> Also how often did their right or wrong predictions contradict the majority?
That's a huge insight to take away. Nevertheless, we still have climate-denier groundhogs consistently predicting winter and being wrong about it (cough Phil cough).
Hopefully, this site helps shine a light on these unscientific groundhogs.
Accuracy data is basically impossible to get. You have to get weather data year by year for different regions of the country and then compare them to historical ranges. It will basically mean 10x the work I've done already. It's open-source though, so it could be that someone wants to collaborate with me in future :D
But you are right, SOC2 is the next step for sure.