Possibly overly cynical, but I think a lot of people who are getting forced to shoehorn LLMs into their applications don’t have a choice about NOT using an LLM for the required use case.
Testing and validation kind of imply that it won’t ship if it isn’t fit for purpose, which isn’t an option and most devs will know it’s kind of junk already.
That’s not to say there aren’t good use cases, but when forced to add an LLM somewhere it doesn’t work, and you have no examples of “correct” output anyway, validation is usually an afterthought.
Ugh I am dealing with an amazingly productive platform team who churn out so much stuff that the product teams just can’t keep on top of all the changes to tools and tech.
That one team is super productive at the cost of everyone else grinding to a halt.
I have to work with a DBA who has decided that nothing new gets developed using Postgres and is confident that our use cases are best served with a document db… all without knowing any of our use cases, requirements or constraints.
Now I just don’t involve him in anything unless forced.
Honestly this is a concern for me in a non boogeyman way. I joined a company to work on their edtech product for kids and got assigned to work on their AI product. I have no idea how to be confident and prove that the gen ai won’t tell the kids harmful things. We can try all kinds of things but I don’t know how to PROVE it won’t.
The company I work at is using step functions heavily and I hate it. Instead of an if statement they make it a new step with the conditional in json, so following the code required you to jump around between files.
It has also been built by contractors who have no incentives to make it run locally or be easy to manage in production, but that isn't specific to step functions, and more due to poor leadership.
To interpret your example in another way, a page working in IE is doing it right. So first you do it, and structure it the way you think it should be done with "correct" markup. Once you have that, you can then do it right and get it working properly in IE. After that, doing it better would be restructuring things so maybe you dont need as many hacks.
Testing and validation kind of imply that it won’t ship if it isn’t fit for purpose, which isn’t an option and most devs will know it’s kind of junk already.
That’s not to say there aren’t good use cases, but when forced to add an LLM somewhere it doesn’t work, and you have no examples of “correct” output anyway, validation is usually an afterthought.