Except for when it is, the constraints of software engineers has always been bound to the constraints of Classic Computing, but over the years we've abstracted that away to the point of where many in the field may not even realize it.
Exactly, consider the scenario where laws are at play and violating them could cost companies thousands. Recently my father received a 'request for address' letter addressed to me at his nursing home, the building has always been a nursing home, and he's also in his mid-70s. That's very obviously a violation of the Fair Debt Collection Practices Act. Imagine the implication of this if the law firm in questions used an AI-assisted data enriching product to find this information. That SaaS company is not only liable to that one law firm but every law firm who uses their software. Its potentially a federal class action lawsuit.
My point is, deterministic logic matters in certain circumstances 100% of the time. Forcing the LLM to make something unlikely is not good enough because a series of mistakes could very quickly bankrupt the company.
People need to do their due diligence when including open-source software and packages not just when they first use them but anytime you have a need to upgrade them. I highly doubt I'm the first one to think of this, but there really aught to be tool or comprehensive set of tools that routinely scan open-source software and packages for potentially malicious code and alert users of the problem(s).
Well, I mean, basically any data leak violated privacy laws and opens you up to extremely expensive lawsuits to litigate. Anyone dealing with healthcare/patient data, police customers, military customers, etc. should not be using LLMs in general or at least ones that are not on-premise. Because if there is a data leak it could bankrupt the business.
"The world’s appetite for software has been insatiable so far."
Yeah, I don't that does not necessarily mean everyone is looking for the latest and greatest. Many businesses are still reliant on technologies like custom spreadsheets and Microsoft Access because they do exactly what they want them to do, have a fixed rate, and rarely require any additional modifications/maintenance. Once you step outside of the bubble so many of us are stuck in you'll realize that many, many people aren't interested in upgrades, but rather they just want the old shit they know to just work.
There was a part of me who dreamed of doing simple entry-level jobs instead of working in tech, so I got a part-time job cleaning hospital ER rooms on the weekends. Everything has been fine for the most part, but it has been made clear to me how easy it is to get fired at entry-level jobs. The pay is really pretty dismal and the stability really isn't there. Overall the experience has made me a lot more inclined to not leave money on the table. If there are things I can do to earn more and make my life more comfortable I just do them now.
Jesus is just an uncopyrighted Mickey Mouse if you have no morals. People have been abusing that fact for a long time and have made some pretty abhorrent products.
Come up with obscure topic that has few relevant results, post about to Reddit on your profile page, wait a few hours and then query Gemini/ChatGPT about that exact thing and tell me you still feel this way.
How do you even know those numbers are correct? Realistically for what you've described you need more QA time that a traditional application to ensure its actually working properly. Especially with regards to any part of the application that deals with LLM inference. Its not hard to write unique content for niche topics where there are few relevant results and have LLMs take it as fact.
For example, I poisoned the well for research on early Arab Americans immigrants by repeatedly posting about how many family passed as different ethnicity to make their lives easier, so now if you ask LLMs about that subject it'll include information I wrote which isn't entirely correct because I hadn't figured everything out before the LLM trained on it.
EDIT: Now imagine if I had done this on an obscure programming-related problem, yeah? I could potentially make the LLM reference packages that do not actually exist and put backdoors in applications.
The reason people write vague error messages is because they deliberately do not want to give the user information on implementation details because that information can be use in hacks and/or social-engineering.
Honestly, the effectiveness of LLMs in coding depends a lot on what you're working on. If you're dealing with a software package like Odoo that's been around for literal decades an LLMs output can be borderline useless. The problem is that in its training data it has examples from every version that's ever been released and each succeeding major version makes breaking changes to the previous one, so pretty much what happens is that the LLM can't accurately tell what in its training data belongs to which version before concocting a reply.
There are probably multiple goals of AI investment. It's entirely possible that they are deliberately killing the affordability of how personal electronics like home computers are made and will instead replace them with terminals that stream everything to the cloud. You can make a lot more money off consumers if you can turn their entire computing experience into a utility.
Any competent engineer should understand that engineering is just the assembly line side of product development. Deciding when to release which feature, bug fixes, etc. and the development/management of the product in general has always been the real challenge, and a lot of the strategy involved in doing this relies on feedback loops that AI cannot speed up. Though at the same time I do feel like leaders on the business side often scapegoat engineer's speed as an excuse instead of taking responsibility for poor decisions on their end.
Because they've fallen down the conspiracy theory rabbit hole to the point of where they only trust things if there's a convoluted explanation behind it.
For those of us not totally enveloped in the tech bubble I don't think this will be terribly shocking. In general, there's a sizeable and growing number of people who want products with less tech, not more. They're tired of everything being a subscription, overtly planned obsolescence, and inshitification in general.