Programming AIs worry me(buttondown.email)
buttondown.email
Programming AIs worry me
https://buttondown.email/hillelwayne/archive/programming-ais-worry-me/
191 comments
> See you in a few years when we can’t even keep up with the deluge of error prone programs.
I think the worst case of this situation will pop up before that; not writing code at all but just hook the AI up, ‘prompt engineered’ directly like so;
Why have the AI make code we can read? Just let it do the entire thing /s
It’ll happen though and it will be used seriously by companies. The quality of the AI, legislation and public/consumer complaint fallout will determine if we get back to code that humans check. Being sceptical about humans wanting quality over paying as little as possible, I am going to say no.
I think the worst case of this situation will pop up before that; not writing code at all but just hook the AI up, ‘prompt engineered’ directly like so;
<body>
<script>doInsuranceForm();</script>
</body>
Why have the AI make code we can read? Just let it do the entire thing /s
It’ll happen though and it will be used seriously by companies. The quality of the AI, legislation and public/consumer complaint fallout will determine if we get back to code that humans check. Being sceptical about humans wanting quality over paying as little as possible, I am going to say no.
> A former colleague of mine challenged his grad students to implement an error-free version of binary search. He gave them the specification and verified each one: the majority of the submissions had errors.
I was in a class on Isabelle where we had to write and prove correct a binary search function. As I recall, everyone admitted that their first function didn’t work.
I was in a class on Isabelle where we had to write and prove correct a binary search function. As I recall, everyone admitted that their first function didn’t work.
Java's was broken till '06
https://en.wikipedia.org/wiki/Binary_search_algorithm#Implem...
People often do (L+R)/2 but that can overflow. Using L+(R-L)/2 avoids the overflow problem.
People often do (L+R)/2 but that can overflow. Using L+(R-L)/2 avoids the overflow problem.
As someone who has a binary search in their interview question, I will claim that far more people run into the problem of trying to do midpoint/2 and midpoint*2, long before overflows.
AI will be a major productivity enhancer. The things that make programming interesting will be what programmers focus on in the future. There's so much potential software that could be written that I don't think that the number of programmers will decrease at all.
> AI will be a major productivity enhancer.
This worries me. The problem of programming isn't that we're too slow at writing code. It's that we're not thinking critically enough about the problems we're solving. And the way we approach programming is to not fully understand what we're doing and derive joy from fixing the result of our neglect later on.
TFA touches on this a little but doesn't dive in deep: how do you know that your software does what you think it does? In such a small example, a world renown researcher missed an error! Several people did. In their current state, LLM's will gleefully give you wrong answers as if they're 100% correct and it is now your job to verify they are correct. Most programmers do not know how to write a proof or other sort of formally verifiable specification. How will they know that the code they are about to commit is correct?
For now, LLM's aren't capable of mathematical reasoning. You can't prompt it for a piece of code and then ask it to demonstrate it's reasoning or prove to you why the change is correct with regards to your specifications.
And to make matters worse, humans are terrible at finding errors in code. Empirical studies on large scale informal code review suggest that humans can only reduce error rates when reviewing somewhere in the realm of 200LOC every couple of hours. We're not effective at finding errors to begin with and our abilities drop precipitously if we read too much code. Now we're going to have an assistant that generates possibly correct code and we're going to have to proof-read hundreds of lines?
This does not seem like a good use of time or enhancing my productivity.
I think we need to wait until we can get ML models that understand proof tactics and kernels to help automate proofs. I would be much more willing to accept code from a bot that can demonstrate its reasoning and isn't simply generating code based off of statistical models. Computers are nefariously discrete machines and "almost right" isn't good enough in many, many cases.
What will enhance productivity, in my humble opinion, are better languages that let us write more precise specifications and can derive the code for us.
This worries me. The problem of programming isn't that we're too slow at writing code. It's that we're not thinking critically enough about the problems we're solving. And the way we approach programming is to not fully understand what we're doing and derive joy from fixing the result of our neglect later on.
TFA touches on this a little but doesn't dive in deep: how do you know that your software does what you think it does? In such a small example, a world renown researcher missed an error! Several people did. In their current state, LLM's will gleefully give you wrong answers as if they're 100% correct and it is now your job to verify they are correct. Most programmers do not know how to write a proof or other sort of formally verifiable specification. How will they know that the code they are about to commit is correct?
For now, LLM's aren't capable of mathematical reasoning. You can't prompt it for a piece of code and then ask it to demonstrate it's reasoning or prove to you why the change is correct with regards to your specifications.
And to make matters worse, humans are terrible at finding errors in code. Empirical studies on large scale informal code review suggest that humans can only reduce error rates when reviewing somewhere in the realm of 200LOC every couple of hours. We're not effective at finding errors to begin with and our abilities drop precipitously if we read too much code. Now we're going to have an assistant that generates possibly correct code and we're going to have to proof-read hundreds of lines?
This does not seem like a good use of time or enhancing my productivity.
I think we need to wait until we can get ML models that understand proof tactics and kernels to help automate proofs. I would be much more willing to accept code from a bot that can demonstrate its reasoning and isn't simply generating code based off of statistical models. Computers are nefariously discrete machines and "almost right" isn't good enough in many, many cases.
What will enhance productivity, in my humble opinion, are better languages that let us write more precise specifications and can derive the code for us.
> I think we need to wait until
I think we should take advantage of any improvements we can get, that result in a net gain, for the individual. Programming is a practical endeavor. There’s no need to wait.
I think we should take advantage of any improvements we can get, that result in a net gain, for the individual. Programming is a practical endeavor. There’s no need to wait.
You’re absolutely right! Don’t wait! Learn how to write proofs and use model checkers today!
But I think most people will prefer the easy way out.
But I think most people will prefer the easy way out.
We often have to spend so a lot of time and energy on boring but critical tasks like writing repetitive CRUD code and tests. LLMs can free up that time and energy for critical thinking.
We will have to review code LLMs write, I don't doubt that, but the overall process should be faster. Additionally, they can review the code we write as well.
LLMs can find bugs in code, not all for sure, but they have the broad knowledge to be able to at least suggest lines of code that aren't working and why their reasoning.
We will have to review code LLMs write, I don't doubt that, but the overall process should be faster. Additionally, they can review the code we write as well.
LLMs can find bugs in code, not all for sure, but they have the broad knowledge to be able to at least suggest lines of code that aren't working and why their reasoning.
This may be true, and it worries me. I have a (completely unproven) conjecture that things that make programming boring (like writing boilerplate code) are more or less things that let us avoid burnout.
How many hours per day can you stay focused on difficult (or at least non-boring) problems? And for how many years?
Whenever someone says "AI = productivity enhancer", I hear "AI = burnout accelerator".
How many hours per day can you stay focused on difficult (or at least non-boring) problems? And for how many years?
Whenever someone says "AI = productivity enhancer", I hear "AI = burnout accelerator".
IME burnout is neither about boilerplate or traditional "interesting" problems, it's about "somebody else didn't do their job right" problems like the infamous "why can't I just serve 5 terabytes?" story or toxic work cultures.
https://rachelbythebay.com/w/2021/10/30/5tb/
If it takes away the boilerplate and the interesting problems leaving us with just with 5 TB problems then yeah, burnout city.
https://rachelbythebay.com/w/2021/10/30/5tb/
If it takes away the boilerplate and the interesting problems leaving us with just with 5 TB problems then yeah, burnout city.
Burnout has different causes.
For burnout because of intense work, it could be that the movement to reduce the number of hours we work will come at the right time. 4 day work weeks at 100%, for example.
Lots of people experience burnout because writing boring code can be mind-numbing. Then there are people factors, such as a bad manager. Perhaps AI can give us more understanding management?
For burnout because of intense work, it could be that the movement to reduce the number of hours we work will come at the right time. 4 day work weeks at 100%, for example.
Lots of people experience burnout because writing boring code can be mind-numbing. Then there are people factors, such as a bad manager. Perhaps AI can give us more understanding management?
If programmer numbers decrease it won't be as a result of running out of things to code or AI, it'll be as a result of market consolidation within a few, extremely powerful firms who can squash most competitors like bugs.
Like the US auto market - a once thriving ecosystem of fiercely competing innovative small businesses consolidated into 3 vertically integrated firms who now think innovation is building a slightly bigger truck.
Of course, they'll probably blame it on AI coz a story about the inevitable march of technological progress invites less government oversight of the abuse of market power, and we will no doubt be plied with plausible sounding stories to that effect.
Like the US auto market - a once thriving ecosystem of fiercely competing innovative small businesses consolidated into 3 vertically integrated firms who now think innovation is building a slightly bigger truck.
Of course, they'll probably blame it on AI coz a story about the inevitable march of technological progress invites less government oversight of the abuse of market power, and we will no doubt be plied with plausible sounding stories to that effect.
I think this idea of having formal methods integrated with AI code gen will be really powerful. If we can write verifiable specs, or even a way to automate approximate verification, then it can be used a s a scoring function to train an machine learning algorithm that performs codegen.
Just like a human, don't use these new AI tools to read your mind.
I've found ChatGPT to be impressively helpful at knowing where to continue to look for something. I can write a sentence and explain what I'm looking for, and maybe even paste in a code snippet, and 10 seconds later I get enough information to go back to Google and continue digging deeper.
For me it's become a bridge tool. Instead of spending an hour trying to get Google to understand my confusion and endless ads thrown my way, I get a very concise answer with full definitions in case I needed to know. It understands me, which is the most important part.
The code examples it spits out are meh, and likely always going to be error prone, but I cannot imagine it won't improve over time.
I've found ChatGPT to be impressively helpful at knowing where to continue to look for something. I can write a sentence and explain what I'm looking for, and maybe even paste in a code snippet, and 10 seconds later I get enough information to go back to Google and continue digging deeper.
For me it's become a bridge tool. Instead of spending an hour trying to get Google to understand my confusion and endless ads thrown my way, I get a very concise answer with full definitions in case I needed to know. It understands me, which is the most important part.
The code examples it spits out are meh, and likely always going to be error prone, but I cannot imagine it won't improve over time.
Yeah it's great as an assistant to a critical thinking human, people who are criticizing it are either doing it for clickbait aka money or expected it go think for them.
Like it showed me how to split an audio file into parts with ffmpeg, I'm sure I knew it at some point but it took me less time than reading the -h or looking for an answer on stackoverflow, especially with google's greed that it stopped showing stackoverflow first and sometimes leaves it to the 2nd page and fills the first with garbage, started using yandex as an alternative to google a year ago, now using it by default and google from time to time, it's sad that the best product is behind us.
Like it showed me how to split an audio file into parts with ffmpeg, I'm sure I knew it at some point but it took me less time than reading the -h or looking for an answer on stackoverflow, especially with google's greed that it stopped showing stackoverflow first and sometimes leaves it to the 2nd page and fills the first with garbage, started using yandex as an alternative to google a year ago, now using it by default and google from time to time, it's sad that the best product is behind us.
>Yeah it's great as an assistant to a critical thinking human, people who are criticizing it are either doing it for clickbait aka money or expected it go think for them.
You forgot the hyped up people who blindly tell you the latter and thus force you to state its limitations loudly. Every time I see these trolls I worry that I am too stupid to enter the right prompts but no, it then always behaves exactly how I think it should.
You forgot the hyped up people who blindly tell you the latter and thus force you to state its limitations loudly. Every time I see these trolls I worry that I am too stupid to enter the right prompts but no, it then always behaves exactly how I think it should.
I think people are not criticising the good uses but the potentially really bad ones. While Im welcoming a human helper artificial agent I am uneasy about the upcoming general downgrading of human labor. Big money will benefit from this more than anybody else
It's like the friendly, patient forum poster who answers your questions and explains things in a "how to make a peanut butter sandwich" manner we all wished we had but are hard to come across.
I like that! A “friendly, patient forum poster” is a great analogy.
This makes me think that what’s really missing from these new AI is a response that asks for more context, instead of directly attempting to give you a correct answer. Often my questions don’t have enough contexts to be answered correctly but the AI never tells me that
Tell ChatGPT that you want it to ask for more context when it's needed and it will do so. I've done this before and it will continue to ask more probing questions until it's satisfied. You kind of have to guide it a little bit but it definitely works.
What you're saying really blows my mind btw
Why? When you intentionally keep your prompts vague it will ask you to clarify. Telling it directly to ask for more context sounds about right.
I just can’t fathom how these AI work. It seems like impossible to me
I'm working on an "AI Programmer" website and have attempted to do a requirements interview like that to get more context. Hoping to get the new release out within a week.
[deleted]
I was asking it some medical related questions today and I had the same thought. “What are some medical conditions that could cause constant fatigue, a weak immune system, and circulation issues resembling Reynauld’s syndrome?”
To which is could reply “Does the patient have skin lesions or joint pain?” to eliminate possibilities.
To which is could reply “Does the patient have skin lesions or joint pain?” to eliminate possibilities.
> Google to understand my confusion and endless ads thrown my way,
Another reason why Google is terrified here. Search is per query cheap. GPT isn't.
Another reason why Google is terrified here. Search is per query cheap. GPT isn't.
If it were down to cost per query, they wouldn't have implemented search-as-you-type, which is probably 5x as expensive as waiting for the user to hit the enter button.
I guess many companies will pay for Github Co-pilot, ChatGPT style things.
No one will pay for access to a search engine.
No one will pay for access to a search engine.
Kagi, a paid search engine, and I, one of their customers, disagree, for what that's worth.
Holy shit this looks AMAZING. Rule out domains from searches entirely? Sold!
I'm still so lost on why they're making a browser only for Apple devices
Could be resource limitation, apple users tend to be more likely to pay for products so it makes more sense to target them exclusively
Regarding that Mozilla and Google now work on their own browser engines for iOS I get the feeling that Kagi should drop Orion and maybe work on their own mail thingy or something else.
But what do I know?
But what do I know?
I tried the 50 free queries, and it was basically exactly the same experience as DDG. I can maybe see paying $10-$20/yr, but $10/mo? No way.
mandatory HN Kagi plug
Honestly if it was not an advert riddled mess tracking my every move I would definitely pay for search.
ChatGPT/Bing is a superior rival to Google's I'm feeling Lucky button, thats a threat to Google's search engine dominance.
When you have two choices and don't really know which one to pick, there's an old trick to throw a coin. Go with the coin, unless you really dislike the outcome, which tells you that the other choice was the right one all along.
I use ChatGPT in a similar manner, for example when writing documentation, I'll ask it: "What are the important topics that should be covered in a documentation on X?" and "Can you give me a suggestion for a structure of such a documentation?". Now I can easily follow along, unless I feel like it is a bad approach.
With coding, it is almost like an improved rubber duck. I don't trust the replies, but the engagement is helping me structure my thoughts.
I use ChatGPT in a similar manner, for example when writing documentation, I'll ask it: "What are the important topics that should be covered in a documentation on X?" and "Can you give me a suggestion for a structure of such a documentation?". Now I can easily follow along, unless I feel like it is a bad approach.
With coding, it is almost like an improved rubber duck. I don't trust the replies, but the engagement is helping me structure my thoughts.
This is exactly how I use it now. It's like a custom SO helper. ChatGPT has read all the doc's. If you can ask the right question it'll give you a good boost.
But the building, that falls back on the engineer.
But the building, that falls back on the engineer.
Of course, AI models are often wrong. Even if they were right in the sense that they gave you exactly what you asked for, they may still be wrong in the sense that their interpretation of your requirements may not be what you intended; they are not great at filling the gaps if there is any context missing. The same amount of time it takes to provide an AI with all the necessary, nuanced context to produce an optimal solution could have been used to actually produce the desired optimal solution yourself. The hard part of software development is not coding, it's the amount of critical thinking and synthesizing required to produce optimal solutions.
I'm optimistic about AIs because they may finally prove once and for all that software engineers cannot be commodified as easily and thoroughly as most other white collar roles. It may be easier to commodify executives who make a small number of abstract high level decisions than it is to commodify developers who make many extremely nuanced, detailed decisions every day.
For sure, AI will replace certain kinds of developers, but not all kinds, at least not for some time. I think senior developers and architects will be among the last people in the firing line.
I'm optimistic about AIs because they may finally prove once and for all that software engineers cannot be commodified as easily and thoroughly as most other white collar roles. It may be easier to commodify executives who make a small number of abstract high level decisions than it is to commodify developers who make many extremely nuanced, detailed decisions every day.
For sure, AI will replace certain kinds of developers, but not all kinds, at least not for some time. I think senior developers and architects will be among the last people in the firing line.
> I think senior developers and architects will be among the last people in the firing line.
And soon there will be no junior developers left to rise up to their ranks... So "last" in more ways than one!
And soon there will be no junior developers left to rise up to their ranks... So "last" in more ways than one!
My worry is code complexity explosion. Once we start using AI to add features to infrastructure (OS's, compilers, data serialization, etc) it will quickly run away from our ability to understand it. It has already run away, it will now be fucking _flying_.
I think AI can replace the "copy everything from Stack Overflow" type of developer but I doubt it is capable of replacing the competent developer. The currently popular form of AI is likely to hit a local maximum just like machine translation has where it is reasonably close most of the time on easy or intermediate problems but sometimes breaks in bizarre ways and doesn't work at all for the hard problems.
Since speeding up the work of incompetent developers by a factor of 100 is a good way to end up with an unmaintainable disaster that will break in bizarre ways that are very hard to fix, I think we probably are going to see complexity balloon way beyond human comprehension in every codebase foolish enough to use it. Just like those incompetent developers create more work for competent developers, AI will create more work for human developers.
Since speeding up the work of incompetent developers by a factor of 100 is a good way to end up with an unmaintainable disaster that will break in bizarre ways that are very hard to fix, I think we probably are going to see complexity balloon way beyond human comprehension in every codebase foolish enough to use it. Just like those incompetent developers create more work for competent developers, AI will create more work for human developers.
Have you actually tried using an LM for programming?
ChatGPT catches me off guard sometimes even when working on "boring" problems. For example, I was consuming structs from a UNIX domain socket in Java and had some tedious translation from C-types to ByteBuffer calls.
It's brain dead easy thing to do, the structs were defined as packed so not alignment issues, ByteBuffers handle endianness, there's a little bit of trickiness with unsigned types but nothing terrible... but with 6 types and writing tests for all 6 definitions, I wondered if ChatGPT could do it and save me a couple of minutes.
What impressed me wasn't that it managed to generate the near identical code to my example definition,it was the fact it recognized one of the fields as padding and pre-emptively discarded its value.
The padding wasn't in any way shape or form covered by typing, yet the fact the comment in the C definition mentioned padding caused ChatGPT to generate code that skipped over the padding instead of trying to needlessly capture its value and commented that was doing so.
And when I had it generate unit tests, it managed to maintained context on the padding and added a helpful comment pointing it out.
-
It's small corner cases like that which really make me see LMs as being the natural productivity multiplier for programmers.
There's a ton of nuance in some problems that by the very nature of an LM, you'd never imagine it'd be able to capture. Yet when you press on it, it actually does better on those than it does on "copy everything from Stack Overflow" problems.
ChatGPT catches me off guard sometimes even when working on "boring" problems. For example, I was consuming structs from a UNIX domain socket in Java and had some tedious translation from C-types to ByteBuffer calls.
It's brain dead easy thing to do, the structs were defined as packed so not alignment issues, ByteBuffers handle endianness, there's a little bit of trickiness with unsigned types but nothing terrible... but with 6 types and writing tests for all 6 definitions, I wondered if ChatGPT could do it and save me a couple of minutes.
What impressed me wasn't that it managed to generate the near identical code to my example definition,it was the fact it recognized one of the fields as padding and pre-emptively discarded its value.
The padding wasn't in any way shape or form covered by typing, yet the fact the comment in the C definition mentioned padding caused ChatGPT to generate code that skipped over the padding instead of trying to needlessly capture its value and commented that was doing so.
And when I had it generate unit tests, it managed to maintained context on the padding and added a helpful comment pointing it out.
-
It's small corner cases like that which really make me see LMs as being the natural productivity multiplier for programmers.
There's a ton of nuance in some problems that by the very nature of an LM, you'd never imagine it'd be able to capture. Yet when you press on it, it actually does better on those than it does on "copy everything from Stack Overflow" problems.
I expect that before my career is over that I will be given more than one project to fix that the AI stopped being able to deliver features for.
I imagine it's going to be one of those laugh so you don't cry affairs.
I imagine it's going to be one of those laugh so you don't cry affairs.
Maybe laugh because you're going to get paid for a really long time.
(And then, yeah, cry because of what you're going to have to do to earn the money...)
(And then, yeah, cry because of what you're going to have to do to earn the money...)
I like it. We're generating "COBOL" debt for our future brethren as we solve present problems with the most modern tools we had.
Yep. Every line of code in your codebase increases the cost of things like refactors and bug investigations. If developers can write code 10x as fast but only refactor or understand it 2x as fast... well then we have an interesting problem.
100%, Programming is all about trying to keep control, trying to prevent it from becoming chaos, trying to ensure you still understand the program you wrote at the end of the day.
With out doing any of this it's easy, easy not to care... sometimes I do this with throw away code and suddenly it feels like I can fly, I can write thousands of lines so fast, progress is lightning fast, and then it all turns to shit really quickly and the friction of chaos grinds it to a halt - at which point I only validate my normal slow mode of plodding through things carefully and minimally.
With out doing any of this it's easy, easy not to care... sometimes I do this with throw away code and suddenly it feels like I can fly, I can write thousands of lines so fast, progress is lightning fast, and then it all turns to shit really quickly and the friction of chaos grinds it to a halt - at which point I only validate my normal slow mode of plodding through things carefully and minimally.
My worry is that AIs reduce the cost for bad actors to engage in harmful behaviour. The companies building these AIs need to take ethics into account. Making it easier to create fake images, video and audio of real people causes harm and should be considered unethical.
We need to strike a balance between the freedom to create new things and the externalization of the cost of the harm these tools inflict upon society. Law makers don't seem to have a clue about the consequences of what's happening, so my expectations are pretty low at this point. Overall, I no longer believe that many new technologies are a net good for society.
We need to strike a balance between the freedom to create new things and the externalization of the cost of the harm these tools inflict upon society. Law makers don't seem to have a clue about the consequences of what's happening, so my expectations are pretty low at this point. Overall, I no longer believe that many new technologies are a net good for society.
> Making it easier to create fake images, video and audio of real people causes harm and should be considered unethical.
Is photoshop unethical? What should we do to balance the harm it causes to society? Should it be outlawed? What about MS paint, or even word processors? Were harms caused to society by the printing press? Was it all a terrible idea?
AI could create tools that end up doing more harm than good. AI could design literal malware. It's unclear if they'd do a better job of it than we do ourselves, and either way it seems likely that it'll take a human to use the code AI produces (or at the very least to direct the AI to use it against other humans)
The technology is rarely the problem. It's almost always the people who choose to use that technology for evil. However involved AI becomes in what gets created, until AI becomes actually intelligent and the singularity begins it's people, not technology, we need to worry about.
I'm skeptical that faked video and audio will be the end of us. We've had doctored images and impersonators for ages. We've had hoaxes and liars. We'll get smarter and learn to put less trust in random things we find on the internet, just like we're gradually learning not to believe everything we see on TV or that nigerian prices aren't going to make us rich beyond our wildest dreams.
Is photoshop unethical? What should we do to balance the harm it causes to society? Should it be outlawed? What about MS paint, or even word processors? Were harms caused to society by the printing press? Was it all a terrible idea?
AI could create tools that end up doing more harm than good. AI could design literal malware. It's unclear if they'd do a better job of it than we do ourselves, and either way it seems likely that it'll take a human to use the code AI produces (or at the very least to direct the AI to use it against other humans)
The technology is rarely the problem. It's almost always the people who choose to use that technology for evil. However involved AI becomes in what gets created, until AI becomes actually intelligent and the singularity begins it's people, not technology, we need to worry about.
I'm skeptical that faked video and audio will be the end of us. We've had doctored images and impersonators for ages. We've had hoaxes and liars. We'll get smarter and learn to put less trust in random things we find on the internet, just like we're gradually learning not to believe everything we see on TV or that nigerian prices aren't going to make us rich beyond our wildest dreams.
There’s a theory I’ve seen in occasional security related things where you publicly publish your private keys when you’re done with them. That way later on you gain some plausible deniability related to any communications and data found later.
When Facebook was new there were stories about people failing interviews because some employer say a picture of them at a bar on Facebook (as if most people don’t go to bars when they’re young). Obviously that doesn’t really happen anymore, but I consider a lot of AI content generation to be the “releasing private keys” for your digital footprint. Was I at that bar or was it stable diffusion?
When Facebook was new there were stories about people failing interviews because some employer say a picture of them at a bar on Facebook (as if most people don’t go to bars when they’re young). Obviously that doesn’t really happen anymore, but I consider a lot of AI content generation to be the “releasing private keys” for your digital footprint. Was I at that bar or was it stable diffusion?
The code might start to resemble biological systems, which are a mess of complexity.
In the future, programs will develop genetic disorders, and will need to be diagnosed by programmers with the equivalent of medical degrees.
Unfortunately, doctors might be replaced by diagnostic AIs. It'll be AIs all the way down by then.
Someone will need to maintain these diagnostic AIs, doctors will take to programming too in some way or another. Unless we devolve into a giant sass where we’ll all become like cells to in larger organism. I have no doubt some will adapt but I wonder whether the humanity in people will survive. AI will certainly be able to make a very good imitation of it though
Considering bugs used to literally be bugs, that’s not far off.
We don't know that was the case: https://en.wikipedia.org/wiki/Bug_(engineering)#History
I think we might get AI that can help in understanding systems. We could run Copilot in reverse and generate documentation.
(Some people are already trying this with generation of commit messages)
(Some people are already trying this with generation of commit messages)
> A ChatGPT-style AI that can only reply with links to reference docs, libraries, or wikipedia articles
This is exactly what we've built at Inngest so that you can learn our SDK: https://www.inngest.com/ai-personalized-documentation.
It can only reply with SDK examples, an explanation, and links to relevant guides. Soon, we're revamping our docs to highlight sections relevant only to your use case - you'll be able to type what you want to do with Inngest and be pointed straight to the relevant guides and docs.
We trust developers to write the code, and only use GPT to assist and point the right way. It's a good compromise, and it's the most immediately applicable use that I can see!
This is exactly what we've built at Inngest so that you can learn our SDK: https://www.inngest.com/ai-personalized-documentation.
It can only reply with SDK examples, an explanation, and links to relevant guides. Soon, we're revamping our docs to highlight sections relevant only to your use case - you'll be able to type what you want to do with Inngest and be pointed straight to the relevant guides and docs.
We trust developers to write the code, and only use GPT to assist and point the right way. It's a good compromise, and it's the most immediately applicable use that I can see!
I know this is getting downvoted for being an ad but this is exactly what I was talking about, so thank you I will look into it!
Humans may he bad at proofreading code, but we’ve had a lot of experience mitigating that too - what else are tests for?
Maybe coding AIs will push everyone into test driven development - write a spec/tests of whatever then just wait awhile for it to crunch through code until something works.
This still assumes that the root of our modern ills as a profession is "we don't have enough code", when in fact the root of our ills is that "the code we have is all crap". Being able to generate code that is worse than ever, faster than ever, will not result in any improvement at all to the state of software reliability.
Frog march the code through a valley of death such that only reliable software survives at the other end. Use Ai to automate the simulation.
Extremely well put!
That's an interesting idea, it sounds almost like using a fuzzer for generating new code instead of finding bugs in existing code.
Until someone makes the AI write the test too, I don't think there's going to be any red-tape style solutions to this problem that isn't just "no AI written code allowed".
What's more likely to happen is "no human written code allowed". I could easily imagine a future where people are socially ostracized for writing code manually because it's dangerous, unsafe, and insecure. In that future, Rust developers will get a taste of how they made C / C++ programmers feel.
I mean either it works or it doesn't. That is, either I can ask the AI to solve literally any problem that I have or I still need a software engineer to tell me why my goals are stupid and ill formed.
So for example I ask the AI to program a replacement for dentists. The AI tells me that you can't fix dentistry with software. So I ask it what could fix dentistry and ask it to program that. And the AI can't program that either. But you just keep asking for the solution to the thing that the AI cant solve and eventually you get it to spit out some gcode for a 3D printer and you bootstrap your way up to AI replaced dentistry.
So either the AI can replace anything or you need some person who can translate domain knowledge into source code.
Yeah eventually rust devs get their comeuppance but in that day they're also the last employed human profession.
So for example I ask the AI to program a replacement for dentists. The AI tells me that you can't fix dentistry with software. So I ask it what could fix dentistry and ask it to program that. And the AI can't program that either. But you just keep asking for the solution to the thing that the AI cant solve and eventually you get it to spit out some gcode for a 3D printer and you bootstrap your way up to AI replaced dentistry.
So either the AI can replace anything or you need some person who can translate domain knowledge into source code.
Yeah eventually rust devs get their comeuppance but in that day they're also the last employed human profession.
Traditionally speaking people used the word God to describe someone who's capable of solving every problem. Talk about raising the bar. Sure you'll need a person. That person could be an AI.
> Talk about raising the bar.
Uh, no I guess my point is that once all of the programmers/software engineers/software developers are out of a job, then everyone else will be as well. I think that if there's literally anyone else working because the AI can't do their job in totality, then there's also a (potential) reason for that person to want some sort of human software developer.
But when everyone can avoid human software developers because the AI can do it, then everyone else can avoid every human professional because the AI can do it.
It would not surprise me if this is a gradual process where at some point there is a 50/50 mix of humans and AI doing software development.
Uh, no I guess my point is that once all of the programmers/software engineers/software developers are out of a job, then everyone else will be as well. I think that if there's literally anyone else working because the AI can't do their job in totality, then there's also a (potential) reason for that person to want some sort of human software developer.
But when everyone can avoid human software developers because the AI can do it, then everyone else can avoid every human professional because the AI can do it.
It would not surprise me if this is a gradual process where at some point there is a 50/50 mix of humans and AI doing software development.
Actually, it probably could write really good test cases. they tend to be boring and verbose.
It's only test data but you get the point
Sure, here are some examples of phone numbers in various formats:
(123) 456-7890 123-456-7890 +1 (123) 456-7890 1-800-123-4567 555.123.4567 123.456.7890 1234567890 (123) 456-7890 ext. 123 +44 123 456 7890 (international format) 011 44 123 456 7890 (international access code format) 123/456-7890 123\456-7890 (123) 456-7890 x1234 (with extension) 123-456-7890 x1234 (with extension) 1 (123) 456-7890 (with country code)
It's only test data but you get the point
Sure, here are some examples of phone numbers in various formats:
(123) 456-7890 123-456-7890 +1 (123) 456-7890 1-800-123-4567 555.123.4567 123.456.7890 1234567890 (123) 456-7890 ext. 123 +44 123 456 7890 (international format) 011 44 123 456 7890 (international access code format) 123/456-7890 123\456-7890 (123) 456-7890 x1234 (with extension) 123-456-7890 x1234 (with extension) 1 (123) 456-7890 (with country code)
I use Copilot to write my tests, but using ChatGPT to generate test data could be a good idea. I'll have to fiddle with it when i'm not on a timer.
Thank you for the suggestion.
Thank you for the suggestion.
Out of curiosity I tried chat GPT for this and it was able to generate a reasonable set of test cases but only after prompting it for including international access and "+" characters. I also had to prompt it to use the parameterize feature, but I guess that's personal preference.
Final test: https://pastebin.com/J36JY78i
First response: https://pastebin.com/S23GFPDq
Final test: https://pastebin.com/J36JY78i
First response: https://pastebin.com/S23GFPDq
All great test cases. Here are a few more test cases for valid phone numbers:
0893514656, 089 3514656, +49893514656, 08921610, 089 2 16 10, 08992949451 ...
0893514656, 089 3514656, +49893514656, 08921610, 089 2 16 10, 08992949451 ...
[deleted]
Iteration via adding more test cases... Sounds like hell, I'd find a new job.
I think the economic reality is that for a lot of software, especially user-facing software, "correct enough that it passed manual QA" is the threshold. It already is; even pre-AI. So I think AI + proofreading will be successful there, and it may not even make things worse than they currently are, especially in the long run
While critical logic, that really really has to be correct, to a point where you should maybe have it all planned out before even writing code, will still need to be hand-programmed for a long time
While critical logic, that really really has to be correct, to a point where you should maybe have it all planned out before even writing code, will still need to be hand-programmed for a long time
I think the problem is that code that passes manual QA testing is fine until it suddenly isn’t. That may be due to the complexity of the code base, the turn around time on doing those tests, the accuracy of people following scripts, or just the number of tests and the size of the code base.
Similarly integration tests are fine until they suddenly aren’t, and you really need things like unit testing.
Unit testing also only works up to a point without tools to reason about the entire codebase and how it interfaces with others.
Organizations flourish or flounder based on recognizing when they are approaching those points and dealing with their underlying issues. Maybe AIs can write code that is good enough in a lot of cases, but unless people can understand that code and how to work with it and refactor it as things grow then they may not be a good idea.
Similarly integration tests are fine until they suddenly aren’t, and you really need things like unit testing.
Unit testing also only works up to a point without tools to reason about the entire codebase and how it interfaces with others.
Organizations flourish or flounder based on recognizing when they are approaching those points and dealing with their underlying issues. Maybe AIs can write code that is good enough in a lot of cases, but unless people can understand that code and how to work with it and refactor it as things grow then they may not be a good idea.
All of those things are true, and yet, that's how a lot of software is built in practice because it still ends up "close enough". The market has shown that this is acceptable in a whole lot of domains
I'm not saying I like it, I'm saying it's the reality
I'm not saying I like it, I'm saying it's the reality
These are really solid concerns. This shift from "creator" to "supervisor" is happening in lots of AI areas - look even at how self-driving cars are currently expecting their human drivers to act as supervisors, which they're bad at.
Also, on this point:
> We can make proofreading easier with better tooling. Unit tests are a means of “proofreading”: we can catch AI errors automatically with tests.
The unit tests will definitely be written with AI as well. If that AI and the coding AI are wrong in the same way, we will have very subtle and hard to catch errors.
Also, on this point:
> We can make proofreading easier with better tooling. Unit tests are a means of “proofreading”: we can catch AI errors automatically with tests.
The unit tests will definitely be written with AI as well. If that AI and the coding AI are wrong in the same way, we will have very subtle and hard to catch errors.
That problem sounds just like when I write tests for my own code. I don’t know if it’s totally avoidable.
It just now occurred to me that a natural solution to this problem could be to move past code review to "other teammate writes the tests for your PR."
You owe them an additional beer at the next happy hour for each test they write you wind up pushing a fix for.
You owe them an additional beer at the next happy hour for each test they write you wind up pushing a fix for.
The “almost correct” aspect will be especially fun when AIs are used to refactor code. I was thinking about how a future legacy codebase might look like after a decade of having AI-generated code added to it, and then of course you would want the AI to apply some refactorings from time to time.
One might argue that it won’t be a problem with proper tests. But do coders want to spend most of their time proofreading AI-generated code and taking care of proper test coverage? That doesn’t sound very appealing.
One might argue that it won’t be a problem with proper tests. But do coders want to spend most of their time proofreading AI-generated code and taking care of proper test coverage? That doesn’t sound very appealing.
Why can’t the AI write tests too? Then you can have 100% coverage of your (probably) correct code…
That's even worse, because now the tests are wrong too! I asked it for a list of test cases for a particular function I wanted to write, and every single one of the cases it presented was incorrect. So it was worse than useless, now I had to check its work and then do the work myself anyway.
It can, but you have to proofread those too.
> The “almost correct” aspect will be especially fun when AIs are used to refactor code.
Hope your function has all invariants covered with one or more property tests!
Hope your function has all invariants covered with one or more property tests!
I’m sure the AI has added all necessary tests like I told it to!
Programming AIs worry me because I think they can get better at programming faster than I can get better at programming
I once coined an aphorism: “Deepmind can train a new model faster than you can go back to grad school.”
What worries me are the low-grade programmers. By delegating their work to tools like ChatGPT (and doing so always a level or two beyond the current abilities of the tools) they will manage to take us to newer lows.
[deleted]
Underrated. It already happened in frontend space. Job market is already rotten by incompetent tech leaders who aren't immune to all the bullshit and bought all the hype from VC-backed bigco.
With the new advancements in AI, it's expected that there will be more code being generated by programs rather than written by programmers. Making sure these programs deliver accurate and error free code is a difficult task. The challenge here lies in effectively proofreading AIs so that any bugs are caught before they reach end users. Good tools such as unit tests can help but may not be enough to make sure AI generated code is flawless. As the use of AI increases, we must look to develop better methods for ensuring quality assurance on these programs if they are to truly be useful.
> it's expected that there will be more code being generated by programs rather than written by programmers.
That has probably been case since the 80s; nearly all code running on your CPU is generated by a program ("compiler") from a textual description ("source")
That has probably been case since the 80s; nearly all code running on your CPU is generated by a program ("compiler") from a textual description ("source")
That's not what is meant. Programming languages are basically huge macro systems that expand to assembly, but the code is still written by a human.
Layers of abstraction are a problem too, but they are a different problem.
Layers of abstraction are a problem too, but they are a different problem.
In my career I've seen resources pulled away from QA teams (sometimes completely disbanding). It would be interesting to see things swing the other way.
This feels GPT generated.
Prediction: An inflection/tipping point will occur when an AI has the ability to programme/create another AI with capabilities or coding superior to its own. That's the moment. It will snowball from there. A begets B which is better than A; B begets C which is better than B; and so on. After that point it will be limited only by processing capability and network connectivity.
Insightful! Of all the creative prompt engineering I've read, I haven't found that yet: the equivalent of "fork a new process and while true."
Hey, ChatGPT. Ask this question at this URL when I talk about farms. But while you do that, ask it to also think on farms. Every few seconds, append its responses to every answer you give me.
Hey, ChatGPT. Ask this question at this URL when I talk about farms. But while you do that, ask it to also think on farms. Every few seconds, append its responses to every answer you give me.
The work of a master craftsman using the most basic hand tools is still leagues ahead of any amateur with advanced tools, automation, fancy CNC, and lasers. Mass production is where the tech would favour that amateur. We are surrounded by cheap mass-produced disposable products and are well aware of their low quality. No gourmet chef should be threatened by the efficiency of McDonald's. Fortunately, our software systems are custom artifacts with zero marginal cost of reproduction. AI will make competent programmers more productive and take expert programmers to new levels.
One transformation will come when a single developer can do the work of a team of ten today. Like solo content creators who no longer need large studios to finance and organise massive production teams and expensive equipment, we'll see more independent developers and small teams creating innovative software with some artistic character. Without the drag of management and upfront financing, the threshold for success will be much lower.
Another will be a market for lemons. Employers will have to find ways to establish competence at building high quality, understandable and maintainable systems. AI can already pass Leetcode-style tests so these are clearly not the right measures.
This is my best optimistic take on things. The rest of the time I'm worried the AI is coming for our jobs.
One transformation will come when a single developer can do the work of a team of ten today. Like solo content creators who no longer need large studios to finance and organise massive production teams and expensive equipment, we'll see more independent developers and small teams creating innovative software with some artistic character. Without the drag of management and upfront financing, the threshold for success will be much lower.
Another will be a market for lemons. Employers will have to find ways to establish competence at building high quality, understandable and maintainable systems. AI can already pass Leetcode-style tests so these are clearly not the right measures.
This is my best optimistic take on things. The rest of the time I'm worried the AI is coming for our jobs.
heh, this was true for stackoverflow or for googling pretty much anything. Nothing new. The tools are not doing our job, they're just making the job easier.
Your head is in the sand. Some time soon, they can and will actually do the whole job.
Human beings struggle to communicate system requirements to other human beings. We're supposed to be "smart", communication is supposed to be something we're good at.
Getting requirements to an AI is going to be grotesquely challenging, and even when we're able to get that far, testing the work is going to be an ever greater challenge.
"Exactly what I asked for, but not what I wanted" is going to rise to an entirely new level.
Getting requirements to an AI is going to be grotesquely challenging, and even when we're able to get that far, testing the work is going to be an ever greater challenge.
"Exactly what I asked for, but not what I wanted" is going to rise to an entirely new level.
But if you can try asking for different things and it takes 10s to create, it might be easier to just iterate like that than ask a developer to spend a day on it.
[deleted]
[deleted]
Nah, never the whole job. Software engineering isn't 100% writing code, so even if an AI could write 100% of the code, there's still all of the other things around it that need to get done.
I am actually super excited for better AI tools to help write so much of the boilerplate crap I have to write. I'm here because I want to solve problems, not write code, and writing less code makes me happy.
I am actually super excited for better AI tools to help write so much of the boilerplate crap I have to write. I'm here because I want to solve problems, not write code, and writing less code makes me happy.
In the book zero to one thiel makes the case that the next boom is not in replacing humans, it’s in assisting humans. IMO he was right a long time ago (not sure when this book was written) and continues to be right to this day. I’m not saying AGI is not coming, but this isn’t it just yet.
as someone who has used copilot/chatgpt since they became available, I must say we are still a ways off.
Mainly, as others have pointed out, only a small % of coding is typing out code.
Mainly, as others have pointed out, only a small % of coding is typing out code.
It looks like it will be able to do every non-programming office job first.
I agree completely. People are forgetting that programming is a difficult technical and creative task. If an AI can master that, it should be able to master many tasks which are far less technical and/or far less creative.
I'm looking forward to seeing how AI manages accounting, that's a field that requires even more creativity than programming.
Why do you believe accounting requires more creativity than software development?
When the domain of possible software projects is infinite, I find this really hard to imagine.
When the domain of possible software projects is infinite, I find this really hard to imagine.
It's a "creative accounting" pun. Think of filing dinners at nice restaurants as business expenses, but cranked up to 11.
Which is what we call "hallucinations" in ChatGPT, it is really good at that. It works really well for creative accounting, similarly creative programming just crashes your app.
And that is the whole reason ChatGPT will replace accounting and such jobs before programming, because those jobs are much more tolerant of inaccurate statements than programming is.
And that is the whole reason ChatGPT will replace accounting and such jobs before programming, because those jobs are much more tolerant of inaccurate statements than programming is.
[deleted]
By then they should also be able to do or make redundant 90% of jobs in general.
If the AI can make the entire product then it can probably market the product, provision resources, handle accounting, user support, etc.
At that point we’ve cracked capitalism and our ability to create has far surpassed our ability to consume.
If the AI can make the entire product then it can probably market the product, provision resources, handle accounting, user support, etc.
At that point we’ve cracked capitalism and our ability to create has far surpassed our ability to consume.
I'm not sure I believe that and It's never going to be that magical.
If you're an accountant, Excel will likely continue to improve to ingest financial data and help you generate reports faster than before. They are usually overworked anyway, so they will welcome improvements there.
My partner writes education grant proposals, and having contextualized search like ChatGPT makes writing those grants more persuasive. Since they could have access to more statistics and data. HOWEVER, note, the data in ChatGPT is already stale and getting new data in there is complex and will require someone to do the actual research.
All kinds of examples of things that I see where it doesn't replace the job, but improves its velocity.
If you're an accountant, Excel will likely continue to improve to ingest financial data and help you generate reports faster than before. They are usually overworked anyway, so they will welcome improvements there.
My partner writes education grant proposals, and having contextualized search like ChatGPT makes writing those grants more persuasive. Since they could have access to more statistics and data. HOWEVER, note, the data in ChatGPT is already stale and getting new data in there is complex and will require someone to do the actual research.
All kinds of examples of things that I see where it doesn't replace the job, but improves its velocity.
But that isn't what this thread is about. The thread is about AI having perfect domain knowledge about every single topic and being able to write any software with that or else you are sticking your head in the sand.
They will allow people to increase efficiency and thereby lowering the amount of time and cost for a developer.
Soon complex web platform development will be like building a wordpress website, you’ll have 10 concurrent clients or projects rather than just one.
Your head is in the sand.
You know what "whole job" means, right?
You know what "whole job" means, right?
Heh, I should develop a tools searches for the first SO response with code and then check to see how often that response contains a security flaw.
Before we have AIs that can program, we will have AIs that can do computer administration tasks, like solving dependency problems with software installations etc. and compiling and installing stuff.
So before we have a tool that can do that, I'm not worried.
So before we have a tool that can do that, I'm not worried.
It would be fun to see one attempt a Gentoo install.
I haven't had much luck using these AI's past surface level stuff.
They are godsent at getting a TikZ drawing jumpstarted.
It's interesting that Hillel, a huge formal methods and specification advocate, sees this as a proofreading problem and not a specification one. I can't say I agree.
I'd be interested in hearing more about your disagreement! If I had to reconcile this post with my overall shameless FM promotion, I'd say that "proofreading is hard" is the human limit that is amplified by AI use, and more specification is a potential solution. But also, historically, an expensive one, so I don't know if that's going to adopted enough to fully offset the drawbacks of mass AI adoption.
I read this but actually not sure what the author is worried about.
I guess they feel proofreading AI-written code is harder than writing code?
That seems like a leap to me. I don't expect AI-assisted coding to be a panacea, and I'm sure there are ineffective ways to do it. But it seems like there are effective ways too.
So, sure, there is work to do to figure out how to use these tools in the best way. But that's not anything to worry about, IMO. It's opportunity.
I guess they feel proofreading AI-written code is harder than writing code?
That seems like a leap to me. I don't expect AI-assisted coding to be a panacea, and I'm sure there are ineffective ways to do it. But it seems like there are effective ways too.
So, sure, there is work to do to figure out how to use these tools in the best way. But that's not anything to worry about, IMO. It's opportunity.
> Might as well share a wishlist. [1/] A ChatGPT-style AI that can only reply with links to reference docs, libraries, or wikipedia articles
That already exists. It's called an effing search engine. But for some reason people have decided, as of ten days ago, that they're not good enough anymore.
As for why? I really have no clue. But they still work extremely well.
That already exists. It's called an effing search engine. But for some reason people have decided, as of ten days ago, that they're not good enough anymore.
As for why? I really have no clue. But they still work extremely well.
> As for why? I really have no clue. But they still work extremely well.
I've found Google to be significantly less useful than it used to be. Largely because I have to jump through hoops to convince it to show me what I'm actually looking for. ChatGPT is already much better often enough that it can turn 20+ minutes of faffing about with Google into one minute or less of just asking for what I want and getting it.
Yeah, it's wrong sometimes, but it still manages to point me in the right direction often enough to be worth the need for extra vigilance
I've found Google to be significantly less useful than it used to be. Largely because I have to jump through hoops to convince it to show me what I'm actually looking for. ChatGPT is already much better often enough that it can turn 20+ minutes of faffing about with Google into one minute or less of just asking for what I want and getting it.
Yeah, it's wrong sometimes, but it still manages to point me in the right direction often enough to be worth the need for extra vigilance
<< Laughs >>
Oh wait, you're serioous?!
<< Laughs harder >>
AI is great for making syntactically correct code; and so is copy-pasting the first result that works after a low quality search.
It still takes real minds to make code that is worth anything, and there will always be buffoons making code that doesn't work. Except the HN website, of course. ;-)
Oh wait, you're serioous?!
<< Laughs harder >>
AI is great for making syntactically correct code; and so is copy-pasting the first result that works after a low quality search.
It still takes real minds to make code that is worth anything, and there will always be buffoons making code that doesn't work. Except the HN website, of course. ;-)
If reading is harder than writing code, why bother with tools that deal with the later instead of the former?
The author begins by giving a number of test cases, like f(35)==6, for which the AI provides a candidate f, but the author doesn't actually implement the test cases, is surprised when the AI is wrong, and laments the resulting burden of proofreading.
Just implement those tests, for heaven's sake!
Just implement those tests, for heaven's sake!
Implementing a correct and comprehensive set of test cases for code is more time consuming and often more difficult than writing the code itself. (The comprehensive part especially - how do you prove to yourself you've covered all the necessary edge cases?)
"Reading code is harder than writing it"
"Debugging code is harder than writing it"
These are pretty common statements, going back years.
They should make the lack of a silver bullet here very obvious.
(Yes, most software is already broken in various ways - my worry is that this will just make it more broken, e.g. non-technical management saying "why do we need to do as much QA, we have AI writing the code now!?")
"Reading code is harder than writing it"
"Debugging code is harder than writing it"
These are pretty common statements, going back years.
They should make the lack of a silver bullet here very obvious.
(Yes, most software is already broken in various ways - my worry is that this will just make it more broken, e.g. non-technical management saying "why do we need to do as much QA, we have AI writing the code now!?")
Yes, writing good tests is hard.
My point is that he already has the relevant test cases. The hard work is done.
My point is that he already has the relevant test cases. The hard work is done.
To clarify, Bertrand Meyer is the one who gave the test cases and accepted the wrong answer without verifying them. I'm the one who lamented the resulting burden of proofreading.
even easier: ask the LLM for those test cases!
Oh wow. That IS what I want. An AI that can offer a code review and suggest things.
it's like alexa, siri, etc. I'd rather do the stuff myself than talking nonsense with an "ai"
To be fair, I’ve seen humans do worse.
Relax.
AI assisted programming is brand new.
It's just the start. So if it gets things wrong, is missing features, annoys people, remain cool, gigantic leaps are coming far beyond your wildest dreams.
And there will be lots of people around right now showing how smart they are by making AI do wrong or silly things. Ignore them - none of these prove that AI is the wrong direction.
AI programming is the single most exciting thing to ever happen to programming - a permanent mentor and pair programmer at your side levels up every programmer in the world.
Even more exciting is innovations that are a direct outcome of AI assisted programming.
When the world wide web first came along, everyone made a website that was like a company brochure. That was Web 1.0 and that's all people had the imagination for. In time, the usages of the web started to arrive that leveraged the core values of the technology, such as interconnection between people. That's where AI programming is now.
We are at AI 1.0
Programming models will come out that are primarily oriented to the strengths of AI programming - but first someone needs to think them up. And they will not only be surprising but absolutely mind blowing. It's entirely possible that in the future the entire internet has an AI sublayer that does all the work of systems talking to systems and people simply don't have anything to do with it. It's possible that the nocode movement is the foundation of the next great innovations in programming. Existing programming models are like the company brochures of web 1.0 - our old, hard, human centric way of doing things.
Programming literally should be as easy as talking to the computer and telling it how to shapoe and change the software and it should happen right there in real time on your screen. It should be an iterative, interactive process that anyone can do. It should be more like painting and less like arcane wizardy that can be performed only by the programmer priests who have spent years learning. Even the most advanced software should be capable of being built this way.
In the 1980's I worked for a startup company run by a super smart guy Peter Dunai, who once said "a revolution in programming is coming". I thought I'd seen that revolution already with the Internet, modern programming languages and StackOverflow, but now I realise none of that many anything compared to what the true revolution is going to look like.
AI 2.0 will come and I can't wait.
AI assisted programming is brand new.
It's just the start. So if it gets things wrong, is missing features, annoys people, remain cool, gigantic leaps are coming far beyond your wildest dreams.
And there will be lots of people around right now showing how smart they are by making AI do wrong or silly things. Ignore them - none of these prove that AI is the wrong direction.
AI programming is the single most exciting thing to ever happen to programming - a permanent mentor and pair programmer at your side levels up every programmer in the world.
Even more exciting is innovations that are a direct outcome of AI assisted programming.
When the world wide web first came along, everyone made a website that was like a company brochure. That was Web 1.0 and that's all people had the imagination for. In time, the usages of the web started to arrive that leveraged the core values of the technology, such as interconnection between people. That's where AI programming is now.
We are at AI 1.0
Programming models will come out that are primarily oriented to the strengths of AI programming - but first someone needs to think them up. And they will not only be surprising but absolutely mind blowing. It's entirely possible that in the future the entire internet has an AI sublayer that does all the work of systems talking to systems and people simply don't have anything to do with it. It's possible that the nocode movement is the foundation of the next great innovations in programming. Existing programming models are like the company brochures of web 1.0 - our old, hard, human centric way of doing things.
Programming literally should be as easy as talking to the computer and telling it how to shapoe and change the software and it should happen right there in real time on your screen. It should be an iterative, interactive process that anyone can do. It should be more like painting and less like arcane wizardy that can be performed only by the programmer priests who have spent years learning. Even the most advanced software should be capable of being built this way.
In the 1980's I worked for a startup company run by a super smart guy Peter Dunai, who once said "a revolution in programming is coming". I thought I'd seen that revolution already with the Internet, modern programming languages and StackOverflow, but now I realise none of that many anything compared to what the true revolution is going to look like.
AI 2.0 will come and I can't wait.
>> AI assisted programming is brand new.
The field that studies the automatic creation of computer programs is called "program synthesis". It is a field that straddles multiple sub-fields of computer science, including AI (where research is usually on inductive program synthesis, from examples).
https://en.wikipedia.org/wiki/Program_synthesis
It is also a very old idea, as old as computer science itself- or even older. Here's wikipedia, again:
*During the Summer Institute of Symbolic Logic at Cornell University in 1957, Alonzo Church defined the problem to synthesize a circuit from mathematical requirements.[2] Even though the work only refers to circuits and not programs, the work is considered to be one of the earliest descriptions of program synthesis and some researchers refer to program synthesis as "Church's Problem".
https://en.wikipedia.org/wiki/Program_synthesis#Origin
Even program synthesis from a natural language specification is not brand new. For example, here is the first example of research work on the subject that I could find on duckduckgo in ~1 second:
https://www.microsoft.com/en-us/research/wp-content/uploads/...
The fact that most people have never heard of all this before is not new, either. What is new is that OpenAI and Microsoft are marketing a program synthesis tool.
The field that studies the automatic creation of computer programs is called "program synthesis". It is a field that straddles multiple sub-fields of computer science, including AI (where research is usually on inductive program synthesis, from examples).
https://en.wikipedia.org/wiki/Program_synthesis
It is also a very old idea, as old as computer science itself- or even older. Here's wikipedia, again:
*During the Summer Institute of Symbolic Logic at Cornell University in 1957, Alonzo Church defined the problem to synthesize a circuit from mathematical requirements.[2] Even though the work only refers to circuits and not programs, the work is considered to be one of the earliest descriptions of program synthesis and some researchers refer to program synthesis as "Church's Problem".
https://en.wikipedia.org/wiki/Program_synthesis#Origin
Even program synthesis from a natural language specification is not brand new. For example, here is the first example of research work on the subject that I could find on duckduckgo in ~1 second:
https://www.microsoft.com/en-us/research/wp-content/uploads/...
The fact that most people have never heard of all this before is not new, either. What is new is that OpenAI and Microsoft are marketing a program synthesis tool.
Yes.
> [programming] should be an iterative, interactive process that anyone can do. It should be more like painting...
You mean like taking years of rigurous practice to master, requirering specialized equipment and expensive materials plus an education in color theory, composition and art history :)
You mean like taking years of rigurous practice to master, requirering specialized equipment and expensive materials plus an education in color theory, composition and art history :)
> It should be more like painting
So, something that takes natural talent, years of practice and a lot of time and patience?
At least programming pays well.
So, something that takes natural talent, years of practice and a lot of time and patience?
At least programming pays well.
Nothing you said relaxes me. I like my arcane wizard status.
Me too, but the future is coming to push the druids and the old gods into the shadows.
I already feel the edges of this. I can SSH to a server and fix it, is no longer a skill that's as in demand today, with everything running in a container on a kube pod. It's still useful here and there, but it's no longer the central focus it once was in my toolbox/grimore.
That thread yesterday about two sticks of RAM making the computer faster made me feel like my knowledge is still relevant but those moments are fewer and further between.
That thread yesterday about two sticks of RAM making the computer faster made me feel like my knowledge is still relevant but those moments are fewer and further between.
I think the opposite is true. The people who need to worry are those that know only the most popular technologies, for which documentation and examples are readily available. The people who know the stuff that's not written down anywhere are the ones who are safe.
If all you do is crank out React code, you might want to start learning new things. If you know how to do RCE or are a networking wizard, on the other hand...
If all you do is crank out React code, you might want to start learning new things. If you know how to do RCE or are a networking wizard, on the other hand...
> Programming literally should be as easy as talking to the computer and telling it how to shapoe and change the software and it should happen right there in real time on your screen.
Kind of like what the intention of Smalltalk was, minus the verbal part. You modify a live image like sculpting clay, except now you have an AI generating code for you. I wonder what Alan Kay thinks of this approach.
Kind of like what the intention of Smalltalk was, minus the verbal part. You modify a live image like sculpting clay, except now you have an AI generating code for you. I wonder what Alan Kay thinks of this approach.
AI development will not be using no-code technologies. Text is the best form of compression (of meaning/intent/information) for 99% of software and that isn't going to change.
What AI will accomplish is creating an entire new package ecosystem from scratch, written at a low-level so that it can be extremely performant, while also being highly extensible, abstract, intuitive, interoperable with all languages, etc.
Then the code will be able to take human prompts and translate them into business logic code that uses all those packages. For fuzzy domains with low correctness requirements, the AI will be able to make all the changes to the codebase without verification. For domains with higher correctness requirements, humans will review all the code, or you could have the AI first generate a formal specification based on the prompt, have the humans review that, and then have the AI generate code along with a proof of that code's correctness in conformance to the spec.
What AI will accomplish is creating an entire new package ecosystem from scratch, written at a low-level so that it can be extremely performant, while also being highly extensible, abstract, intuitive, interoperable with all languages, etc.
Then the code will be able to take human prompts and translate them into business logic code that uses all those packages. For fuzzy domains with low correctness requirements, the AI will be able to make all the changes to the codebase without verification. For domains with higher correctness requirements, humans will review all the code, or you could have the AI first generate a formal specification based on the prompt, have the humans review that, and then have the AI generate code along with a proof of that code's correctness in conformance to the spec.
I agree for the most part, but I am also intrigued how this will play out on the regulation side of things.
Hollywood loves to treat AI as something that can go rogue, but these tools will have all kinds of safety locks on it and will only give you answers to things its allowed to. Plus, it will be logged and tracked just like everything else.
You never want to openly allow anyone access to an AI bot that can spit out encryption cracking algorithms with ease. Or worse, something that can solve infrastructure bricking paths that could take down important systems. Or an AI bot that can interfere with real-life things, for example, find someone that doesn't want to be found.
So yea, I cannot wait for this to get better and remove all the boilerplate I have to deal with, and let me focus on the parts of the job I actually love.
Hollywood loves to treat AI as something that can go rogue, but these tools will have all kinds of safety locks on it and will only give you answers to things its allowed to. Plus, it will be logged and tracked just like everything else.
You never want to openly allow anyone access to an AI bot that can spit out encryption cracking algorithms with ease. Or worse, something that can solve infrastructure bricking paths that could take down important systems. Or an AI bot that can interfere with real-life things, for example, find someone that doesn't want to be found.
So yea, I cannot wait for this to get better and remove all the boilerplate I have to deal with, and let me focus on the parts of the job I actually love.
> web started to arrive that leveraged the core values of the technology, such as interconnection between people
A lot of people blame the advanced social media products on the internet for a decline in mental well being and for increasing isolation and lack of human connection in society. That might not be the best example of something we want the AI path to emulate.
A lot of people blame the advanced social media products on the internet for a decline in mental well being and for increasing isolation and lack of human connection in society. That might not be the best example of something we want the AI path to emulate.
How will you know when it's actually working as intended? The issue is /strongly/ epistemological.
I think this is the main thing that sticks out to me. I think Hillel Wayne is right to point out that it moves the main task of a developer from writing code to proofreading it (to a small degree now, perhaps to a larger degree in the future). Most programmers prefer to write their new code than to read someone else's. So if we are delegating our favourite parts to AI, how long before we ask it to write the unit tests as well? Who's going to validate that the unit tests it writes are correct? I forsee a bunch of frustrated QA teams at least in the (near) future being asked to test sloppy code which has passed a bunch of sloppy unit tests, but has had only minimal review from the developer(s) responsible for it. Perhaps I'm overthinking it.
The QA team composed of AIs says it looks OK — ship it!
> AI 2.0 will come and I can't wait.
It might happen in a decade or more.
It might happen in a decade or more.
[deleted]
Hahaha, try asking GPT-3 about electronic engineering.
---
Transcript of IRC channel #electronics:
Mordae: Hi. So I was designing a transformerless LED bulb and I was wondering about the filter capacitor. What happens when the load (one of the LEDs) fail?
Rampant: Hi Mordae. It really depends on the design, but the general idea is that the filter capacitor will help prevent spikes and other issues caused by the failing LED. It can also help with voltage regulation.
---
Actual answer: Light bulb needs a bulky electrolytic cap. Those are usually not 400V tolerant. You are looking at 60V at best at the required sizes. Losing the load will swiftly charge up the capacitor to 325V. But well before that happens, there are going to be fireworks: https://www.youtube.com/watch?v=6WUxgmMDts4
I believe that there are eventually going to be useful mentors for bunch of fields, but right now it's crazy dangerous to blindly follow its recommendations.
---
Transcript of IRC channel #electronics:
Mordae: Hi. So I was designing a transformerless LED bulb and I was wondering about the filter capacitor. What happens when the load (one of the LEDs) fail?
Rampant: Hi Mordae. It really depends on the design, but the general idea is that the filter capacitor will help prevent spikes and other issues caused by the failing LED. It can also help with voltage regulation.
---
Actual answer: Light bulb needs a bulky electrolytic cap. Those are usually not 400V tolerant. You are looking at 60V at best at the required sizes. Losing the load will swiftly charge up the capacitor to 325V. But well before that happens, there are going to be fireworks: https://www.youtube.com/watch?v=6WUxgmMDts4
I believe that there are eventually going to be useful mentors for bunch of fields, but right now it's crazy dangerous to blindly follow its recommendations.
ChatGPT does a much better job. Not perfect, but at least identifies overcharging the capacitor as a danger (though not fireworks) and provides some possible mitigations to look into (NOT take at face value as definitely correct though, to be sure)
---------------------
I'm designing a transformerless LED bulb and I was wondering about the filter capacitor. What happens when the load (one of the LEDs) fail?
---------------------
When a load (such as one of the LEDs in your transformerless LED bulb) fails or becomes open-circuited, the current through the filter capacitor will drop to zero, and the capacitor will start to charge up to the peak value of the AC voltage from the mains supply.
This can be dangerous because the voltage across the capacitor can quickly rise to a high level, depending on the capacitance value and the AC voltage level. If a person were to touch the exposed conductors of the failed LED bulb, they could receive a potentially lethal electric shock from the charged capacitor.
To mitigate this risk, it's important to include some form of overvoltage protection in your design, such as a discharge resistor or a metal oxide varistor (MOV). The discharge resistor will discharge the capacitor when the load is disconnected, while the MOV will clamp the voltage across the capacitor to a safe level by diverting the excess energy to ground.
It's also important to provide adequate insulation between the live components and any exposed conductors or casings to prevent accidental contact with live parts. Additionally, it's recommended to follow established safety standards and guidelines when designing any electrical product to ensure its safety for users
---------------------
I'm designing a transformerless LED bulb and I was wondering about the filter capacitor. What happens when the load (one of the LEDs) fail?
---------------------
When a load (such as one of the LEDs in your transformerless LED bulb) fails or becomes open-circuited, the current through the filter capacitor will drop to zero, and the capacitor will start to charge up to the peak value of the AC voltage from the mains supply.
This can be dangerous because the voltage across the capacitor can quickly rise to a high level, depending on the capacitance value and the AC voltage level. If a person were to touch the exposed conductors of the failed LED bulb, they could receive a potentially lethal electric shock from the charged capacitor.
To mitigate this risk, it's important to include some form of overvoltage protection in your design, such as a discharge resistor or a metal oxide varistor (MOV). The discharge resistor will discharge the capacitor when the load is disconnected, while the MOV will clamp the voltage across the capacitor to a safe level by diverting the excess energy to ground.
It's also important to provide adequate insulation between the live components and any exposed conductors or casings to prevent accidental contact with live parts. Additionally, it's recommended to follow established safety standards and guidelines when designing any electrical product to ensure its safety for users
Out of curiosity I looked up the circuit for that.
Designing a LED power supply as anything else than a current source is asking for trouble.
Designing a LED power supply as anything else than a current source is asking for trouble.
I'm sorry, is there enough context in the original question as presented to come up with your "actual answer"? I mean, I don't know, I'm not an EE, lucky I can make a light switch work. But given the dearth of details in the query, is that AI answer grotesquely wrong? Just seems like generic advice.
> I'm sorry, is there enough context in the original question as presented to come up with your "actual answer"?
I'd say there is. "Transformerless" implies a specific design, especially when you mention there's a filter capacitor.
> is that AI answer grotesquely wrong?
Yes. It's fine to say the filter cap helps with "spikes and other issues", but not ones caused by a failing LED. It took half a true sentence and made it into a false sentence. In this design the filter cap is something you give protection to much more than something that provides protection.
I'd say there is. "Transformerless" implies a specific design, especially when you mention there's a filter capacitor.
> is that AI answer grotesquely wrong?
Yes. It's fine to say the filter cap helps with "spikes and other issues", but not ones caused by a failing LED. It took half a true sentence and made it into a false sentence. In this design the filter cap is something you give protection to much more than something that provides protection.
[deleted]
Anybody worrying that programmers loose jobs stop worrying.
All the automation in the world does not replace the need for people to debug problems.
Even more interestingly, the more something is automated the more time is spent on solving the issues when they happen. Finding a simple mistake in a very complex automation can be a challenging project.
But... if you have been programming Cobol your entire life you might want to start learning some newer technology.
All the automation in the world does not replace the need for people to debug problems.
Even more interestingly, the more something is automated the more time is spent on solving the issues when they happen. Finding a simple mistake in a very complex automation can be a challenging project.
But... if you have been programming Cobol your entire life you might want to start learning some newer technology.
You think the programming AIs are going to be trained on Cobol?
I mean, I have no ability to check it for correctness, but ChatGPT is certainly capable of writing Cobol. Just asked it for an implementation of Dijkstra's algorithm and it looks reasonable enough to me.
Why _wouldn't_ the programming AIs be trained on Cobol?
Why _wouldn't_ the programming AIs be trained on Cobol?
Ok, I would think you probably would not want to devote a lot of resources to training for Cobol given the smaller niche it represents which I figured would make it less useful.
We already write more code than we understand given how lowly the industry regards formal methods and how divided it is even on testing.
It is not surprising to me that Bertrand Meyer could miss that error in the code! A former colleague of mine challenged his grad students to implement an error-free version of binary search. He gave them the specification and verified each one: the majority of the submissions had errors. It’s a marvel that computers and software work so well.
And instead of tackling the problem of programming, as Dijkstra put it, we’re going to automate writing more code. Great.
See you in a few years when we can’t even keep up with the deluge of error prone programs.
Although we have gotten this far with very few proofs and models. Who knows, maybe we’ll just tolerate it?
It would be nice to have code synthesis work on larger examples from deep specifications. And models that understand proof tactics and can speed up the process of formalizing proofs. That would be nice.