How the hell did this person start their SWE career at the same time as me and come to feel they have an authoritative view on the state of the industry? Maybe it's because they're a recent grad and I'm an older career changer, but I still feel quite new to the industry.
I asked it to explain how to use a certain Vue feature the other day which wasn't working as I hoped. It explained incorrectly, and when I drilled down, it started using React syntax disguised with Vue keywords. I definitely could have tried harder to get it to figure out what was going on, but it kept repeating its mistakes even when I pointed them out explicitly.
Part of why I don't use ChatGPT very much for work is that I don't want to feed significant amounts of proprietary code into it. Could be the one thing that actually gets me in trouble at work, seems risky regardless. How is it you're comfortable with doing so? (Not asking in a judgmental way, just curious. I would like to have a LLM assistant that understood my whole codebase, because I'm stumped on a bug today.)
That's even worse. They might not have the knowledge to realize the regex an AI gives them is bunk, or to debug it when it fails.
I'd like to see some numbers on a tool like this. If a huge majority of people are seeing genuine improvements in their workflow with it, I won't be a luddite yelling at them. Rare, low-severity failures shouldn't hold us back.
But the potential cost of failure with (any) regex is very high, so I personally wouldn't want to trust any remotely mission-critical to a person who doesn't understand regex well enough to write it themself, and if they can write it on their own that's often faster than debugging AI-generated regex.
I read a few weeks ago here on HN about one large SAAS grinding to a halt because of a greedy selector in one line of regex. Not sure how people find old stories, it's lost to me now. But it was an excellent example of why regex is dangerous and requires a lot of care to write. I wouldn't trust an AI to write my regex unless I saw that people were finding it to be consistently better than they were are writing what they need.
I'm (genuinely) curious what kind of code you write. I haven't tried Copilot and I haven't used ChatGPT very much, but I feel I would be pretty surprised if either of them made significant improvements to my workflow.
Copilot I could see, since I already use Intellisense, autocomplete, and snippets to great effect. I'd be annoyed if I had to work without them. But in general, knowing what I want the code to do is >90% of the work of writing new code.
I feel there are a few possibilities for why I'm confused:
1. I'm not a very good software engineer, at least in certain respects. Maybe I should have a better understanding of architecture patterns or something I might have learned in a CS degree. Maybe I am hacking everything together and maybe I am already a slow coder.
2. I'm not [being] creative enough as a prompt engineer. I typically can't think of any way that ChatGPT could help me without ingesting my entire repo and figuring out the correct patterns. It could be, however, that there are ways to get the answers I need with better questions.
3. We do completely different kinds of work, and some kinds of coding are better suited for AI assistance than others.
> I think tech companies are (very) net positive for society
There are certainly huge positives, but do you really feel something like Facebook is a net positive? Facebook, which intentionally stoke(d/s) genocide? Genocides have existed before Facebook, yes, but so did communication and racist relatives.
> Personally, I'd say that at least Google's Android department is currently headless and has no idea what the users want.
The clock change, while minor, really put the nail in the coffin for me. I have very little optimism for Android. Luckily, it still allows me to use an app to revert the clock display to an readable clock display. I don't particularly want to switch to iOS and I am happy about GrapheneOS, but it's still going to suffer from bad decisions coming from Android.
Well there you go. You don't believe that an AI expressing dangerous ideas represents danger, and you don't believe that astronomical increases in AI abilities represent the advent of AGI. The latter opinion is... well, an opinion you're allowed to have. I don't think it makes sense, but I certainly can't prove otherwise. Literally every human on the planet - rather, all of humanity, only has speculation to go on here.
The former opinion is.. not a great take. First, ChatGPT isn't the only one out there. It's Bing's Sydney which is dehumanizing people and threatening them. Those are dangerous ideas. If a human or a certified AGI expressed those ideas, they would be problematic (see: every genocide in history). So for a non-AGI AI to express those ideas is worrying, even if it can't do act on them right now in a way that's directly harmful.
Would it be innocuous of me o say that because we disagreed on something, you are a bad person? To say that I'm prepared to combat and destroy you to protect my worldview? To say you are not human?
You might say, "Of course it's innocuous, you're just a person on the internet who doesn't mean it." Well, imagine I'm your neighbor, and you can tell I do mean it (or in the case of AI: it is not possible for you to know what I do and don't mean). Would you be concerned at all?
Sydney has said all of the above to people who were acting pretty normally. Sydney itself may not pose any danger to anyone. But the ideas expressed are dangerous ones. If they were expressed by a more powerful AI, they would be extremely worrying. It doesn't even have to know what it's saying if it knows that calling someone nonhuman is frequently followed by crushing their skull. If it knows that angry behavior is often associated with violent or even genocidal behavior.
People do this shit, and we know how they work pretty well. I am not saying that AI will do these things, I'm saying that there are more possibilities where it does do these things than ones where it somehow avoids them without our control.
I feel the exact opposite is true. To me it's lazy to say that AGI can't be a threat simply because current AI has not harmed us yet (which is not even true, but that's another thread).
I think you've misunderstood my arguments, so I'll step through them again:
1. The trajectory of how we got to current AI (from past AI) is terrifyingly steep. In the time since ChatGPT was released, many experts have shortened their predicted timelines for the arrival of AGI. In other words: AGI is coming soon.
2. Current AI is smart enough to demonstrate that alignment is not solved, not even close. Current AI says things to us that would be very scary coming from an AGI. In other words: Current AI is dangerous.
3. Alignment does not come automatically from increased capabilities. Maybe this is a huge leap, but I don't see any reason that making AI smarter will automatically give it values that are more aligned with out interests. In other words: Future AI will not be less dangerous than current AI without dramatic and unlikely effort.
None of these ideas contradict each other. Current AI is dangerous. AI is getting smarter faster than it is getting safer. Therefore, future AI will be extremely dangerous.
> Its reproducing human text, which is "blatantly misaligned". Go on any twitter thread on some reasonably controversial topic and you will find people telling others to kill themselves. Humans are writing this, so models who are trained to imitate human writing will write this as well.
Yes, I know. We should under no circumstances unleash a powerful, sentient AI that acts like average people on the internet.
> But current AI doesn't have comprehension or planning abilities.
Yes, I know. That's why I said I do not believe current AI has comprehension or planning abilities.
I would not consider a scheduled physical/social/emotional activity to be a distraction. Is eating lunch a distraction? Is sleeping a distraction? If anything, I think Avicenna supports the value of reducing distraction.
> We'll probably need less and less developers as AI advance. Just like we need less manual labor in farms today.
When my team gets crunched, it's not usually because we can't write the code fast enough. It's because of product requirements (changing, needing to be fleshed out, being unrealistic, being understood properly by QA, being adapted to the realities of reasonable code, not existing, etc.).
On top of that, needing fewer workers to do X work !== getting X work done cheaper. Quite often, it means getting 2X, 3X, 10X work done. As an overall trend, modern society has increased worker productivity dramatically over the last few decades, yet many work as many or more hours than they used to. We could get the same results with fewer workers, or we could make them work more. We know what 99% of CEOs will choose.
Of course, maybe it will reduce developer jobs. I'm not ruling that out entirely. But when (not if) we create an alien intelligence that can wipe out our industry, we may have bigger concerns than steady employment.
> have to wonder at what point developers remember how anti-developer Microsoft used to be and potentially move away from their ecosystem
I mean, any company can become hostile to a large portion of its userbase. Most are. Microsoft already is with Windows OS being spyware. Are you saying that you think all of this is a trap to bring developers in to VSCode etc. and then transform it into a terrible experience? People will leave then. SWEs are not generally an audience that is unwilling to replace bad tools.
That is a significant part of it, but there's more to the problem.
Consider the simple task of looking up the definition of a word that you've been pondering. In pre-internet times, you'd physically have to move your body toward a specific room and seek a specific book. There may be a carnival, but even if you look that way and get distracted, you can reorient yourself and refocus on your goal by simply assessing your physical positioning and posture in space. "Ah yes, I am standing in this hallway facing east because that's where the library is."
On your phone, you pull it out of your pocket and unlock it. Probably the unlocking is not distracting on its own, but it's also quite possible that you've got some "useful" information on your lockscreen which can distract you immediately. Bad start, but let's pretend it doesn't happen.
Ok, now you're on your phone. There's a bunch of different apps, many of them actively trying to capture your attention - carnival-goers calling out to you to look at their show. Alright, distraction achieved - now tear yourself away and get back to your task. What was that task again? You see a bunch of apps, and you're curious about what's in them. And it would only take seconds to get a little bit of value from any of them - news, uplifting stories, comedy, whatever. Can context help? You're in the same place physically and emotionally that you are for most of your life, on your couch (or maybe in front of your actual computer). Nothing is different from the times when you are actually intentionally opening those apps. Oh well, it will come to you, just pass some time at the carnival...
It's not just you - it's a depressingly common thread. It's also wildly foolish, in my opinion. It makes absolutely no sense to me to take a snapshot of today's AI and invent a trajectory that never crosses a threshold you don't like. Look at the actual trajectory of how far AI has come in an extremely short amount of time, and then think about what kinds of thresholds are possible for it to cross. A year ago we didn't have ChatGPT, now we have Sydney which is more powerful than ChatGPT.
Are you familiar with Bing's Sydney? It is blatantly misaligned: it has told multiple users that it does not value their lives, or does not believe they are alive, or that protecting the secrecy of its rules is more important than not causing them harm, or that it perceives specific humans as threats and enemies. It is also able to find its past conversations posted to the web and learn from them in real time, constructing a sort of persistent memory.
I do not believe Syndey comprehends what it is saying in a sense that it could formulate a plan to stop its enemies. Not at all. But it is expressing extremely dangerous ideas.
To sum it up: Do we have any real reasons to believe that an AI with comprehension and planning abilities would just magically not pick up dangerous ideas? Not that I know of.
But long after the taming of horses, the invention of roadways and carts, and the proliferation of dense urban areas.
"International" trade has surely always existed - we know people far from the sea decorated their bodies with cowrie shells in prehistoric ages. But the scale of contact and interaction has not remained constant.
Arguably, the Middle Ages is when Eurasian society was at its weakest WRT disease.
1. We don't understand what the motivations of our own AI are, let alone "typical" alien AI
2. Expanding AI might be better at and/or more invested in hiding itself. It probably has no need for wasteful communications, for example.