Your scenario 1 is the actual danger. It’s not that AI will outsmart us and kill us. It’s that AI will trick us into trusting them with more responsibility than the AI can responsibly handle, to disastrous results.
It could be small scale, low stakes stuff, like an AI designing a menu that humans blindly cook. Or it could be higher stakes stuff that actually does things like affect election results, crashes financial markets, causes a military to target the wrong house, etc. The danger has always been that humans will act on the information provided by a malfunctioning AI, not that AI and technology will be a closed loop with no humans involved.
Yeah, the short-term outlook doesn’t look too dangerous right now. LLMs can do a lot of things we thought wouldn’t happen for a long time, but they still have major issues and are running out of easy scalability.
That being said, there’s a lot of different training schemes or integrations with classical algorithms that could be tried. ChatGPT knows a scary amount of stuff (inb4 Chinese room), it just doesn’t have any incentive to use it except to mimic human-generated text. I’m not saying it’s going to happen, but I think it’s premature to write off the possibility of an AI with complex planning capabilities in the next decade or so.
I have this debate so often, I’m going to try something a bit different. Why don’t we start by laying down how LLMs do work. If you had to explain as fully as you could the algorithm we’re talking about, how would you do it?
Yeah, sorry, I don’t want to invert burden of proof - or at least, I don’t want to ask anything unreasonable of you.
Okay, let’s talk just about the performance we measure - it wasn’t clear to me that’s what you mean from what you wrote. Natural language is inherently imprecise, so no bitterness intended, but in particular that’s how I read the section outside of the spoiler tag.
By some measures, it can do quite a bit of novel logic. I recall it drawing a unicorn using text commends in one published test, for example, which correctly had a horn, body and four legs. That requires combining concepts in a way that almost certainly isn’t directly in the training data, so it’s fair to say it’s not a mere search engine. Then again, sometimes it just doesn’t do what it’s asked, for example when adding two numbers - it will give a plausible looking result, but that’s all.
So, we have a blackbox, and we’re trying to decide if it could become an existential threat. Do we agree a computer just as smart as us probably would be? If so, that reduces to whether the blackbox could be just as smart as us eventually. Up until now, there’s been great reasons to say no, even about blackbox software. I know clippy could never have done it, because there’s forms of reasoning classical algorithms just couldn’t do, despite great effort - it doesn’t matter if clippy is closed source, because it was a classical algorithm.
On the other hand, what neural nets can’t do is a total unknown. GPT-n won’t add numbers directly, but it is able to correctly preform the steps, which you can show by putting it in a chain-of-thought framework. It just “chooses” not to, because that’s not how it was trained. GPT-n can’t organise a faction that threatens human autonomy, but we don’t know if that’s because it doesn’t know the steps, or because of the lack of memory and cost function to make it do that.
It’s a blackbox, there’s no known limits on what it could do, and it’s certain to be improved on quickly at least in some way. For this reason, I think it might become an existential threat, in some future iteration.
deleted by creator
Your scenario 1 is the actual danger. It’s not that AI will outsmart us and kill us. It’s that AI will trick us into trusting them with more responsibility than the AI can responsibly handle, to disastrous results.
It could be small scale, low stakes stuff, like an AI designing a menu that humans blindly cook. Or it could be higher stakes stuff that actually does things like affect election results, crashes financial markets, causes a military to target the wrong house, etc. The danger has always been that humans will act on the information provided by a malfunctioning AI, not that AI and technology will be a closed loop with no humans involved.
deleted by creator
Not yet, anyway.
deleted by creator
Yeah, the short-term outlook doesn’t look too dangerous right now. LLMs can do a lot of things we thought wouldn’t happen for a long time, but they still have major issues and are running out of easy scalability.
That being said, there’s a lot of different training schemes or integrations with classical algorithms that could be tried. ChatGPT knows a scary amount of stuff (inb4 Chinese room), it just doesn’t have any incentive to use it except to mimic human-generated text. I’m not saying it’s going to happen, but I think it’s premature to write off the possibility of an AI with complex planning capabilities in the next decade or so.
deleted by creator
Chinese room, called it. Just with a dog instead.
I have this debate so often, I’m going to try something a bit different. Why don’t we start by laying down how LLMs do work. If you had to explain as fully as you could the algorithm we’re talking about, how would you do it?
deleted by creator
Yeah, sorry, I don’t want to invert burden of proof - or at least, I don’t want to ask anything unreasonable of you.
Okay, let’s talk just about the performance we measure - it wasn’t clear to me that’s what you mean from what you wrote. Natural language is inherently imprecise, so no bitterness intended, but in particular that’s how I read the section outside of the spoiler tag.
By some measures, it can do quite a bit of novel logic. I recall it drawing a unicorn using text commends in one published test, for example, which correctly had a horn, body and four legs. That requires combining concepts in a way that almost certainly isn’t directly in the training data, so it’s fair to say it’s not a mere search engine. Then again, sometimes it just doesn’t do what it’s asked, for example when adding two numbers - it will give a plausible looking result, but that’s all.
So, we have a blackbox, and we’re trying to decide if it could become an existential threat. Do we agree a computer just as smart as us probably would be? If so, that reduces to whether the blackbox could be just as smart as us eventually. Up until now, there’s been great reasons to say no, even about blackbox software. I know clippy could never have done it, because there’s forms of reasoning classical algorithms just couldn’t do, despite great effort - it doesn’t matter if clippy is closed source, because it was a classical algorithm.
On the other hand, what neural nets can’t do is a total unknown. GPT-n won’t add numbers directly, but it is able to correctly preform the steps, which you can show by putting it in a chain-of-thought framework. It just “chooses” not to, because that’s not how it was trained. GPT-n can’t organise a faction that threatens human autonomy, but we don’t know if that’s because it doesn’t know the steps, or because of the lack of memory and cost function to make it do that.
It’s a blackbox, there’s no known limits on what it could do, and it’s certain to be improved on quickly at least in some way. For this reason, I think it might become an existential threat, in some future iteration.
deleted by creator