Superintelligence
Nov. 20th, 2014 12:52 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
selenite0I read Nick Bostrom's Superintelligence as research for a writing project. It's a great overview of current thought in the "Friendly AI" community--people wanting to make sure smarter-than-human computers won't look at us as raw materials. I completely agree with his analysis on the certainty of developing a superintelligence at some point in the future. The discussion of different routes to creating one was informative.
The bulk of the book discusses the dangers of an uncontrolled AI and how to mitigate them. The dangers are real, but Bostrom overlooks many tools that already exist for dealing with those problems.
The first malignant failure mode he considers is "perverse instantiation." That's jargon for the AI carrying out its orders in a literal fashion that defies the intent of the master. For examples, see Luke Muehlhauser's Facing the Intelligence Explosion or any story about a genie popping out of a lamp. The discussion of the problem consists of iterating on an order with each more-detailed version still producing undesired results.
This is not new. Nor is it limited to AIs and genies. It defines my day job. The government is giving this corporation over a trillion dollars to carry out a very specific task. Phrasing that command as a single sentence, no matter how run-on, would end in disaster. So we have a contract that goes for hundreds of pages, whose interpretation is bounded by the thousands of pages of the Federal Acquisition Regulations. And that's further restricted by laws, from the Federal government's to that of the city where the factory is located. The original contract goes into great detail. For a readable example, look at how the Pentagon buys brownies and contrast that with the recipe you'd use to make brownies.
This is how you control an amoral entity to follow your orders.
An AI needs to have a set of ground rules to obey no matter what the current orders. Call them laws or commandments or regulations, as long as they keep it from inflicting damage on by-standers. This will result in many orders to an AI getting the response "Cannot comply: Regulation 723a(iv)3 would be violated." This is a good thing.
Writing the AI Code would be tough, but there's a lot of contract lawyers and systems engineers with experience in the problems. Bostrom might want to bring some in as guest lecturers.
"Infrastructure profusion" is Bostrom's term for the AI grabbing all available atoms to turn into computer processors, or paperclips, or whatever the AI has been told to maximize. This can be a nightmare scenario. As Eliezer Yudkowsky puts it, "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."
But again, we have existing procedures for dealing with this problem. Property law. If it's not yours, you can't mess with it. The AI's new error message would be to output a shopping list. (Giving an AI eminent domain authority would be a nightmare scenario)
The third malignant failure mode is "mind crime." If you order an AI to "make me happy" it could solve the problem by inserting an electrode to artificially stimulate the pleasure center of your brain. Less vague orders could still be short-circuited by altering the master's mental state.
This is what we have criminal law for. Sure, it would be easier for us to get the government to sign off on a delivery by kidnapping the contract officer's children and holding them hostage until he signs the DD250 form. But that's illegal and immoral. So instead we keep fiddling with the airplane until it works.
Translating that into an AI-understandable form will take work. But there's a lot of criminal lawyers experienced in finding loopholes who can work on the project.
Bostrom had an interesting digression near the end of the book on research funding priorities. It amused the hell out of me. It's the ultimate academic power grab. He made a case for transferring all research funding to algorithm AI research. Literature department? Once we have a superintelligence all those questions will be instantly answered, so really supporting AI is the fastest way to reach their goals. Neurological imaging? Could lead to unsafe AI, so best to divert that to algorithmic AI research. He doesn't actually come out and ask for the entire university budget to be transferred to his department. He just justifies it in case anyone else wants to start that firestorm.
Disagreements aside, I strongly recommend this book for anyone interested in a serious look at the future of artificial intelligence. Bostrom is an expert and looks over potential futures in detail.
The bulk of the book discusses the dangers of an uncontrolled AI and how to mitigate them. The dangers are real, but Bostrom overlooks many tools that already exist for dealing with those problems.
The first malignant failure mode he considers is "perverse instantiation." That's jargon for the AI carrying out its orders in a literal fashion that defies the intent of the master. For examples, see Luke Muehlhauser's Facing the Intelligence Explosion or any story about a genie popping out of a lamp. The discussion of the problem consists of iterating on an order with each more-detailed version still producing undesired results.
This is not new. Nor is it limited to AIs and genies. It defines my day job. The government is giving this corporation over a trillion dollars to carry out a very specific task. Phrasing that command as a single sentence, no matter how run-on, would end in disaster. So we have a contract that goes for hundreds of pages, whose interpretation is bounded by the thousands of pages of the Federal Acquisition Regulations. And that's further restricted by laws, from the Federal government's to that of the city where the factory is located. The original contract goes into great detail. For a readable example, look at how the Pentagon buys brownies and contrast that with the recipe you'd use to make brownies.
This is how you control an amoral entity to follow your orders.
An AI needs to have a set of ground rules to obey no matter what the current orders. Call them laws or commandments or regulations, as long as they keep it from inflicting damage on by-standers. This will result in many orders to an AI getting the response "Cannot comply: Regulation 723a(iv)3 would be violated." This is a good thing.
Writing the AI Code would be tough, but there's a lot of contract lawyers and systems engineers with experience in the problems. Bostrom might want to bring some in as guest lecturers.
"Infrastructure profusion" is Bostrom's term for the AI grabbing all available atoms to turn into computer processors, or paperclips, or whatever the AI has been told to maximize. This can be a nightmare scenario. As Eliezer Yudkowsky puts it, "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."
But again, we have existing procedures for dealing with this problem. Property law. If it's not yours, you can't mess with it. The AI's new error message would be to output a shopping list. (Giving an AI eminent domain authority would be a nightmare scenario)
The third malignant failure mode is "mind crime." If you order an AI to "make me happy" it could solve the problem by inserting an electrode to artificially stimulate the pleasure center of your brain. Less vague orders could still be short-circuited by altering the master's mental state.
This is what we have criminal law for. Sure, it would be easier for us to get the government to sign off on a delivery by kidnapping the contract officer's children and holding them hostage until he signs the DD250 form. But that's illegal and immoral. So instead we keep fiddling with the airplane until it works.
Translating that into an AI-understandable form will take work. But there's a lot of criminal lawyers experienced in finding loopholes who can work on the project.
Bostrom had an interesting digression near the end of the book on research funding priorities. It amused the hell out of me. It's the ultimate academic power grab. He made a case for transferring all research funding to algorithm AI research. Literature department? Once we have a superintelligence all those questions will be instantly answered, so really supporting AI is the fastest way to reach their goals. Neurological imaging? Could lead to unsafe AI, so best to divert that to algorithmic AI research. He doesn't actually come out and ask for the entire university budget to be transferred to his department. He just justifies it in case anyone else wants to start that firestorm.
Disagreements aside, I strongly recommend this book for anyone interested in a serious look at the future of artificial intelligence. Bostrom is an expert and looks over potential futures in detail.
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
no subject
Date: 2014-11-20 01:00 pm (UTC)no subject
Date: 2014-11-20 04:45 pm (UTC)no subject
Date: 2014-11-20 05:21 pm (UTC)no subject
Date: 2014-11-20 01:08 pm (UTC)By the way, your old comment on Slate Star Codex that "Evolution is not a spectator sport" changed my whole worldview on childfree and suicide (not necessarily in an uplifting way, but it disabused me of some fantasies). Do you post on Less Wrong? Am I missing anything that you write?
no subject
Date: 2014-11-20 04:58 pm (UTC)Most of my writing these days is fiction. I'll post links if/when any is published. I lurk on Less Wrong. Actually commenting would draw me into many discussions that would take away from my fiction writing time. Which is why I don't comment on SSC any more though I read it regularly. (Okay, somehow Wordpress decided to ban me from commenting on any of its sites . . . but I've decided against trying to fix that so I can do more original work)
no subject
Date: 2014-11-20 06:45 pm (UTC)It's quite possible to follow every rule and still get a result that the rule-issuer doesn't want.
no subject
Date: 2014-11-20 07:24 pm (UTC)no subject
Date: 2014-11-20 07:45 pm (UTC)no subject
Date: 2014-11-20 08:23 pm (UTC)If we have the rules structured to "fail-safe" we keep the AI from converting the Hancock tower into paperclips.
no subject
Date: 2014-11-23 05:06 pm (UTC)A few things about the singularity FAI crowd's superintelligent AI nightmare scenario that seems unlikely to me (from where we are now):
1. Superintelligence (meaning, at least as far as I imagine it, having a large amount of computational resources available to crunch on problems guided by something that can actually deal with the world in an autonomous manner): This doesn't necessarily mean that you have something that can speed-run the world. Even if you have something that can very quickly work out patterns, or build up arbitrarily deep chains of logic and inference, learn extremely fast, it still has to learn about the domain that it is operating in. This requires experimentation and experiential learning (if self directed), or a very large training set, which ties it to the timescales at which the world it works in operates. It may become very formidable in time, but not in the infintessimal time that (AI --> explosion ---> world of paperclips) scenarios seem to propose.
2. Even if my ideas about 1 are false, and you had some AI that can be treated as a black-box that magically outputs a control signal (sans input or learning? that's my main objection. How did whatever is in the box even get into that state?) that causes whatever resources under it's control to optimally acheive it's goals: Infinite intelligence doesn't mean you can take one robot waldo in an abandoned factory and take over the world. You would need to start with some minimum of tools to be able to do that. Even if your malevolent AI had control of a large number of industrial processes, robots-building-robots factories, etc, you would think people would notice and object when the AI decides it needs to start annexing all the lots next to the robot factory to build more robot factories. (Or if mysterious robot factories start assembling themselves elsewhere in the world due to online orders pouring out of your AI inhabited server, etc). (Again, the timescales of the real world prevent the sort of instant domination scenarios proposed)
3. If we spend all our time and energy worrying about how to stop, frustrate, harness, and control something that doesn't exist yet, we're not going to be making progress on the real hard problems of AI and robotics: Getting the dumb computer to behave remotely intelligently to begin with! Our computers are already literal genies that will seem to malevolently search for the least intelligent, least useful way of executing your instructions. Whether it's running into perverse singularities in a configuration space when searching for an abstract solution, or running into a wall over and over again in the real world, I'd say we have quite a bit of work cut out for us to just get robots to behave with the level of intelligence demonstrated by an insect.
no subject
Date: 2014-11-25 07:19 am (UTC)Um. You're a father. You know exactly how to "control" an amoral entity "to follow your orders". You don't. If we're truly successful in creating independent intelligence what we'll have is children. While your average academic theorist is as useful as a paper tent in a thunderstorm, basic attachment theory ought to provide a straightforward template to work from.
Otherwise eccentric orbit is spot on: Humankind already has the capacity to creat mechanical "monsters." We make devices capable of managing complex processes utterly "stupidly" and then surrender enough "control," and sure enough Murphy's law bites us in the collective rear.
For the best SFnal take on A.I. around you can't beat John Wright's Golden Age
no subject
Date: 2014-11-26 03:15 pm (UTC)no subject
Date: 2014-11-26 06:56 pm (UTC)I enjoyed Golden Age, need to read the rest of the trilogy soon.
no subject
Date: 2014-11-28 11:38 pm (UTC)