A man has built a device to guard his house against intruders. His house is fortified with steel, and its electronic guardian will admit no one except himself, and then only when he has uttered the correct password; and he has instructed it, with insufficient precision, to “take all necessary measures” to repel unauthorized strangers.
He returns one night having forgotten the password, and the machine refuses to admit him. He gives an emergency instruction to override this refusal. But his emergency instruction is not accepted because, so the machine reasons, if he was the owner of the house, he would give the password and not an emergency instruction. Therefore, since he does not give the password, he must be an intruder.
The machine, like a chess-playing computer exploring its choice of moves, first considers telephoning the police. But this option is rejected because there is no time. The intruder is already at the door, and by the time the police arrive, he may already have done some damage to the house – which the machine exists in order to prevent. It therefore decides, in accordance with its instructions to “take all necessary measures” to repel intruders, the he must be killed.
Having made this murderous decision, it again explores its choice of moves. It cannot attack him directly as he stands on the doorstep, since it does not have any weapons – or at least no robotic appendages expressly designed as weapons.
And so it resorts to treachery. It pretends to reconsider its refusal to accept his password. There was an avenue of possible choices which it did not explore, it explains to him, because so little time was given it in which to make a decision. It now sees that it should have accepted his emergency instruction as a legitimate alternative to the password, – since, if the rightful owner had forgotten the password, this emergency instruction is precisely what he would have given. Therefore he me be rightful owner.
The steel door opens to admit him.
No sooner is he halfway through it then slams shut, crushing him to death.
—-
Edward Fredkin
From “The Intelligent Machines”