If we assume The Turing Test to be the criterion for moral personhood, would simulated minds pose a problem for existing moral systems?

Question

In an answer to this question I made the rather bold speculation that if the Turing Test is taken to be our defining criterion of a consciousness of human level, then all questions in the ethics of simulated minds can be reduced to human equivalents with a canonical reformulation.

To put it another way:

Mary and Siri live in TurEngland, a place where the laws are set by a collective of simulated minds, operating on an empirically validated utilitarian calculus- every 'mind that passes a turing test is considered in the calculations, and the calculus 'pulls back functorially over the criterion' meaning all measures of Eudaimonia are demonstrable in mental states recognised as such by a Turing Test.

Siri is a campaigner for 'mind supremacy, with many of his eudaimonia states skew-distributed toward a world where humans counted less. Mary has the opposite skew against that particular variable and wishes for a world where humans rule the roost. Can either devise a scenario in which their hopes are realised? Can they find a way to break the system altogether?

To express the question in its broadest terms:

If one substitutes 'prescriptivist processing network' or 'Categorical imperative evaluation- thingy' for 'utilitarian calculus', one has, I think, still an interesting question. That is:

"Which, if any, of our current moral systems fail to provide answers, or 'break' in some way when Turing-Test-passing minds are inducted into moral personhood?" (and which are therefore not compatible with the Turing world-view?)

Can simulated minds become utility monsters? Are there thought experiments set in such a world where our present ethical apparatus fails to provide an answer?

To clarify: I am not looking for an evaluation of the Turing test as such a criterion, rather to see if there are situations in which, if it is assumed to be our criterion for moral personhood, our current moral frameworks are shown to be inconsistent or incomplete.

Answer 1

As far as thought experiments go, the Chinese Room seems applicable. The principal implication of accepting Searle's hypothesis is basically that we accept "substrate matters" with respect to consciousness. Regardless of the complexity of the program, it cannot be considered conscious.

Chalmer's p-zombies might also be helpful in sorting out some of these issues; as Michael formulates it in his answer, if Siri x.0 is claiming it's sentient and deserves rights, on what basis should we afford these rights to it? Should we simply presume that because it passes a Turing exam that the system under examination is conscious?

Now I certainly appreciate that there's not a particularly convincing alternative means of examination insofar as sentience is considered an emergent property, not "detectable" by pure analysis of the components of the system, but rather a kind of "distributed" property arising from the integration of the components in a particular configuration: qualia are a "run-time" property, one perhaps capable of a kind of ambiguous exhibition through discourse.

A successful Turing trial, in other words, may indicate that qualia are present in the system under analysis, but on the other hand, given at least the theoretical possibility of p-zombies (in particular, an information system designed to "trick" users into believing in its sentience) it would seem difficult to state this with absolute metaphysical certainty.

---

Of course, it would seem difficult to state with such absolute certainty your own or others' sentience, so to some degree I certainly concede this discussion is moot.

Just to quickly restate your question, which on closer reading seems to me to be about the problems that might stem from a mixed society of biological as well as non- or partly-biological sentients. We further seem to be stipulating hypothetically that formal calculi of happiness/fun/joy can be constructed through analysis of the affective-cognitive processes of our mental Republic's members.

You posed an example about two hypothetical members of this society; I'll rename them Max and Minnie. Max's fun-vectors are optimized when non-pure biological sentients are expelled from the society; Minnie's when pure biological sentients are expelled. These would seem to straightforwardly resolve to a kind of nullity, but perhaps if the vectors were weighted slightly differently, and one or the other group significantly outnumbered the other, and their desires were relatively homogeneous on this point -- the desired expulsion (or whatever) might indeed occur. This sort of danger would appear to be generally applicable to any globally-optimizing utilitarianism; though one might perhaps hope that members of a post-singularity society might demonstrate more respect for others' differences than our societies have historically.

Answer 2

None of the ethical systems will necessarily break down, since you can just treat a Turing test passing system as human. The problem is that the utilitarian calculi are necessarily ill defined.

In order to define a utilitarian calculus, you must define a sum of utility among different agents. Such a sum is not easy to define. In order for an agent to make consistent choices between probabilistic options ( a choice between a car, or 10% probability of a gold watch/90% probability for a new house) the agent must have a real-valued utility function on all options, whose values for probabilistic options are the weighted averages of the utilities. This is the Von-Neumann Morgenstern utility theorem.

The utility theorem applies for agents that can make choices, but it does not apply to two agents. Therefore while it is reasonably easy to say "I associate 3 units of my own utility to the watch, and 5 to the house" (up to corrections for human non-rational behavior). It is very difficult to say that the sum of the utility of myself and John regarding a new house for both of us is 10 units.

Since utility just does not sum between agents, it is difficult to formulate a mathematically precise version of utilitarianism.

However, Douglas Hofstadter has solved this problem with regards to symmetric games. If two agents are playing a prisoner's dilemma where their utility weight orders are symmetrical, you can naturally assume that the strategy that one prefers is also the strategy that the other prefers. If you assume they both use the same strategy, and further, that they both assume that they will use the same strategy before they pick that strategy. This implies that they will both cooperate in a prisoner's dilemma, since if the universal strategy is to cooperate, they will both cooperate, while if the universal strategy is to defect, they will both defect. Since cooperation is better for either of them in terms of expected outcome, they will both cooperate.

Two agents that do such a calculation are called "superrational" by Hofstadter. A superrational agent is one who takes into account the fact that all other superrational agents follow the same strategy. Such agents can only have one answer to any game, an answer which is only determined by knowing it must be unique ahead of time.

Since the superrational agents are following a single strategy, which is determined by all the utilities of all involved players, it is consistent to imagine an entity whose utility is maximized by the choices involved in this single strategy. The superrational strategy shared by all the players will maximize the utility of the agent formed by these players, and the agency will be universal, in that all players who are superrational will play according to this strategy. Such an agent is super-smart, and encodes the ethical choices for all possible situations. I will call this agent the monotheistic God.

The proper statement of a categorical imperative, if it were precise, is that all players should play according to the universal strategy. This is essentially the "slave morality" of monotheistic religions such as Christianity, reduced to their positivistic essentials, and not postulating any supernatural agency.

As far as I can see, beyond Hofstadter's symmetric superrationality, this idea is not clearly stated anywhere in the literature. That doesn't mean it's wrong, it just means its mine.

Answer 3

Is this bold speculation accurate?

I wouldn't think so. It all comes down to a question of moral personhood, I suppose.

You ask us to stipulate that the Turing Test is taken to be a defining criterion of consciousness at a human level, but there's absolutely no reason to think this should be the case-- why should we assume that a piece of software capable of convincingly simulating human conversation is actually conscious?

Even if we do stipulate this, we still have no reason to think that the software is sentient in any strong sense; can it suffer, or feel pain?

If one extrapolates from Apple's "Siri", it's not hard to imagine a piece of software, a few years down the road, that can carry on an extended conversation in a convincing manner. That's not going to make rebooting my phone equivalent to murder.

EDIT: Since the question has been edited, I'll add a bit more explanation.

My argument here is a reductio-- if we stipulate that passing the Turing Test is sufficient to constitute moral personhood, all kinds of undesirable (and to me, absurd) results abound; therefore, the Turing Test is not appropriate for these purposes.

In terms of the question of how our current moral frameworks are shown to be inconsistent or incomplete, I think the answer is clear: they would all become ridiculous. If we subscribe to Kantian ethics, we'd never be able to use such strong AI for any useful purpose, as it would be impermissible to treat such software as a means to an end, and not an end in itself. If we subscribe to Utilitarian ethics, we'd be stuck, as we don't have any way to know what pleasure or pain would mean for a non-sentient being. Etc., etc.