Not Another Look at The Not Another Look at The Turing Test! Turing Test!

Kevin WarwickKevin Warwick

22.1.201222.1.2012

Alan TuringAlan Turing

• 1912-19541912-1954

• Ratio ClubRatio Club

Turing’s predictionTuring’s prediction

• “ “I believe that in about fifty years’ I believe that in about fifty years’ time it will be possible to programme time it will be possible to programme computers … to make them play the computers … to make them play the imitation game so well that an imitation game so well that an average interrogator will not have average interrogator will not have more than 70% chance of making the more than 70% chance of making the right identification after five minutes right identification after five minutes of questioning” (Turing, 1950). of questioning” (Turing, 1950).

Turing’s Imitation GameTuring’s Imitation Game

• At one terminal is a human at At one terminal is a human at another is a machine/computer – at a another is a machine/computer – at a third terminal is an interrogator. third terminal is an interrogator.

• ““The interrogator stays in a room The interrogator stays in a room apart front the other two. The object apart front the other two. The object of the game for the interrogator is to of the game for the interrogator is to determine which of the other two is determine which of the other two is the man and which is the machine”.the man and which is the machine”.

TranslationTranslation

• A machine can be said to have passed the A machine can be said to have passed the Turing Test if - an average interrogator has Turing Test if - an average interrogator has 30% chance or more of making the wrong 30% chance or more of making the wrong identification after five minutes of paired identification after five minutes of paired questioningquestioning

• Right identification – interrogator can tell Right identification – interrogator can tell which is the machine and which is the which is the machine and which is the human. Wrong identification – any other human. Wrong identification – any other conclusion, incl. don’t knowconclusion, incl. don’t know

• A machine must fool at least 30% of the A machine must fool at least 30% of the interrogators so they do not know which is interrogators so they do not know which is the machine and which is the humanthe machine and which is the human

TestTest

• The machine must fool the The machine must fool the interrogator into thinking that it is interrogator into thinking that it is more human than the hidden human!more human than the hidden human!

• Expect 50% result with 2 humansExpect 50% result with 2 humans

• Tough test for machinesTough test for machines

Turing’s ThoughtsTuring’s Thoughts

• ““The game may be criticised because the The game may be criticised because the odds are weighted too heavily against the odds are weighted too heavily against the machine. If the man were to try and pretend machine. If the man were to try and pretend to be the machine he would clearly make a to be the machine he would clearly make a very poor showing. He would be given away very poor showing. He would be given away at once by slowness and inaccuracy in at once by slowness and inaccuracy in arithmetic. May not machines carry out arithmetic. May not machines carry out something which ought to be described as something which ought to be described as thinking but which is very different from thinking but which is very different from what a man does? This objection is a very what a man does? This objection is a very strong one, but at least we can say that if, strong one, but at least we can say that if, nevertheless, a machine can be constructed nevertheless, a machine can be constructed to play the imitation game satisfactorily, we to play the imitation game satisfactorily, we need not be troubled by this objection”.need not be troubled by this objection”.

Test for Human Test for Human Intelligence?Intelligence?

• According to French (1990) “the test According to French (1990) “the test provides a guarantee not of intelligence provides a guarantee not of intelligence but of but of culturally-oriented human culturally-oriented human intelligenceintelligence””

• But Turkle (1997) clearly assigns But Turkle (1997) clearly assigns intelligence to machines “our general intelligence to machines “our general tendency to treat responsive computers tendency to treat responsive computers as more intelligent …”as more intelligent …”

• Hanard (1992): the Turing Test “sets AI’s Hanard (1992): the Turing Test “sets AI’s empirical goal” – it is not a mindless empirical goal” – it is not a mindless parlour game.parlour game.

What does the test actually What does the test actually test?test?

• Turing posed the game instead of Turing posed the game instead of answering the question “Can Machines answering the question “Can Machines Think?”Think?”

• Perhaps the test indicates that a Perhaps the test indicates that a machine appears to think (if it passes)!machine appears to think (if it passes)!

• Can we do any better if we test a Can we do any better if we test a human – how do we human – how do we knowknow that they that they think?think?

Turing on IntelligenceTuring on Intelligence

““Intelligent behaviour presumably consists in a Intelligent behaviour presumably consists in a departure from the completely disciplined departure from the completely disciplined behaviour involved in computation, but rather a behaviour involved in computation, but rather a slight one, which does not give rise to random slight one, which does not give rise to random behaviour, or to pointless repetitive loops”behaviour, or to pointless repetitive loops”

Penrose said: “there is a great deal of Penrose said: “there is a great deal of randomness in the (human) brain’s wiring”. randomness in the (human) brain’s wiring”. This is simply not true!! This is simply not true!!

Loebner PrizeLoebner Prize

• A unique aspect of the A unique aspect of the situation can be gleaned from situation can be gleaned from the annual Loebner Prize for the annual Loebner Prize for Artificial Intelligence, a contest Artificial Intelligence, a contest based on the Imitation Game.based on the Imitation Game.

• Named after/by Hugh Loebner Named after/by Hugh Loebner who puts up the prize money who puts up the prize money for the best machine each for the best machine each year year

Loebner 2004-2006Loebner 2004-2006

• Format: parallel-paired comparison of Format: parallel-paired comparison of each of four hidden-machines against each of four hidden-machines against each of four hidden-humanseach of four hidden-humans

• 25 minute test!25 minute test!• Task of each Task of each Interrogator,Interrogator, in four tests is in four tests is

to identify the machine/human in each to identify the machine/human in each pair – assigns a mark out of 100 to each pair – assigns a mark out of 100 to each (e.g. Entry A 49/Entry B 51 – B is deemed (e.g. Entry A 49/Entry B 51 – B is deemed to be slightly more human than A) to be slightly more human than A)

Downward Trend!Downward Trend!

Loebner Loebner ContestContest

Machine’s Machine’s highest Score highest Score from a Judgefrom a Judge

Highest totals to Highest totals to all 4 machines all 4 machines from a Judgefrom a Judge

20042004 4848 123 – Judge 2123 – Judge 2

20052005 4545 100 – Judge 2100 – Judge 2

20062006 2828 97 – Judge 397 – Judge 3

Expectations high?Expectations high?

• Expectation of Loebner Judges may Expectation of Loebner Judges may be increasing, or the machines be increasing, or the machines themselves are getting worse, or …? themselves are getting worse, or …?

• No machine since 2003 has deceived No machine since 2003 has deceived any Judge that it is human, whereas any Judge that it is human, whereas this was a regular occurrence up to this was a regular occurrence up to that time.that time.

• What about networking???What about networking???

How to spot a humanHow to spot a human

• How can a machine be caught out?How can a machine be caught out?• What about Turing’s argument from What about Turing’s argument from

disability, let’s take one example – “disability, let’s take one example – “have have a sense of humor”a sense of humor”

• Rickman (1999) asserts that “computers, Rickman (1999) asserts that “computers, whatever artificial intelligence they may whatever artificial intelligence they may display, cannot make jokes”display, cannot make jokes”

• So would humour catch the machine So would humour catch the machine out?out?

What is a joke?What is a joke?

• Telling a joke, Farghol asserts, is “an Telling a joke, Farghol asserts, is “an intentional act” (2006)intentional act” (2006)

• Farghol - exploiting conversational tenets Farghol - exploiting conversational tenets “generates conversational implicature and “generates conversational implicature and subsequent laughter” subsequent laughter”

• Is it possible that a machine Is it possible that a machine cancan ‘create’ ‘create’ (produce from its linguistic armoury) an (produce from its linguistic armoury) an original response that the receiver may deem original response that the receiver may deem witty during conversational interaction? witty during conversational interaction?

Machine Humour?Machine Humour?• Actual responses during the competition:Actual responses during the competition:

• User: User: Ok, but are you always right?Ok, but are you always right?• Jabberwacky:Jabberwacky: Sometimes I am leftSometimes I am left

• User:User: How small is small any how?How small is small any how?• Jabberwacky:Jabberwacky: SmallSmall• User:User: Yes but exactly how small?Yes but exactly how small?• Jabberwacky:Jabberwacky: Very smallVery small

• User:User: So why don’t you take over the world?So why don’t you take over the world?• Jabberwacky:Jabberwacky: I wouldn’t know what to do with I wouldn’t know what to do with

it… it… • Anyway, I’m pretty busy alreadyAnyway, I’m pretty busy already

Intentional Humour?Intentional Humour?

• What about this one?What about this one?• Again – taken from an actual competition transcript:Again – taken from an actual competition transcript:

• User: I need to ask you if you have had a User: I need to ask you if you have had a circumcision.circumcision.

• Jabberwacky: Ask then the question you need to ask Jabberwacky: Ask then the question you need to ask me.me.

• User: Have you had a circumcision?User: Have you had a circumcision?• Jabberwacky: It's been an average day. Not much Jabberwacky: It's been an average day. Not much

happening on my end.happening on my end.• User: Very witty I must say.User: Very witty I must say.

2008 Competition2008 Competition

• Held at UoR in October 2008Held at UoR in October 2008• 5 minute paired machine/human tests5 minute paired machine/human tests• Winning machine was Fred Roberts’ Winning machine was Fred Roberts’

‘Elbot’‘Elbot’• Elbot fooled 25% of interrogators (30% is Elbot fooled 25% of interrogators (30% is

‘pass mark’) into thinking it was human‘pass mark’) into thinking it was human• Elbot openly admits to interrogators, Elbot openly admits to interrogators,

when asked, that it is a robot!!!when asked, that it is a robot!!!• Interrogators included software Interrogators included software

engineers, scientists, philosophers and engineers, scientists, philosophers and newspaper journalists--hardly ‘average’newspaper journalists--hardly ‘average’

PloysPloys

• Various ploys can be used to try to trip up Various ploys can be used to try to trip up machines competing in the Turing testmachines competing in the Turing test

• Asking mathematical questions does not Asking mathematical questions does not work--these machines are not stupid--work--these machines are not stupid--machines are just as unlikely as a human to machines are just as unlikely as a human to attempt to give a mathematical answerattempt to give a mathematical answer

• Better to ask topical or local questions, to Better to ask topical or local questions, to discuss what the weather is like or what discuss what the weather is like or what colour the walls are paintedcolour the walls are painted

• The point is not whether a machine gives a The point is not whether a machine gives a ‘correct’ answer, but if it gives one that a ‘correct’ answer, but if it gives one that a human would human would

Try the testTry the test• At the Reading competition, one At the Reading competition, one

interrogator asked the same question to interrogator asked the same question to hidden entities:hidden entities:

• What did they think about the weather that What did they think about the weather that morning? morning?

• Three different responses came from a male Three different responses came from a male human, a female human and a machine. human, a female human and a machine. Which is which? Keep in mind that they all Which is which? Keep in mind that they all knew it was autumn/fall in England.knew it was autumn/fall in England.

• A. "I do tend to like a nice foggy morning, as A. "I do tend to like a nice foggy morning, as it adds a certain mystery"it adds a certain mystery"B. "Not the best, expecting pirates to come B. "Not the best, expecting pirates to come out of the fog"out of the fog"C. "The weather is not nice at the moment, C. "The weather is not nice at the moment, unless you like fog"unless you like fog"

AnswerAnswer

• Did you guess correctly that A was Did you guess correctly that A was the machine, B the male and C the the machine, B the male and C the female? Not that easy, is it.female? Not that easy, is it.

2008 Transcripts2008 Transcripts• Machine/Ultra Hal - scored ‘human’:Machine/Ultra Hal - scored ‘human’:• Judge: MorningJudge: Morning• Ultra Hal: Good morning JudgeUltra Hal: Good morning Judge• Ultra Hal: It’s eight something, maybe you ate Ultra Hal: It’s eight something, maybe you ate

somethingsomething• Ultra Hal: Get it?Ultra Hal: Get it?• Judge: YesJudge: Yes

• Foreign female human - scored ’40/100 machine’:Foreign female human - scored ’40/100 machine’:• Judge: So let’s cut to the point. Are you the Judge: So let’s cut to the point. Are you the

human or the computer?human or the computer?• Hidden human: What do you think?Hidden human: What do you think?• Judge: That’s just the sort of answer I would Judge: That’s just the sort of answer I would

expect from a computer ……………. expect from a computer ……………. Sarah Palin!!!Sarah Palin!!!

• Successful double bluff – Elbot adjudged human Successful double bluff – Elbot adjudged human male:male:

• Judge: OK, fair enough, I’ll stop with the slang, Judge: OK, fair enough, I’ll stop with the slang, how are you?how are you?

• Elbot: Just a moment! I’ll start my self-diagnosis Elbot: Just a moment! I’ll start my self-diagnosis algorithm and let you know the resultsalgorithm and let you know the results

• Judge: Very funny, that’s exactly what you’d say if Judge: Very funny, that’s exactly what you’d say if you were a computer right?you were a computer right?

• Character forming – Elbot adjudged to be human Character forming – Elbot adjudged to be human adult male:adult male:

• Elbot: It seems to me with you human beings, the Elbot: It seems to me with you human beings, the more proof there is of something, the less likely more proof there is of something, the less likely you are to believe it.you are to believe it.

• Judge: Yeah, that’s human nature. But you Judge: Yeah, that’s human nature. But you wouldn’t know about that, would you?wouldn’t know about that, would you?

• Elbot: You’re contesting. I think I got it right the Elbot: You’re contesting. I think I got it right the first time.first time.

• Judge: Has anyone ever told you that you’re hard Judge: Has anyone ever told you that you’re hard work. Wife maybe?work. Wife maybe?

Comments on 2008Comments on 2008

• Difficulty with ‘average interrogator’ – how Difficulty with ‘average interrogator’ – how many/practically? What is average? – many/practically? What is average? – immense problem, even ethical issues to get immense problem, even ethical issues to get large numbers of ‘appropriate’ interrogators large numbers of ‘appropriate’ interrogators

• Interesting how interrogators can be fooled, Interesting how interrogators can be fooled, but do not realise it – Times reporter/Oxford but do not realise it – Times reporter/Oxford philosopher – both reported how easy it was philosopher – both reported how easy it was to tell which was which, both made incorrect to tell which was which, both made incorrect decisions! decisions!

Importance of Turing Test?Importance of Turing Test?

• Human-centricHuman-centric

• Human communication only – Human communication only – understanding?understanding?

• AI milestoneAI milestone

• Cyber crimeCyber crime

• Learn a lot about humans – biases, Learn a lot about humans – biases, preconceptions etc – outward appearancepreconceptions etc – outward appearance

• TerminatorTerminator

Penrose’s PitfallPenrose’s Pitfall

• ““Genuine intelligence requires that Genuine intelligence requires that genuine understanding must be present”. genuine understanding must be present”.

• ““Actual understanding cannot be achieved Actual understanding cannot be achieved by any computer”. by any computer”.

• As a result, “Computers will always remain As a result, “Computers will always remain subservient to us (humans), no matter subservient to us (humans), no matter how far they advance”. how far they advance”.

ThoughtsThoughts

• Turing Test will likely be passed in Turing Test will likely be passed in the near future – 2012 is Turing the near future – 2012 is Turing centenary yearcentenary year

• 2323rdrd June 2012 (100 June 2012 (100thth Anniversary) – Anniversary) – Tests being carried out at Bletchley Tests being carried out at Bletchley Park (Enigma)Park (Enigma)

Contact InformationContact Information

• Web site: Web site: www.kevinwarwick.comwww.kevinwarwick.com

• Email: Email: k.warwick@reading.ac.ukk.warwick@reading.ac.uk

• Tel: (44)-1189-318210Tel: (44)-1189-318210

• Fax: (44)-1189-318220Fax: (44)-1189-318220

• Professor Kevin Warwick, Department Professor Kevin Warwick, Department of Cybernetics, University of Reading, of Cybernetics, University of Reading, Whiteknights, Reading, RG6 6AY,UKWhiteknights, Reading, RG6 6AY,UK