9th June 2014, 8:34 AM
http://arstechnica.com/information-techn...ring-test/
I was never too impressed with the surprisingly lax standards of a turing test to begin with. I always thought some more rigor in the form of providing some intelligent questions for test takers to ask should be required.
Namely, if this chat bot is intended to be self aware AI, then it fails completely. If all it was meant to do was fool some test takers about 30% of the time, I have to ask what the point of chat bots aiming for such low hanging fruit is supposed to be.
Here's the crux of the matter. I have been able to fool every single chat bot I've ever come across with a ridiculously simple question: "What were we just talking about?". Eugene is, essentially, designed in the same way as every other chat bot out there. As I read on it, the only unique tricks were lowering the expectations of test takers by purposefully emulating a foreign 13 year old (one from the Ukraine that apparently can't answer basic questions about what happened last month IN the Ukraine). The problem is that every single chat bot programmer is stuck in the "Chinese box" notion of how to design a chat bot. Namely, they try to build the biggest database of responses they can (either themselves or trying to build one from user input ala the Alice chatbot) and then, if ONLY that database is big enough, it'll be able to pass for human. If only! Except, it's trying to build "perfect" responses to one-off inputs, ignoring the critical notion of CONTEXT. It doesn't matter if the "learning" AI is only "learning" which response to use for one specific input, because a "good" response to MOST input in a real world conversation changes moment by moment based on the situation. In a word: Context! If I ask "how are you doing today?" no SINGLE response will do, your response is determined by a multitude of events during that day and your current state. The same could be said for something as simple as "How's the weather?" or "Have you eaten?". However, no matter what tricks you set up for those SPECIFIC questions (often they'll set up a special exception for weather questions to check sites like weather.gov to determine a response, which by making such an exception is a tacit acknowledgement that their whole general strategy is flawed), the one question they can never find a "trick" to work around is a basic question about what you and the AI were JUST talking about. No specific response, short of actually making a competent chat program that can understand context, is going to fool such a question. The best they could hope for is to treat such questions with a special exception that just regurgitates whatever the AI stated last, but even that will be broken in certain series of strings.
In short, I've lost a lot of interest in the field of chat bots because programmers can't seem to get past the whole "build a bigger database" paradigm they're all stuck in.
I was never too impressed with the surprisingly lax standards of a turing test to begin with. I always thought some more rigor in the form of providing some intelligent questions for test takers to ask should be required.
Namely, if this chat bot is intended to be self aware AI, then it fails completely. If all it was meant to do was fool some test takers about 30% of the time, I have to ask what the point of chat bots aiming for such low hanging fruit is supposed to be.
Here's the crux of the matter. I have been able to fool every single chat bot I've ever come across with a ridiculously simple question: "What were we just talking about?". Eugene is, essentially, designed in the same way as every other chat bot out there. As I read on it, the only unique tricks were lowering the expectations of test takers by purposefully emulating a foreign 13 year old (one from the Ukraine that apparently can't answer basic questions about what happened last month IN the Ukraine). The problem is that every single chat bot programmer is stuck in the "Chinese box" notion of how to design a chat bot. Namely, they try to build the biggest database of responses they can (either themselves or trying to build one from user input ala the Alice chatbot) and then, if ONLY that database is big enough, it'll be able to pass for human. If only! Except, it's trying to build "perfect" responses to one-off inputs, ignoring the critical notion of CONTEXT. It doesn't matter if the "learning" AI is only "learning" which response to use for one specific input, because a "good" response to MOST input in a real world conversation changes moment by moment based on the situation. In a word: Context! If I ask "how are you doing today?" no SINGLE response will do, your response is determined by a multitude of events during that day and your current state. The same could be said for something as simple as "How's the weather?" or "Have you eaten?". However, no matter what tricks you set up for those SPECIFIC questions (often they'll set up a special exception for weather questions to check sites like weather.gov to determine a response, which by making such an exception is a tacit acknowledgement that their whole general strategy is flawed), the one question they can never find a "trick" to work around is a basic question about what you and the AI were JUST talking about. No specific response, short of actually making a competent chat program that can understand context, is going to fool such a question. The best they could hope for is to treat such questions with a special exception that just regurgitates whatever the AI stated last, but even that will be broken in certain series of strings.
In short, I've lost a lot of interest in the field of chat bots because programmers can't seem to get past the whole "build a bigger database" paradigm they're all stuck in.
"On two occasions, I have been asked [by members of Parliament], 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able to rightly apprehend the kind of confusion of ideas that could provoke such a question." ~ Charles Babbage (1791-1871)