My long term worry has been with the definition of intelligence itself as “finding a solution to a problem”. What is human, according to me is much much deeper and complex, than a set of problem solving abilities. In fact what we are most of the times is not “logical” but illogical, due to our complex mental makeup.
Our minds are not just doing problem solving, but many other things. That is why Freud, Jung, Maslow, Rogers, Seligman and others are still very relevant for us today. Though through tests and games we may evaluate the strategies and speed by which one solves the problem, or wins the game, if we take life and living as a wholesome process involving human relations, human actions, personality traits, psychological signatures, and an urge for true freedom, like in the character Andrew (Issac Asimov’s novel – The Positronic Man), what is human is by definition much beyond the goals of AI.
So, even if we create a self-aware Data (Star trek) like android entity, who may able to simulate emotion-like phenomena, still that entity will not be human.
Yet, I am curious to see where AI would take us in this century …
Well, here is the latest on AI – human interaction through Games – AlphaGo being tougher than chess. “Machines have had increasing success in the past half-decade at narrow humanlike capabilities, like understanding speech and vision. However, the goal of “strong A.I.” — defined as a machine with an intellectual capability equal to that of a human — remains elusive”. More at http://www.nytimes.com/2016/03/10/world/asia/google-alphago-lee-se-dol.html?_r=1
About Consciousness 