Intelligence is not consciousness
so without consciousness AI is only an I/O supervisor, making decisions in isolation based on it's inputs. It can iterate many possible outputs and learn which might be optimal within certain parameters, but is not aware whether those might not be optimal for those using it for whatever purpose. If a self driving car for example decides to drive off a cliff to save driving over 30 cyclists that cannot be avoided, is that optimal or not? From a conscious drivers point of view a whole slew of decisions would flood their mind, taking into account a lot of 'unconnected' things that an AI wouldn't. Should I sacrifice myself for the greater good? Should I steer into the rock wall on the other side so I might have a chance of surviving? Do I actually care about the cyclists that much?
Just a thought.