This is a real concern
It is mostly a concern for those affected, but the number of people affected continues to grow. Between automation, outsourcing, and other economic shifts (i.e. away from coal/oil to green energy) more and more jobs in rich countries like the US are going away. When it is an economic shift, generally jobs are created that offset the loss of jobs. The problem is that they might be created elsewhere - i.e. coal miners and oil workers in the US who lose their jobs because of solar energy becoming more and more viable will be "replaced" by Chinese workers making solar panels.
The unrest of those who are displaced could be explained away by the powers that be as communist/socialist sympathizers when it was only factory workers - i.e. those who claim unions are communist or socialist organizations and should stamped out. Now that it is starting to impact some white middle class jobs it will be harder and harder to ignore as the number of voices grow. They see the republicans as caring only about the rich and screw the little guy, while the only answer they see from democrats is to have the government help them with handouts. They don't like either answer, so it is no surprise they are rejecting both and looking for alternatives in greater numbers. Even those who aren't affected today see friends who are and worry what will happen to them, or their children.
Self driving cars will be great, but all those "jobs" created by Uber (by displacing cab drivers) will simply go away within a decade of the first 'official' self driving car hitting the road. Truck drivers will go away also, with people needed only to load/unload goods at each end. School bus drivers probably go away - unless school buses go away. They might be replaced by slack capacity from the local self-driving cab company, or parents in a neighborhood could band together and use one of their self driving cars. Better to transfer them in groups of 4-8 than 40-80 that inevitably means it takes the kids a lot longer to get there with all the other stops.
When we reach the point where we can build a robot able to do most of the physical things a human can do (some better, some worse) and is able to carry on a conversation well enough to serve customers and communicate as needed with human "co workers", look out. Those displaced will be permanently out of the workforce. A society can't educate its way out of this, those who are simply not intelligent enough or physical capable enough to do better than robots are "useless" to society, and their number will grow as the capabilities of the robots improve.
What do you do with people who are effectively useless as far as productive work in society? There are various options, some that will be unpalatable to those who own the robots and think they deserve all the benefits from their investment, to those that will be unpalatable to (or threaten the very existence of) those deemed "useless" to society. Just look at what we do with the "useless" today, using the example of those with severe mental or physical handicaps. While some places (typically non-profits) create what are effectively charity jobs for them, mostly they are given handouts. That's fine for a small number of them, and the temporarily (for some definition of temporarily) unemployed. That won't work well when you're talking 25% of the population, and is unsustainable when you reach 50%, for obvious reasons.