I think you (partially) answered your own question...
"...why do some of them ignore specially provided cycle lanes,..."
'On "shared space" there can be children playing as well as dogs both on and off leads...'
If they are not designated as shared, then they often have inadequate separation from the footpath, and wandering pedestrians encroach. So, a responsible cyclist takes the road instead.
Other problems with cycle paths include uneven surface, insufficient width and frequent forced stops.
"What might well work is cyclists taking more care and cycling at a more genteel pace, but these are options they are either unwilling or incapable of taking."
Is that surprising, if you are using a bicycle as transport? How about bringing the 4mph limit and red flag for motor vehicles, and merging roads and footpaths?
A shared space of pedestrians and casual recreational cyclists could be very nice, but town planners and the car lobby shouldn't pretend it's a route for people meeting their transport needs without burning oil.