I doubt very much I'll ever own a smart watch (so perhaps I should stop commenting now) but one of the barriers to wide spread adoption I see is that they all look about a elegant as wearing half a house brick on your arm. There's no way I would wear a lump like that on my wrist and I'm a medium sized guy. I can only imagine how daft it would look on someone smaller which, lets be honest, is about half the population.
I suspect that the current generation of smart watches are a bit like very early mobile phones. A few people will find them useful but most people will just laugh at them for a decade. It'll be interesting to watch the technology improve though, I wonder how thin and lightweight they will be able to make them.
The other barrier of course is a lack of differentiation with the phone in your pocket. Yes, checking the time on my wrist would be a tiny be faster than reaching into my pocket but does it bother me enough to go back to wearing a watch?