" bothered to vote."
That's the error - or one of them.
Anyone who was a "don't know" or couldn't make their minds up strongly enough to take the decision is, in effect, being considered as not actively wanting to remain. There was no option in the referendum for "I'm not sure if I want to remain but I don't actually want to commit us to leave" or words to that effect.
It was very much a matter of forced choice or no choice. In effect the voices of an undoubtedly large minority for Leave had greater weight than the majority who didn't vote for changing the status quo.
For this reason, most constitutional changes, even for a sports club let alone a group of nations, there usually has to be a threshold. For a strike ballot to be valid there's a proposed 40% of all members. But we are saying that the electorate have spoken because a minority who actively wanted change outvoted the majority who either didn't or weren't sure