The human mind is weird
I think that having a conversation with somebody in your car isn't too taxing for your brain. But if you have that same conversation with somebody over the phone, your brain suddenly zones out and forgets to pay attention to the road instead of the conversation. I think your brain is busy visualizing the person on the other end of the phone instead of the road in front of you (at least I and several others I've talked to have said the same thing). If that person were right in front of you, your brain wouldn't have to do that and could stay focused on the road.
Yes, there are other distractions in your car that can cause accidents, but they are brief in nature. Adjusting the radio, fiddling with climate controls, and yelling at your kids only take a second or two. Phone conversations last minutes. The longer the distraction, the more likely it is to cause an accident.
With texting, the brain isn't zoned out, but that distraction lasts a a few minutes because it's extremely difficult to type on a phone while looking at the road at the same time. After a few seconds of frustrating typing, you're tempted to look down at the phone for several seconds at a time just to get the text finished. And while your looking down, you're obviously not looking at the road.
The handsfree kits are worthless. They don't address any of the concerns listed above. Phones are much more distracting than anybody realizes. That's why people ignore the ban, and that's why people who use handsfree kits still get into just as many accidents.