Long ago I used to work for the company that originated Aspartame (in the south of the Netherlands) - I heard enough stories from "inside" to be at least a bit worried.
But I have a much easier reason why I rarely touch the stuff: I intensely dislike the taste of it. I don't like the taste when I'm drinking it, and I don't like it lingering in my mouth for up to an hour afterwards. I have "gourmet" friends who can't taste it, but I can tell its presence immediately in anything I drink or eat and it puts me off (ditto with the traditional Oriental additive monosodium glutamate, aka MSG - that I can taste too) . I also dislike the idea of "fake" sweetness - I think I'll be in better control of my weight if I actually taste for real what I eat. The good news is that that has permanently put me off chewing gum - you cannot get a chewing gum without Aspartame now.
Incidentally, there are new reports showing up that suggest Aspartame may make a mess of our blood sugar levels. That's a far more direct danger IMHO.