The EDL is considered, rightly or wrongly, to be far-right because of its deeds rather than because of its words. On several occasions, the EDL has planned a protest only for it to descend into hooligans who self-associate with the EDL smashing up private property. People like those caught on video here: http://www.liveleak.com/view?i=794_1369521652
The public perception has therefore become that EDL marches are likely to descend into hooligans destroying private property. People also generally associate hooliganism with right-wing sentiments, probably because of the serious problems football has had with both hooliganism and with racism, tying the two things together in the public imagination.
Even taking your argument as accurate, and starting from the premise that the connection is a misperception, EDL supporters like yourself further cement the conclusion by blaming it on "the liberal left wing". Where is a group that despises the liberal left wing most likely to be on the political spectrum?
To a much lesser extent there's the 'English' in 'English Defence League'. Where do groupings that explicitly reference England, Britain or the United Kingdom in their name tend to fall on the political spectrum?
Those are the reasons that the EDL is perceived, and continues to be perceived, as right wing. If, as an EDL supporter, you want to shed the label, those are the problems you want to address.
Here's a test: I strongly dislike the EDL. From that statement in isolation, where would you assume my political beliefs lie?