self-publishing != vanity publishing
self-publishing is when you act as your own publisher, doing all the stuff that comes before (and alongside) actually printing the books (or packaging them into the e-book format of your choice). for many niche writers, this may be more effective - publishing houses are notorious for not actually doing any marketing outside of the latest pot-boiler, and even in some cases for what amounts to sabotage of the writer's own marketing efforts.
vanity publishing is when a normal publisher lets *anyone* pay to publish books on the publisher's label. vanity publishers usually have lower-than-normal acceptance criteria (i.e. non-existent aside from legal liability checks), as the vain writer is paying to have the book published.