Side-by-side is the solution to the DLL problem, and it works very well when used properly.
Given that you cannot guarantee binary* compatibility of all versions of a DLL with all versions of all programs that use it, you have two choices:
1) Install a copy of the DLL with every single application.
This uses lots of disk space (how many copies of the same release of MSVCP90.dll do you need?), and perhaps more importantly the user cannot (easily) update the DLLs to fix bugs.
The advantage is that the application will always use the exact same version it originally shipped with. One hopes that's also the version it was tested with!
2) Have a central repository of DLLs that maintains a list of all versions installed and ensures the most up to date binary compatible version is loaded by each program.
This saves disk space and means DLL updates can easily be applied - and rolled back.
The downside is that every program needs a correct manifest stating which version is binary compatible - and a bad application/installer can of course screw that up or forget it altogether!
3) Install the newest version of the DLL into a central repository and don't bother checking anything.
This will blow up in your face. Microsoft did finally learn that.
4) Compile everything from source so it all uses the same version of the DLL.
Not an option for proprietary software!
* Or source compatibility either.