Re: We are
Maybe the Von Neumann machines do not use the same criteria for selecting target systems as we would. Perhaps they avoid systems with short-lived yellow stars, cold gas giants and life-bearing worlds and instead go for long-lived red dwarf stars with hot Jovians and/or lots of asteroids, and no pesky natives.
Or how about:
4) Civilisations generally don't do this because landscaping an entire galaxy is rude. But it only ever takes one, so perhaps:
4a) Species that engage in this sort of thinking tend to go extinct because they take the same approach towards their home system and hit Malthus's limit before they can spread too far. We may well kill ourselves off before we reach the necessary tech level, for instance.
Using fewer assumptions there are still a few other options:
6) It has in fact been done, but colonising civilisations are sparsely spread through space and time so that we simply missed them. If the last lot in the Milky Way disappeared a mere 50000 years ago, no trace of their works would remain visible from here (unless they left megastructures behind). In fact, we could go extinct again without ever being concurrent with a single other civilisation in the Local Group, even if generally speaking life is abundant in the universe.
7) Colonising the galaxy turns out to be really hard and its history is full of single-planet cultures wondering where all the aliens are. Most go extinct without ever seeing any.