I have just finished building a low-power 1U server. It takes 15 W, compared to the 80 - 100 W that this type of box would normally need. That equates to about £65 of electricity per year for an always-on server, before you start to consider the cost of aircon.
It's certainly true that the power supply makes a significant difference. The PSU supplied with my case was only 40% efficient at low load; I have replaced it with a picoPSU and separate 240 to 12v supply, which have a combined efficiency of about 75%. That has saved me around 12 W.
Large savings can also be made elsewhere in the system. For example, I'm using a solid-state disk which takes about 0.25 W most of the time and less than 1 W peak. That's saving perhaps 10 W compared to a magnetic disk.
More savings can be made by enabling the power-saving features in the hardware and O.S. For example, this board can switch its CPU clock frequency depending on load; this is described as a laptop feature and is not enabled by default, but it's just as useful in a server application and probably saves a couple of watts.
But I think that the greatest savings can be made by not using a faster processor than is actually necessary. In my server, I'm using a 1.2 GHz VIA C7-M. As the article says, Intel and AMD have largely based their businesses on making faster and faster processors, and I imagine that they make much greater margins on their bleeding-edge chips than on slower, lower-power ones, so they don't have much incentive to encourage people onto the lower-power chips: hence this "blame-displacement" exercise aimed at power supply manufacturers. I like VIA because they make exclusively low-power processors.
One final consideration is the role of software. We all know that "bloatware" expands to occupy the resources (CPU, memory, disk) available. With a bit of effort, it's possible to benchmark your server and track down the bloat; having identified the worst-offending applications you can tune or replace them. In my recent experience the things that need attention are databases, which work well only once they have been "tuned", and free scripts (e.g. PHP message boards) which might look nice but turn out to be very inefficiently written.