this has gone on for ages and ages. And it will continue for ages and ages...
The problem is two-fold:
1) Management is paid (as already indicated) partially by bonuses upon delivery of something. That leads to rushed schedules. Always has and always will.
2) Even when the project is intended to be agile, there's always somebody fixing the budget. That leads to delayed functionality.
I always ask my clients: Is this an agile project? If they say yes, I present them with a never-ending, all-encompassing contract where they pay me $y a month, and I promise to deliver x hours worth of work doing whatever it is they tell me to do. They always come back with: "But we only have $ amount"... To which I always reply: "So the project is not agile, then, is it?"
If the clients indicate that it is a waterfall project, I immediately pull out a Business Analyst and let that person have a go at finding all the requirements. Then I add a System Architect / Data Architect to translate those requirements in costable work-units. Then I add a Finance Guru / Sales specialist to come up with a cost for the *agreed* scope. When everything is signed, I add a dogmatic scope-and-finance driven project manager to control the client, whilst I let developers get on with the job at hand.
If the client wants to restrict time, or start early, we can always talk about versions of the software-to-be-built. The balance the *client* has to find is: what functionality is needed for the first version. A very difficult balance to strike.
Ultimately, even in properly scoped, documented, managed projects, there will inevitably be delays. What most projects and project managers forget is that you *need* a communications matrix that tells you who your friendly counterpart is to help you solve which type of problem (money, schedule, scope). For *massive* replacement projects, you need to take all the end-users from day one on a journey that tells them about the new system, shows them how it works, and gives key-users the ability to interact with the new system while it is being build... *Never* *ever* develop something in a closed-door environment to reveal it on D-Day to the hordes of end-users... They will inevitably not like what they see, because it is "different"...
There you have it... More than $0.02 worth, but you're very welcome!