Re: why don't we:
We won't generally go on the record and clearly state that "yes" we're prepared to make things easier for terrorists, paedophiles and assorted other nefarious characters.
It certainly isn't the binary alternative you present.
A backdoor is a backdoor for anyone who comes past to try turning its handle and they're not all good guys. Introduce a backdoor to aid law enforcement and you're also aiding some of those nefarious characters as well. Was that what you wanted?
Another factor is that a society that believes in freedom under the law has got to build on elements such as due process and presumption of innocence. Backdoors are antithetical to these.
And, for the record, I spent a good proportion of my working life in a job gathering evidence to prosecute, terrorists, sex offenders and other nefarious characters. One thing that I take away from that is a strong belief in due process of law and the presumption of innocence. Terrorists would remove those if they got their way so why should we give in to them by removing them ourselves?