Whole books have been written on the subject, but there were a few key things.
Commercially, commodore didn't spend enough on R&D or promotion. They were used to the very long life cycle of the C64 and presumed the Amiga could survive in the same basic format for a decade.
Technically, moore's law means that commodity hardware (x86 PCs) always wins. What can only be achieved on custom hardware today will be done cheaper and better on standard hardware tomorrow.
Piracy saw developers move to machines perceived to be more secure - the cartridge and CD based consoles.
There are many other issues but the last one I'll mention is architecture. It's cleverness was also it's downfall. All the clever interplay between chips and systems and RAM limited the ultimate speed of the machine. You have to break the tight integration to go faster, but doing that breaks the backward compatibility of software that talks directly to the hardware. A faster Amiga wouldn't have run any Amiga software. The tech reached the end of the line. Modern Amiga implementations that are faster retain compatibility only through emulation.