Re: The processor really shouldn't matter to applications these days.
A problem particular to Linux is that the system-call numbers are different on different architectures (notably between x86 32-bit and 64-bit). I'm sure it's not an unsurmountable issue, but fixing it does require the final-stage compiler to know a little more about the bitcode than a straight compiler would.
The reason why adoption would be slow is that storage space is cheap, and posting binaries allows the maintainers to keep compatibility promises: basically if there's an ARM binary up, you can be sure the maintainer has at least run the package on ARM. Right now, there are only three or four major architectures in use (which three or four depends on whether the package is used in embedded or server/desktop applications), so it's not too much hassle for a package maintainer to build 3 versions of a package and post them to a repo. The alternative saves a little time, but then the maintainer will have to deal with the possibility of someone using their package on an architecture they've never built it on themselves.