others will jump in too
I'm sure if enterprise use of this cloud storage starts to take off the likes of HP, NetApp, EMC etc will respond with products that can integrate there, at least at the file/NAS level, integrating the cloud tier into the box itself.
S3 is one service I have to admit Amazon does pretty well for what it provides. The rest of their cloud is absolute shit, but S3 isn't so bad. Still latency and bandwidth are still the biggest factors here, even with a local cache how much cache are you going to have? Users are not going to want to wait 20 minutes while an appliance downloads something from the cloud, and companies are probably going to find massive internet pipes to access cloud storage is probably going to cost them more than having storage locally. The amount of data being stored is rising at a much much faster pace than the decline of the cost of bandwidth. For me with a gigabit connection with about 17ms of latency to S3 I manage to get about 4MB/second of throughput from them.
I can see cloud storage (in conjunction with encryption) replacing frozen storage (i.e. tape), stuff that you rarely need to get access to, but higher tiers it'd be tough. It seems the trend is storing more and more data online, analytics and stuff.
Spoken as someone who has used S3 and other AMazon cloud things (ugh) for the past 1.5 years.
I think HP and Dell and stuff can do well in cloud storage, they know what they are doing, one thing(among countless many) Amazon does poorly is they do not offer tiers of storage, you can't pay them more to put more important data on enterprise storage (e.g. for database). They are introducing(or have recently introduced) SSD storage but I'm sure it's just a bunch of SSDs in a crapbox server like the rest of their infrastructure. This tiering doesn't really apply to S3 though, only to their other offerings.
My only concern for the likes of HP/Dell/etc building their own clouds is they may end up alienating their larger customers as they go head to head competing with them in the same market. Sort of like Microsoft and the Zune..