back to article Cloud mega-uploads aren't easy

Google and Microsoft don't offer formal data ingestion services to help users get lots of data into the cloud, and neither seems set to do so anytime soon. Quite how would-be users take advantage of the hundreds of terabytes both offer in the cloud is therefore a bit of a mystery. Data ingestion services see cloud providers …

COMMENTS

This topic is closed for new posts.
  1. NZ ITer

    Try importing into Office 365

    It's not just the TB/PB users who have difficulty. We are in the process of trying to migrate a client with 150GB of data on their file server to an Office 365 SharePoint solution however all the tools supplied by Microsoft (SharePoint Workspace 2010 and something else) run at a trickle taking hours per GB regardless of the speed of the Internet connection.

    We are now having to investigate 3rd party tools for a one off upload job; not a great "Cloudy" solution.

    NZ ITer

  2. jake Silver badge

    ::shrugs:: Cloud storage is a tired, meaningless meme.

    The marketards don't seem to understand the difference between "bandwidth" and "latency" ... see this post.

    As a side note, see this post.

    1. Destroy All Monsters Silver badge
      Meh

      Re: ::shrugs:: Cloud storage is a tired, meaningless meme.

      Yes Jake.

      But on the other hand, no-one mentioned latency?!?

      1. jake Silver badge
        Pint

        Re: ::shrugs:: Cloud storage is a tired, meaningless meme.

        Did you even bother to read the article, DAM? The author (or, more likely, editor) might not know what latency is, but clearly they were discussing the concepts of bandwidth & latency in regard to so-called "cloud" computing, and the ramifications thereof.

        It's after 1700 hours somewhere ... and almost 18:30 here ... Beer?

  3. Anonymous Coward
    Anonymous Coward

    The thought of an unencrypted 1TB+ CSV file...it brings shivers to anyone who cares for their data.

  4. Anonymous Coward
    Anonymous Coward

    Teething problems, or something worse?

    The premise of "cloud" is that it'll be wonderful once we're all using it. It's the getting started that's lacking. This does seem to be a bit of an oversight on the pushy vendors' parts. It also spells doom for the idea of bailing out later on, ie getting all that data back out.

    So you decided to "go cloud". Did you even think about the getting data in, much less the getting data back out issues? And I don't mean on just the technical level; there'll be an entrepeneur for that.

    1. Lee Dowling Silver badge

      Re: Teething problems, or something worse?

      Data out is certainly more interesting than data in. Just what incentive do the cloud vendors have to help you remove the data after you've threatened to move elsewhere? I wouldn't be surprised if most or all of them offer import "somehow" if you kick up a fuss but not export at all.

      Do you really want to extend your contract by another month/year so that you can repeat the "guy on a plane with a NAS" trip in the hope of somehow getting all your business data back? Or spend quite literally weeks redownloading it all before you can move off to another provider? Or have to post them a huge drive array and wait for them to copy all your data out at their convenience and then pay to send it back to you?

      Cloud is one of those ideas that STILL doesn't know what it's supposed to be used for. There are lots of use cases, of course, but none for which cloud is the "optimal" solution.

      If you are storing PETABYTES of business critical data and requiring cloud-level redundancy and availability, are you seriously telling me that you COULDN'T buy servers yourself around the world and do it cheaper with your existing talent?

      1. Irongut

        Re: Teething problems, or something worse?

        Once your data is in one cloud you move to another by downloading it directly to the new cloud from the old. Thus using the providers' bandwidth leaving your own out of the equation. Of course it will still cost you a fortune and be somewhat time consuming to download all that data.

    2. badger31

      Re: Teething problems, or something worse?

      If it were me, I wouldn't trust my data entirely to the cloud. I would keep an on-site back-up that was always in sync with the 'live' data in the cloud. That way, if the cloud stuff goes tits up I would still have all the data safe and sound. The sames goes if I wanted to change cloud services provider. No need to get the data out, just delete it.

      1. Morg

        Re: Teething problems, or something worse?

        That's the thing, if you only sync it can remain cheap on bandwidth, plus it's only downolad which is much more accessible.

      2. Colin Millar
        IT Angle

        Re: Teething problems, or something worse?

        And immediately the "cloud" loses its meaning.

        If you have on-site and off-site copies the sensible way round is always going to be the off-site as the backup.

        And then the "cloud" is shown up for exactly what it is - a marketing buzzword for a reinvented wheel.

      3. KayKay
        Boffin

        Re: Teething problems, or something worse?

        This Cloud is free, is it?

        Yeah didn't think so.

        So for the same money or less, why not just keep TWO copies at home, one for everyday use and one for backup. You have the guys to run one, they can manage two, surely?

        1. jake Silver badge

          @KayKay (was: Re: Teething problems, or something worse?)

          Look up the concept of "off-site backup". I have lots, on five continents. None is "cloud" ... I own it all, it costs pennies per day to operate, and maintenance is fairly close to nil.

  5. Gordan

    Bitcasa?

    Actually, Bitcasa's offering is extremely good. It has features and advantages that none of the other similar services (Dropbox, Skydrive, Google Drive) offer:

    1) You can "cloudify" any folder transparently (including removable media and network shares).

    2) You can cache any amount of data locally (managed automatically) up to the amount of free space on the local machine. Actual data volume in cloudified folders is unlimited.

    3) It's all you can eat (but if you're going to abuse it you're going to need a LOT of internet bandwidth.

    Having had a quick play with it, it is quite awesome. There are a couple of limitations:

    1) No linux client. Tthis makes it mostly useless to me - having to run Windows in a VM to keep my folders in sync is a bit too much of a requirement. It also makes it less easy to keep the data encrypted (surely no sane person would keep their data in the cloud unencrypted?) and versioned (I have to stack cryptfs and copyfs on top, and then cloudify the bottom layer).

    2) No Android client.

    The lack of Linux and Android support makes it not yet ready to replace Dropbox for me. Once those two platforms are supported, I'll be sold.

    The only concern is how they plan to monetize it - all you can eat is a lot of disks to be maintaining without revenues. Without that, the service might just disappear with little or no warning, taking all the data with it. Even if there is reasonable warning, how much of a run on local disk supplies is it going to cause, and how long is downloading, say, 10TB going to take (i.e. how much of a warning are you going to need)?

    1. Irongut

      Re: Bitcasa?

      Is this a blatant advertising shill? The post seems to have no relevance to the article.

  6. Paul Johnson (vldbsolutions.com)
    Facepalm

    If "mega-uploads" aren't easy...

    ...what chance "tera-uploads"!?!?!?

    This all harks back to the 1990's when we shipped tapes around in vans to load data warehouses, send extracts to 3rd parties and send backups to safe locations. Not many went missing either ;-)

    Getting data in/out of 'da cloud' is the big gotcha, surely?

  7. bpfh
    Flame

    Big Data and commercial clouds

    I agree with Lee Dowling : if you have terabytes of data (or even big tens of GB), having it cloud stored is probably not the best solution... Clould backup, maybe, or replicated datasets between several different datacenters. I'd go with a replicated dataset between master and slaves... it would give the interns a nice project to work on for their college reports, and come out megabucks cheaper than a cloud solution... plus if you negociate a direct access to your datacenters or sweet talk your hoster's NOC support team, you can trundle across town with a couple of NAS or firewire drives and ask them nicely to chuck the disks onto a crash cart and plug them into your machine for a couple of hours rather than flying 35 hours in a Budget Air Dakota to a cloud datacenter somwhere in the Andes...

    My 0.02's worth...

  8. Phil Endecott

    DVDs

    Amazon's service works well for multi-terabytes, but I'd be interested in something that would work for smaller uploads, e.g. a single DVD. I could burn a DVD and walk to the postbox much more quickly than uploading the same data.

  9. Tim 11

    Legacy data

    I suspect the author of this piece isn't fully buying into the "new world order". Everything you do from now on will be stored in the cloud and manipulated by scraping your greasy fingers around on the tiny screen of your phone. This is the way of the future.

    Anything else you did before is "legacy", and Microsoft doesn't give a shit about it.

  10. Alan J. Wylie

    Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway

    http://en.wikiquote.org/wiki/Andrew_S._Tanenbaum

    1. jake Silver badge

      Re: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway

      We were using a similar phrase at DEC long before the supposed Tanenbaum quote. See:

      http://forums.theregister.co.uk/forum/containing/561112

  11. TReko

    Try importing to Google Apps

    We had the same problem moving our data to Google Apps. At least Amazon provides (for a large fee) the ability to ship them drives. Uploading 100's of Gigs of data to Google proved impossible via their web software. Luckily, we discovered is a third party app, Syncdocs http://syncdocs.com, that does reliable migration of huge data sets to Google Drive.

    The cost of migrating data will provide a new vendor lock-in, just like Microsoft locked us in with Office file formats.Once they have all your data, you're stuck.

  12. mrgareth
    IT Angle

    Salesforce Bulk API upload

    Great article Simon - Derek Laney from salesforce.com expands on the guidance provided to Simon here in this blog post: http://blogs.salesforce.com/au/2012/05/how-to-bulk-uploads-to-the-salesforce-cloud.html

This topic is closed for new posts.

Other stories you might like