back to article Zero. Zilch. Nada. That's how much Netflix uses its own data centres now

Netflix has announced it no longer does any meaningful work in its own data centres. The video streamer says it was turned off on-premises IT when it suffered database corruption in 2008. It's taken eight years of work to turn off that on-premises tech. During that time Netflix moved everything into Jeff Bezos' Discount Bit …

  1. A Non e-mouse Silver badge

    Single Source

    So Netflix now have a new single point of failure: Amazon. Do you really want your entire business dependent on just one supplier?

    1. Destroy All Monsters Silver badge
      Holmes

      Re: Single Source

      Yeah, but is it REALLY a single point of failure?

      Internally the system is likely to have enough redundancy and loos coupling to survive quite a beating, possibly even downtime in several AWS centers.

      Netflix also has a single point of failure "The Earth" but that fact is not really important or interesting.

      1. Franco

        Re: Single Source

        You'd expect it not to be a genuine single point of failure other than generically saying "Amazon", but only five months ago http://www.theregister.co.uk/2015/09/23/aws_outage_explained/

      2. keithpeter Silver badge
        Coat

        Re: Single Source

        "Internally the system is likely to have enough redundancy and loos coupling to survive quite a beating, possibly even downtime in several AWS centers."

        @Destroy All Monsters

        Above is a quote from OA that caught my eye. Your comment...

        "Internally the system is likely to have enough redundancy and loos coupling to survive quite a beating, possibly even downtime in several AWS centers."

        ...I'm wondering about consistency of the data across independent geographical centres and vertically across different microservices. A relational database does guarantee (eventual) consistency with various arrangements for rollback &c

      3. Anonymous Coward
        Anonymous Coward

        Re: Single Source

        You never outsource risk. I've seen it so many times, higher management thinking someone else can do the tech that should be done in-house. They think someone else is liable for the risk, but no. Amazon may supply the tech, the business risk is still netflix.

      4. roblightbody

        Re: Single Source

        So in summary, you're saying AWS is too big to fail?

    2. Don Dumb
      Stop

      Re: Single Source

      @A Non e-mouse "So Netflix now have a new single point of failure: Amazon. Do you really want your entire business dependent on just one supplier?"

      No only that but would you want that one supplier to be your direct rival?

      1. Warm Braw

        Re: Single Source

        would you want that one supplier to be your direct rival?

        I think that might depend on whether you believe that supplier has the clout to put you out of business anyway and that my cozying up to them you might secure reasonable terms for a buyout.

      2. Named coward

        Re: Single Source

        Some business sense says you wouldn't want your single supplier to be your direct rival but in some very unscientific single-sample tests Netflix streaming is somehow better than Amazon prime (no buffering, ever vs from none to too much depending on time, as well as possibly on weather patterns and phase of the moon)

        1. Anonymous Coward
          Anonymous Coward

          Re: Single Source

          "Some business sense says you wouldn't want your single supplier to be your direct rival but in some very unscientific single-sample tests Netflix streaming is somehow better than Amazon prime (no buffering, ever vs from none to too much depending on time, as well as possibly on weather patterns and phase of the moon)"

          That is certainly a strange claim of dependability towards Netflix as I upgraded to 100Mbps and still get Netflix delays, thank you very much.

      3. Stevie

        Re: would you want that one supplier to be your direct rival?

        [MODE = 'mos']

        They wouldn't do anything bad. They're our friends.

        On no!

      4. Captain DaFt

        Re: Single Source

        "No only that but would you want that one supplier to be your direct rival?"

        Works both ways.

        Maybe Amazon should be asking itself, "How can we successfully compete against a major client without hurting our own bottom line?"

      5. joed

        Re: Single Source

        on the other hand dependency is on both ends. It's the same way a bank relies on viability of its debtors. For small ones they send a repo man but they'll negotiate with ones that are too big too fail (and take significant percentage of revenue with them). I bet that Netfix had some contingency plan - maybe difficult and costly but useful in pricing negotiations.

    3. Anonymous Coward
      Anonymous Coward

      Re: Single Source

      It would be odd if they didn't at least have a test version of their services ready to go on some other cloud providers kit, at least to use to negotiate cheaper rates with Amazon.

      Surely they do? It can't be a big job ( for a company like Netflix ) to maintain that, given the pricing advantages it would give them in the long term.

  2. Steve Davies 3 Silver badge

    Netfix is at the mercy of Amazon

    how long before AWS prices rise?

    as has been said, it is a single point of failure

    Miss paying your bill by 1 second? Bye-bye Netflix.

    Whoever made the decision to put the whole lot into AWS should be sacked

    Unless..........

    Netflix wants to be bought by Amazon.

    As for Apple wanting out of AWS. Well, they are building large bit barns of their own all over the world (Planning permision in Galway permitting) so it sort of makes sense for Apple to at least reduce their dependency on AWS. I'd still use AWS where there isn't a local Data Centre but for the rest of the world? As is usual for the fruity company, they like to do things their own way.

    If they do get out of AWS totally perhaps this rush for 'everything must be in the cloud' might slow down a tad. Probably wishful thinking though...

    1. Destroy All Monsters Silver badge

      Re: Netfix is at the mercy of Amazon

      Miss paying your bill by 1 second? Bye-bye Netflix.

      Do you really think Netflix is paying by VISA via an AWS account?

      1. Tim Jenkins

        Re: Netfix is at the mercy of Amazon

        "Do you really think Netflix is paying by VISA via an AWS account?"

        No, but they'll be buggered when the member of staff who signed up for the AWS account leaves, and they realise no-one else in the office knows what their first pet was called...

        1. ben edwards

          Re: Netfix is at the mercy of Amazon

          Please. Netflix is a large-scale internet company full of millenials.

          Just look for a post-it on the ex-staffer's monitor with MASTER PASSWORD as the title.

    2. Velv
      Facepalm

      Re: Netfix is at the mercy of Amazon

      "Miss paying your bill by 1 second? Bye-bye Netflix."

      Don't be so stupid!

      A) there are laws that protect even corporate customers who are late paying. There are very serious consequences if you as a supplier withhold service from a customer and interfere in the operation of their business, and simply "paying late" very much falls into this category. There is case law supporting this!

      B) you don't piss off your customers by cutting them off instantly. You pick up the phone and talk to them. And I'm pretty sure Jeff and Reed are already on first name terms.

    3. Anonymous Coward
      Anonymous Coward

      @Steve Davies 3

      If they do get out of AWS totally perhaps this rush for 'everything must be in the cloud' might slow down a tad. Probably wishful thinking though...

      Why would Apple leaving the public cloud for their own private cloud slow down the "everything must be in the cloud" rush? Apple isn't abandoning a 'cloud strategy', just using their own cloud.

  3. Voland's right hand Silver badge

    What they did makes sense. Just lots of iffs

    What Netflix did makes a lot of sense if and only if (iff) the following iffs are met:

    1. There is more than one supplier

    2. You can put a money tag on every (micro)service.

    3. They have an alternative supplier for monitoring and troubleshooting infrastructure.

    They failed on (1). They moved everything to single supplier. That has been noted by quite a few people.

    Part 2 is the really interesting bit for me. Cloud is pay per use. You have a monitoring service. How do you put the "financial contribution" on it and how do you know it is good value for the money. Most businesses operate IT purely as a cost center. For them, moving to microservices and moving to the cloud immediately results in making things like monitoring, etc available for beancounters to cull. By the way, if they are still purely "cost" the beancounters would be bloody right to cull them too. So all the technical parts of what Netflix have done are not interesting, show me their IT financial modeling methodology.

    And the obvious 3rd one - how do you monitor Amazon cloud performance off Amazon's cloud. The idea of doing that is technically and financially complete and utter idiocy.

  4. PleebSmasher
    Dead Vulture

    WWAPPLD?

    "If true, Apple thinks it can get out of the cloud. And if it does, what might that do to the rest of us heading in if it takes a huge slab of AWS' revenue with it?"

    Lower AWS prices?

    1. Anonymous Coward
      Anonymous Coward

      Re: WWAPPLD?

      Not sure why the downvote, but Apple leaving would free up a lot of capacity. If Amazon wants to sell that capacity they'd have to lower prices to generate new demand.

      The only way around that is if they are seeing enough growth from existing customers needing more capacity and new customers joining that they could cover Apple's departure by simply slowing down their build out of new capacity over the next year or something.

  5. Don Dumb
    WTF?

    A familiar smell

    "...made way to continuous delivery, engineering teams making independent decisions using self service tools in a loosely coupled DevOps environment, helping accelerate innovation."

    Why is it that any statement involving the term 'DevOps' is mix of various buzzwords?

    Even terms which have usually have meaning somehow lose any meaning at all in the context of a statement like this.

    1. Really Anonymous Coward

      Re: A familiar smell

      Have an upvote sir. Would anyone who's made this sort of thing work at any scale care to explain?

      I've worked in loads of loosely coupled environments - but not in the sense anyone would want to boast about!

      1. Anonymous Coward
        Anonymous Coward

        Re: A familiar smell

        I work in such an environment, and it can work very well. DevOps thrives where your staff is experienced, multi-disciplined and professional. You don't need to be a DBA to maintain a SQL system, or promote a deployment through environments, but you do have to know when you need your DBA, or when to allocate resource from your most experienced server admin. And that distinction is where DevOps can fall down if you sacrifice any of the trinity (experience, knowledge, professionalism). There's a massive difference between having a professional senior developer supporting a live deployment, and a junior cowboy developing in production.

        1. Anonymous Coward
          Anonymous Coward

          Re: A familiar smell

          A-bloody-men. One thing I made damned sure of was not allowing my staff pigeon-holing themselves. The more hats you can effectively wear, the more paths for advancement. It also kept some fun to your job as all of us were technologists - that familiar "that's curious."

          Watching my people make the jump from admin/technician to engineering was the most rewarding aspect of my job. Designing/building/fixing tech not quite as much.

  6. Anonymous Coward
    Anonymous Coward

    What Netflix produces?

    Nothing. It does resell you some digital media files produced elsewhere. For them, the move to cloud makes a lot of sense. Amazon takes care of making the files their resell available worldwide - and why not put billing there as well?

    It becomes a little more difficult when you actually produce -maybe physical goods - on premises.Very different needs - you may want your production lines keep on even if your ISP or anything upstream has issues, for example.

    1. dinsdale54

      Re: What Netflix produces?

      For all major markets, Netflix put their own hardware in to ISPs to deliver content. Basically a rack full of compute and disks. They push content updates once a day. For smaller countries, AWS makes a lot of sense, especially when you first offer a service. I assume there's a crossover point when it's worth swapping from AWS to your own kit.

  7. Orwell44

    Netflix need to have proper indexing and search options.

    It's very hard to find material to match my interests. For example, I might want to see films set in London. Type in London and you get about two movies which is clearly pathetic. Can you search by the date the film was made - No, in fact you can search on bugger all. Ideal for 4 year olds.

    1. Captain Server Pants

      They denormalized all their data

      So it will NEVER be fixed.

    2. Lamont Cranston

      I rely on Netflix recommending things that I might like to watch.

      Once if runs out of recommendations, I expect I'll ditch it. Both it and Amazon Prime are rubbish for films, anyway, thanks to Sky having the market stitched up.

    3. CCCP

      Remember Netflix is suggestions for the masses. There are probably less than 100k films world wide that get watched with any frequency, excluding pron. That is a small universe.

      Also, Netflix is becoming a production house with a website attached. So don't expect the selection to grow. It it the opposite.

  8. Anonymous Coward
    Anonymous Coward

    Physical assets

    I remember when a natural part of being a business was the accumulation of physical assets, specifically land/buildings/machinery. Even when the business went out of business, those assets could still hold tremendous value (especially the real estate). Now, what does a company like Netflix really own, besides a customer list and studio relationships? Not saying those aren't valuable, but imagine if they had invested in building data centers around the world instead of handing all that money to Amazon (forever). Would Amazon the box shifter being anywhere near as valuable if all their business was running on Azure?

  9. SysKoll

    Our own company moved its storage from AWS to our own data center. The Amazon bill was enough to pay for several storage server racks and their admins every month. We still save money even after factoring in the hardware, manpower, datacenter utilization, and bandwidth.

  10. Roland6 Silver badge

    Cautionary Tale...

    Some very good learnings obviously went on there, be interested to know if Netflex did it themselves or if they used an SI partner.

    What is clear "moving to the cloud" like downsizing mainframes a couple of decades ago, to get the best out of it, you need to re-engineer both the business and the IT systems. So isn't quite as trivial as some would have us believe.

  11. Anonymous Coward
    Anonymous Coward

    Should have known

    when my netflix services deteriorated even though i have 1GbE coming into the house.

    It all makes sense know.

    $%%#$%

  12. Infostack

    Best video platform out there. Period

    I use numerous video platforms and Netflix' is leagues ahead of everyone at every level. Could there be improvements? Of course, but compared with the rest, there is no comparison. I also study their business model and believe there could be changes, refinements and additions.

    That said, the article appears to have a slightly negative tone and fails to underscore the incredible flexibility and speed of innovation that results from moving to AWS/cloud. The results clearly show up in the above two ways of looking at the company.

  13. John 104

    OK, that is fixed. Now get some decent content. Sick of all the documentaries and crap movies.

  14. Anonymous Coward
    Anonymous Coward

    Marketing lies!

    The problem of course is that everything isn't in AWS and NetFlix do still operate real tin in real datacentres today. Admittedly, they may have moved their IT side and a lot of other functions, even those relating to their core service in house. They do still however have a rather massive global CDN - which is a fairly major part of the reason they could launch a global offering recently.

    All is never what it seems in marketing releases. Sure, most of the stuff will be in AWS - and for the naysayers, having came through physical/virtual/cloud, I prefer cloud, I get less hassle and it's more flexible. Those saying DevOps is a buzzword, utter ballwollops, it's real, it works and I massively prefer it (current org isn't DevOps friendly, last one was - we got tons more done in the last).

    My view is while most things will be in AWS and they no longer "operate" datecentres, they've still chunks of colo around the world with routers/switches/spinning disks for their CDN. Oh, it has it's own website here btw - https://openconnect.netflix.com/. You might want to look at the peering connections - https://openconnect.netflix.com/peeringLocations/ - and wonder why they need all that Bandwidth to serve all their content out of AWS ;)

  15. nilfs2
    Headmaster

    If you have all your things In-House...

    ...you become your own point of failure, so cloud haters stop whining, you (your mother's son) and your budget constrained bodging are a single point of failure, and one with even more chances of fucking things up than a multi-million datacenter mesh like AWS.

    1. OnlyMee

      Re: If you have all your things In-House...

      Agree completely. I have seen how many of the "in house" data centers are run especially outside of the absolutely largest organisations. Many of the people running these are first to point raise huge noise on cloud downtime event but cannot even provide real uptime figures of their own bit barn.

      Or when they do those figures do not include "scheduled maintenance" that usually consist of every situation where IT was first to figure DC is not responding.

  16. Anonymous Coward
    Anonymous Coward

    Not surprising Apple is leaving

    They've been building mega data centers of their own, clearly they had a reason for that. Some type of cloud environment is the only thing data centers on that scale would reasonably be used for since I don't think Apple is going to try to become the next Facebook or Google Search. Once they have their own why should they pay Amazon?

    When their needs were small it made sense to use AWS instead of doing it internally since that gave better fault tolerance and a better price, and it was easier to keep up with growth. That's no longer true as they can afford to build multiple data centers to provide that fault tolerance, and at their scale can probably do it cheaper than what Amazon charges them.

    Even if it doesn't save them any money keeping control of their data is important in this age with customers concerned about privacy and our government doing its best to pressure companies to strip away that privacy.

  17. Steve Medway

    'Only real businesses have Data Centres'

    It was once famously said "Only real men have fabs"...

    Surely it won't be long before someone says "Only real businesses have Data Centres"

    Shame that neither saying is true, 'Cloud' my arse it's just shiny wording for old idea, we'll get back to the real world when one of the Clouds goes REALLY tits up, but not until then.

  18. Anonymous Coward
    Anonymous Coward

    I'm fairly certain this article is incorrect. IBM has Netflix on their website doing a promo video for their success using XIV storage. I'm not sure where that is located, but not AWS.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like