back to article Want to train a dragon? You'll need 500 million files, 730TB of data, 54,000 CPU cores...

Family favourite DreamWorks Animation has built a cloud platform powered by microservices that uses a graph database and Gremlin query language to guide the production of its films. This digital pipeline handles everything from early pre-visualisation to the final render and will be employed in its next feature. This setup has …

  1. Craig 2

    Gremlin...

    Don't get them wet. (The servers)

    Keep away from bright lights. (The staff)

    No coding complex queries after midnight??

  2. Aladdin Sane Silver badge

    Yeah, pretty sure this could run Crysis.

  3. Tromos
    Joke

    730TB of data and 54,000 CPUs

    Or, alternatively, a mid-range 2029 smartphone.

    1. Zarno Bronze badge
      Joke

      Re: 730TB of data and 54,000 CPUs

      But will that future phone still have one byte(bite) missing from close to the core?

      Sorry, Apple puns, they write themselves...

      1. bombastic bob Silver badge
        Devil

        Re: 730TB of data and 54,000 CPUs

        and that future phone, with all of it's CPU cores, RAM, and computing power, will still have a 2D FLATTY interface, allegedly because it "wastes CPU cycles" to make it 3D Skeuomorphic... (whether people PREFER the 3D or not, and every study _I_ have seen says it's about 2:1 in favor of the 3D-like appearance of things as far back as Windows 3 and OS/2 PM 1.2...)

        and people will STILL use their uber-powerful smart phones just to look at cat videos, e-mail, and the next gen of Fa[e][cebook. And maybe a game app or two...

        meanwhile, Dreamworks would hopefully settle for an integrated solution where all xxx,xxx cores can work with one another gracefully. And probably running Linux, or a BSD or Solaris derivative OS...

        thinking of 'mythical man month' there's 'mythical CPU core count increase' too, for determining how effective a 'moore's law' architecture evolution is when you can't just double the frequency any more.

  4. Anonymous South African Coward Silver badge

    Here there be Dragons.

  5. Pirate Dave
    Pirate

    Files

    Million or billion? The headline says 500 million, the article says 500 billion. That's a lot of files either way (for a 2 hour movie that would have a max of 172,800 individual frames at 24FPS), but one seems a whole order of magnitude greater than the other.

    1. Korev Silver badge

      Re: Files

      It's a huge number of files.

      In HPC land we'd normally use something like HDF5 to store the images (or other matrices) which are then very fast to access compared to have zillions of files all over the filesystem. Some people also use SQLite for this and I assume there are other similar tools that'd also work.

      1. I.Geller Bronze badge

        Re: Files

        If they use AI technology each word in each pattern is annotated with at least one dictionary definition, which is a file; at least 3-5 million files. Plus synonymous clusters on each paragraph are separate files; 1-50 files on each paragraph.

        1. Kobblestown

          Re: Files

          I don't know what you're talking about, but I'm pretty sure you're doing it wrong.

    2. David 132 Silver badge
      Headmaster

      Re: Files

      Technically, three orders of magnitude greater.

      1. Korev Silver badge
        Coat

        Re: Files

        Do you know where to log a complaint?

        1. bombastic bob Silver badge
          Coat

          Re: Files

          /var/log/complaints.log

          it's a symlink to /dev/null

  6. I.Geller Bronze badge

    Apache Cassandra, a NoSQL database originally developed at Facebook, and Apache TinkerPop, a graph computing framework supported by multiple database vendors.

    I hope they create synonymous clusters?

  7. Anonymous Coward
    Anonymous Coward

    Hold on a sec. Somebody mentioned “How To Train Your Dragon” and there’s no NetApp reference yet ???

    Well did you know that NetApp was involved in the production of “How To Train Your Dragon” ?

    One of dreamworks employees sent an email. And that email was actually stored on a NetApp system somewhere.

    And then a graphic designer had a NetApp mousepad.

    So proud to be part of something big and important !!! Wouldn’t be possible without the great culture.

  8. swm Silver badge

    Wow! However they do it the end result (movie) is impressive!

  9. Bob Ajob

    Key messages

    The key messages here are very encouraging from a DevOps perspective. They had the balls to experiment at a scale that affected their whole pipeline. Creating a dummy show probably cost less than the cloud hosting and time spent learning the new tech. Bravo to whoever had the vision to do this.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020