GPU-flingers' bash: Forget the Matrix, Neo needs his tensors

Last week, Nvidia held its biggest ever GPU Technology Conference (GTC). The big walk-away is that GPUs are rapidly becoming an expected and standard component of computing, table stakes in many cases, across the computing platform. Big deal right there and hence the frothiness of much of the coverage. Before the conference, I …

1. Well duh.

"A Tensor is just a generalized definition of such mathematical objects, an n-dimensional object that follows specific transformation rules."

Well, obviously.

Glad we cleared that up O.o

(Any definitions for intelligent but non-mathematical people welcome)

1. Re: Well duh.

A 'scalar' is a single number. Think of a single point in space.

A 'vector' is a column (or row, depending on your notation) of numbers. Stretch the dot out into a line, and the values of the numbers gives the direction of the line in however many dimensions you are using. e.g. A two number vector is like coordinates on graph paper, adding a third gives height as well, and so on into harder to visualise directions.

A 'matrix' is a grid of numbers, and can be thought of as a collection of stacked vectors wrapped up in an easy to process format. It can also represent a transformation - a way of converting one vector into another. The representation is the same, just the use has changed.

All of these are also tensors, of zero, one and two dimensionality. This can be extended, so a three dimensional tensor could be displayed as a 3D grid of numbers - a stack of matrices. Then stack those into 4D, etc, until your head turns inside-out.

However, the term 'dimension' starts to get confusing, as a vector with three numbers in it can be said to represent a vector in three dimensions, but it is also a one dimensional tensor.

2. Re: Well duh.

A tensor is a linear function from a bunch of vectors (and dual vectors, but don't worry about those) to numbers. Vectors are pretty much what you think -- little brass arrows of various lengths -- and 'linear' means that T(u + v) = T(u) + T(v) and things like that. This is an oversimplified description, of course, but the whole 'tensors are matrices with certain transformation rules' thing is what physicists still learn (I was one) and it's just not helpful.

2. What's a tensor?

To all the espers out there:

Tenser, said the Tensor.

Tenser, said the Tensor.

Tension, apprehension,

And dissension have begun.

1. Re: What's a tensor?

I can barely remember anything from that book but that ditty is burned into my mind for some reason and I think of it whenever I see the word 'tensor'.

1. Re: What's a tensor?

@ Frank ly

I can barely remember anything from that book

One thing I remember is that on my first reading of the book (many years ago) I thought the spelling of the characters' names was weird, bizarre, arty-farty bollocks and that nobody would ever do things like that in reality.

And then SMS came along to prove me completely wrong.

2. Re: What's a tensor?

+1 - exactly what I was going to post!

3. Cores?

With 5,120+2,560+640=8,320 cores of various types, the V100 is an 8 kilo-core chip.
But those aren't independent, each FP64 core consists of two FP32 cores.

1. Re: Cores?

640K cores should be enough for anyone

4. Very, very droolworthy pieces of kit

I could think of quite a few applications I would love to run on that kind of hardware.

5. DGX Station

..so at last, a system that probably can run Crysis then?

6. Weighing in at a robust 815mm2, oodles of flops of various kinds, and 300W, it’s big, it’s fast, and it’s hot

Hooray more half-filled racks :)

Seriously, it's getting to the point that datacentres are going to have to up their power and cooling if they have any real computational geeks as the servers/accelerators become so power hungry. My work's "high density" racks were optimised for blades in a corporate DC (think SAP, Oracle). We've just put in 100ish P100s and have had to spread them out in two racks.

1. "Hooray more half-filled racks"

5 years ago I was asked to name worst-case scenario heat generation for our server room and the facilities people visibly gulped when I said "Well, I can fairly trivially put 45kW in a single rack and it's only going to get worse"

The limiting factor isn't cooling capacity but how much power we can get onsite. They wrote a report saying I was asking for impossibly large numbers when I stuck my finger in the air and said "I need 50kW _now_ and the capacity for up to 250kW in future would be nice, but 100kW would probably do(*)" - sure enough about 6 months later someone filed a request for systems which would have maxxed out the 100kW in the first pass and the 250kW fairly easily. Thankfully he didn't get budget for it (I told him that doing this would require major infrastructure in addition to the rack-mounted kit). You have to love academics shooting for the moon.

(*) I have about 25kW in the room and maxxed-out cooling due to manglement rejecting my initial estimate of 75kW as "too high", with another 20kW in old computer rooms that was supposed to be migrated in, but can't be. Pointing out that even a very low 3kW/rack over 18 racks adds up to 52kW didn't go down well.

7. 815 sq mm?

I didn't know reticle sizes had reached that high!

8. We're doing linear algebra here. It's called a matrix.

Somehow, while I wasn't looking, the word "matrix", referring to N-dimensional arrays of scalars which are the subject of linear algebra, got renamed "tensor".

I find that a bit odd, since a tensor is a very specific use of the matrix by physics, a use which bears little or no relation to the use of matrices in the programming of neural networks.

I'll give you this, though: the word "tensor" is a lot cooler and edgier than the far more appropriate, but old and stodgy term, "matrix".

Source: someone who started studying linear algebra and physics around 1972.

POST COMMENT House rules

Not a member of The Register? Create a new account here.