WEBVTT

00:00:00.046 --> 00:00:02.798
Zero percent. A hundred percent. Which

00:00:02.856 --> 00:00:05.422
GPU is really busy? Some legends

00:00:05.480 --> 00:00:07.396
live in data centres too. Just

00:00:07.442 --> 00:00:09.044
because you believe something doesn't make

00:00:09.079 --> 00:00:09.636
it true.

00:00:12.058 --> 00:00:13.834
To carve up this GPU, must

00:00:13.881 --> 00:00:15.773
we reboot the server? What would

00:00:15.820 --> 00:00:16.365
you guess?

00:00:18.031 --> 00:00:20.005
In our test, no. Every layout

00:00:20.063 --> 00:00:21.247
step took less than a tenth

00:00:21.282 --> 00:00:21.943
of a second.

00:00:22.444 --> 00:00:24.638
The layout changed. Restarting the model

00:00:24.708 --> 00:00:26.089
servers was the slow part.

00:00:28.046 --> 00:00:29.869
Two halves instead of one whole.

00:00:30.310 --> 00:00:32.017
More throughput? And would a quarter

00:00:32.063 --> 00:00:33.828
slice make one user four times

00:00:33.875 --> 00:00:34.362
slower?

00:00:36.053 --> 00:00:37.644
In our test, two halves won

00:00:37.690 --> 00:00:39.246
at light load, then lost when

00:00:39.304 --> 00:00:42.125
busy. Single-user speed barely changed.

00:00:42.597 --> 00:00:45.638
The lone user was overhead-bound. Combined

00:00:45.697 --> 00:00:48.564
capacity follows the slice's share. Slicing

00:00:48.611 --> 00:00:49.946
doesn't create capacity.

00:00:51.055 --> 00:00:53.170
Flood the neighbour. Does MIG protect

00:00:53.213 --> 00:00:54.929
our probe? And without the wall,

00:00:54.985 --> 00:00:56.058
does sharing wreck it?

00:00:57.668 --> 00:00:59.616
In our small test, neither layout

00:00:59.672 --> 00:01:02.294
showed probe slowdown. Yet with MIG,

00:01:02.383 --> 00:01:04.065
whole-card power rose from about a

00:01:04.132 --> 00:01:05.902
hundred and fifty-four to three hundred

00:01:05.935 --> 00:01:06.799
and one watts.

00:01:07.201 --> 00:01:09.159
Separate slices still share one power

00:01:09.215 --> 00:01:11.661
and clock budget. No slowdown here

00:01:11.727 --> 00:01:13.631
isn't proof of zero interference.

00:01:14.070 --> 00:01:16.775
Can this twenty-seven-billion-parameter model pass through

00:01:16.810 --> 00:01:18.098
that quarter-slice door?

00:01:19.770 --> 00:01:21.430
In our test, Qwen loaded and

00:01:21.500 --> 00:01:22.579
served a request.

00:01:23.074 --> 00:01:25.802
Four-bit weights fit. The serving configuration

00:01:25.860 --> 00:01:26.801
had to fit too.

00:01:31.046 --> 00:01:33.786
Our haunted readings return. Zero percent

00:01:33.844 --> 00:01:35.679
or a hundred percent: which tells

00:01:35.725 --> 00:01:37.118
you how busy the tensor cores

00:01:37.165 --> 00:01:37.478
are?

00:01:39.368 --> 00:01:42.282
Under MIG, nvidia-smi read zero while

00:01:42.328 --> 00:01:44.755
GPM showed the SMs active sixty-four

00:01:44.801 --> 00:01:45.846
percent of the time.

00:01:46.306 --> 00:01:47.711
In a different test, a hundred

00:01:47.745 --> 00:01:50.300
percent utilisation still left tensor cores

00:01:50.381 --> 00:01:52.262
idle about eighty-six percent of the

00:01:52.308 --> 00:01:52.738
time.

00:01:53.192 --> 00:01:55.154
One counter missed activity. The other

00:01:55.200 --> 00:01:57.975
wasn't tensor saturation. Decoding was waiting

00:01:58.022 --> 00:01:59.113
on memory bandwidth.

00:02:00.058 --> 00:02:01.741
Can draft tokens turn that spare

00:02:01.776 --> 00:02:03.425
compute into free speed at every

00:02:03.483 --> 00:02:03.878
load?

00:02:05.558 --> 00:02:07.369
Two draft tokens gave one user

00:02:07.427 --> 00:02:09.482
nearly twice the output. At two

00:02:09.540 --> 00:02:11.584
hundred and fifty-six users, the gain

00:02:11.642 --> 00:02:12.361
disappeared.

00:02:12.797 --> 00:02:14.934
Spare compute helps. A full batch

00:02:15.003 --> 00:02:16.362
pays for the speculation.

00:02:18.035 --> 00:02:20.786
And FP8 KV? More capacity, almost

00:02:20.856 --> 00:02:21.936
unchanged output.

00:02:23.060 --> 00:02:24.689
How much faster is one B200

00:02:24.748 --> 00:02:27.151
than one RTX PRO6000? Would you

00:02:27.211 --> 00:02:28.581
use the same multiplier for every

00:02:28.621 --> 00:02:28.959
load?

00:02:30.623 --> 00:02:32.411
The B200's lead grew with load:

00:02:32.620 --> 00:02:33.901
two point five six times at

00:02:33.951 --> 00:02:35.579
one user, four point one five

00:02:35.608 --> 00:02:36.860
at two hundred and fifty-six.

00:02:37.260 --> 00:02:38.969
A GPU comparison is a workload

00:02:39.018 --> 00:02:39.604
comparison.

00:02:40.046 --> 00:02:41.370
Then surely a bigger model is

00:02:41.416 --> 00:02:41.916
slower?

00:02:43.534 --> 00:02:45.717
Not this one. Nemotron's thirty billion

00:02:45.775 --> 00:02:47.935
parameters include only three billion active

00:02:47.981 --> 00:02:49.932
per token. It served one user

00:02:49.978 --> 00:02:52.288
faster than dense Qwen. Count what's

00:02:52.358 --> 00:02:54.100
read, not just what's stored.

00:02:58.046 --> 00:03:00.287
Some busted. Some dependent on load.

00:03:00.612 --> 00:03:02.725
For your next GPU decision: measure

00:03:02.760 --> 00:03:04.304
the workload, not the legend.

00:03:05.058 --> 00:03:06.858
Just because you believe something doesn't

00:03:06.904 --> 00:03:08.773
make it true. Just because you

00:03:08.843 --> 00:03:10.573
disagree with something doesn't make it

00:03:10.608 --> 00:03:12.500
wrong. If anything, it might make

00:03:12.535 --> 00:03:13.882
it more likely to be right.

00:03:18.035 --> 00:03:20.949
Metrum AI. Next episode: AI agents

00:03:20.984 --> 00:03:22.087
and CPU cores.
