WEBVTT

1
00:00:00.000 --> 00:00:03.456
Are you benchmarking and deploying your on-prem AI models correctly?

2
00:00:03.466 --> 00:00:05.483
Or do you only believe you do?

3
00:00:05.493 --> 00:00:07.744
Buying GPUs you think are full?

4
00:00:07.754 --> 00:00:09.154
Scheduling a reboot?

5
00:00:09.245 --> 00:00:11.828
New observations challenged Earth-centred astronomy.

6
00:00:12.045 --> 00:00:14.069
Medical handwashing met resistance.

7
00:00:14.200 --> 00:00:16.325
We believed some GPU rules too.

8
00:00:16.335 --> 00:00:18.966
Then we benched a Dell PowerEdge R770.

9
00:00:19.231 --> 00:00:23.401
We measured it with Metrum AI Bench-CLI, our open-source benchmark tool.

10
00:00:23.666 --> 00:00:25.990
Can zero and a hundred both mislead?

11
00:00:26.000 --> 00:00:28.307
Changing hardware partitions sounds disruptive.

12
00:00:28.405 --> 00:00:30.105
Must the host reboot?

13
00:00:32.699 --> 00:00:34.220
In our test, no.

14
00:00:34.230 --> 00:00:36.895
Each layout step took under a tenth of a second.

15
00:00:36.905 --> 00:00:38.605
Model restarts were separate.

16
00:00:39.000 --> 00:00:40.615
Two lanes sound faster.

17
00:00:40.625 --> 00:00:42.325
Predict the busy case.

18
00:00:44.748 --> 00:00:48.998
In our test, two halves helped at light load, then lost at high load.

19
00:00:49.008 --> 00:00:51.608
A lone request barely noticed the slice.

20
00:00:51.782 --> 00:00:54.646
Single-user speed is not combined capacity.

21
00:00:57.000 --> 00:00:58.615
A wall feels safer.

22
00:00:58.625 --> 00:01:01.245
Flood the neighbour: which probe slows?

23
00:01:04.380 --> 00:01:06.680
In our small test, neither did.

24
00:01:06.702 --> 00:01:09.902
But partitions still share one power and clock budget.

25
00:01:10.208 --> 00:01:13.316
No slowdown here is not a universal guarantee.

26
00:01:15.000 --> 00:01:16.946
Twenty-seven billion parameters.

27
00:01:17.055 --> 00:01:18.160
A quarter slice.

28
00:01:18.170 --> 00:01:19.570
Does it fit?

29
00:01:22.087 --> 00:01:25.972
In our test, yes—with four-bit weights and a short context.

30
00:01:26.476 --> 00:01:30.276
We had to fit the serving configuration, not just the model.

31
00:01:31.000 --> 00:01:33.004
Those zero and hundred readings.

32
00:01:33.183 --> 00:01:36.083
Surely one tells us the GPU is full?

33
00:01:38.962 --> 00:01:41.848
Under MIG, zero hid active SMs.

34
00:01:42.074 --> 00:01:46.179
In a different test, a hundred still left tensor cores mostly idle.

35
00:01:46.520 --> 00:01:48.820
The memory pipe was the bottleneck.

36
00:01:48.865 --> 00:01:50.565
Read what is waiting.

37
00:01:55.000 --> 00:01:56.700
Extra guesses sound cheap.

38
00:01:56.853 --> 00:01:59.453
Are they free in a full batch?

39
00:02:01.818 --> 00:02:04.201
Spare seats help; full buses pay.

40
00:02:04.329 --> 00:02:07.354
Two draft tokens nearly doubled one-user output.

41
00:02:07.818 --> 00:02:12.583
At 256 users, the gain disappeared—and first-token tails grew.

42
00:02:13.000 --> 00:02:14.284
FP8 KV?

43
00:02:14.358 --> 00:02:15.458
More capacity.

44
00:02:15.508 --> 00:02:17.990
Almost no speed gain in this test.

45
00:02:18.000 --> 00:02:20.173
One multiplier makes sizing easy.

46
00:02:20.183 --> 00:02:22.483
Does it survive a load sweep?

47
00:02:24.572 --> 00:02:29.584
The B200 lead ranged from 2.45 to 4.15 times.

48
00:02:29.926 --> 00:02:33.148
Compare the workload, not a fixed hardware promise.

49
00:02:34.000 --> 00:02:36.042
More parameters, more work?

50
00:02:39.286 --> 00:02:43.956
The thirty-billion model used only three billion per token—and ran faster.

51
00:02:44.358 --> 00:02:47.364
Measure what is read, not just what is stored.

52
00:02:47.374 --> 00:02:49.691
Then measure quality beside speed.

53
00:02:50.000 --> 00:02:52.330
Like the history we opened with, test the belief.

54
00:02:52.340 --> 00:02:53.475
Sweep concurrency.

55
00:02:54.028 --> 00:02:56.028
Read GPM and memory bandwidth.

56
00:02:56.169 --> 00:02:58.372
Flood the neighbour in your real layout.

57
00:02:58.382 --> 00:02:59.862
Report per-load ratios.

58
00:03:00.070 --> 00:03:01.654
Measure quality beside speed.

59
00:03:01.664 --> 00:03:04.785
Configure concurrency and output-length checks in Bench-CLI.

60
00:03:05.170 --> 00:03:08.711
For deployment, Metrum AI Router routes requests among configured models.

61
00:03:09.000 --> 00:03:12.200
Just because you believe something doesn't make it true.

62
00:03:12.239 --> 00:03:15.724
Just because you disagree with something doesn't make it wrong.

63
00:03:15.734 --> 00:03:19.534
If anything, it might make it more likely to be right.
