WEBVTT

1
00:00:01.235 --> 00:00:05.333
On this desk-side system, a 122-billion-parameter model

2
00:00:05.542 --> 00:00:08.816
answered faster than a 27-billion-parameter model.

3
00:00:09.420 --> 00:00:11.312
How can the bigger model be faster?

4
00:00:13.953 --> 00:00:16.055
The larger model produced nineteen point one

5
00:00:16.101 --> 00:00:17.309
tokens per second.

6
00:00:17.692 --> 00:00:19.991
The smaller one produced eleven point three.

7
00:00:20.432 --> 00:00:24.751
One user, same system, llama.cpp on ROCm.

8
00:00:26.030 --> 00:00:28.305
If your team has sixteen people, start

9
00:00:28.340 --> 00:00:31.034
with llama.cpp on ROCm.

10
00:00:31.545 --> 00:00:36.212
GPT-OSS-20B delivered one hundred fifty-three total tokens

11
00:00:36.270 --> 00:00:36.978
per second.

12
00:00:37.489 --> 00:00:42.237
Qwen3.6-35B delivered one hundred thirty-one, with stronger

13
00:00:42.284 --> 00:00:42.934
quality.

14
00:00:43.375 --> 00:00:45.035
But why that answer is the interesting

15
00:00:45.105 --> 00:00:45.581
part.

16
00:00:46.882 --> 00:00:48.496
You might expect a bigger model to

17
00:00:48.565 --> 00:00:49.262
be slower.

18
00:00:49.541 --> 00:00:51.375
So what is the machine actually doing

19
00:00:51.468 --> 00:00:52.861
when it produces a word?

20
00:00:53.674 --> 00:00:56.205
Pause and guess: does it use every

21
00:00:56.251 --> 00:00:57.621
parameter, every time?

22
00:01:00.262 --> 00:01:01.075
Not always.

23
00:01:01.586 --> 00:01:03.815
The larger Qwen model uses a mixture

24
00:01:03.850 --> 00:01:04.674
of experts.

25
00:01:04.953 --> 00:01:06.636
Think of a team of specialists.

26
00:01:07.147 --> 00:01:09.260
For each token, the model selects only

27
00:01:09.318 --> 00:01:09.957
some of them.

28
00:01:10.235 --> 00:01:12.220
Its name says one hundred twenty-two B

29
00:01:12.371 --> 00:01:14.473
in total, but about ten B are

30
00:01:14.519 --> 00:01:15.727
active for a token.

31
00:01:16.423 --> 00:01:18.409
To generate a token, a system often

32
00:01:18.455 --> 00:01:19.941
has to move weights from memory to

33
00:01:19.999 --> 00:01:20.823
the processor.

34
00:01:21.264 --> 00:01:23.157
At low load, moving those weights can

35
00:01:23.203 --> 00:01:24.202
be the constraint.

36
00:01:24.585 --> 00:01:26.558
A sparse model can use fewer active

37
00:01:26.605 --> 00:01:28.660
weights for each token than its total

38
00:01:28.729 --> 00:01:29.832
size suggests.

39
00:01:30.274 --> 00:01:32.642
That helps explain why total parameter count

40
00:01:32.688 --> 00:01:34.627
alone is a poor speed forecast.

41
00:01:35.069 --> 00:01:36.903
The model design and the runtime matter,

42
00:01:36.926 --> 00:01:37.391
too.

43
00:01:37.901 --> 00:01:39.364
For your team, the label on the

44
00:01:39.411 --> 00:01:40.723
box is not the answer.

45
00:01:41.106 --> 00:01:42.905
Measure the model you intend to run.

46
00:01:44.185 --> 00:01:46.832
Now imagine sixteen people arriving at once.

47
00:01:47.111 --> 00:01:48.028
Same system.

48
00:01:48.202 --> 00:01:49.293
Same model file.

49
00:01:49.502 --> 00:01:51.453
Which setup serves more output for the

50
00:01:51.488 --> 00:01:52.266
whole team?

51
00:01:52.707 --> 00:01:55.958
Pause and guess: ROCm or Vulkan?

52
00:01:58.367 --> 00:02:03.905
For GPT-OSS-20B, llama.cpp on ROCm delivered

53
00:02:03.963 --> 00:02:06.551
one hundred fifty-three total tokens per second.

54
00:02:06.830 --> 00:02:09.094
The Vulkan path delivered one hundred eleven.

55
00:02:09.698 --> 00:02:11.462
For one user they were close.

56
00:02:11.846 --> 00:02:13.750
At sixteen, the gap is visible.

57
00:02:13.959 --> 00:02:16.014
A server can batch requests together.

58
00:02:16.188 --> 00:02:18.022
That lets one pass over weights help

59
00:02:18.092 --> 00:02:19.950
serve several active requests.

60
00:02:20.333 --> 00:02:23.038
Continuous batching can admit waiting requests as

61
00:02:23.108 --> 00:02:23.944
others finish.

62
00:02:24.454 --> 00:02:26.219
Now look at a different model family.

63
00:02:26.463 --> 00:02:28.901
In the tested Ollama version, Qwen requests

64
00:02:28.982 --> 00:02:30.143
run one at a time.

65
00:02:30.585 --> 00:02:32.663
Sixteen people can still ask, but they

66
00:02:32.721 --> 00:02:33.557
form a queue.

67
00:02:33.998 --> 00:02:38.212
With Qwen3.8-27B at sixteen offered users, the

68
00:02:38.305 --> 00:02:41.254
ROCm path delivered sixty-three total tokens per

69
00:02:41.301 --> 00:02:41.823
second.

70
00:02:42.264 --> 00:02:44.017
Ollama delivered ten point eight.

71
00:02:44.528 --> 00:02:46.351
The queue is the practical difference.

72
00:02:46.676 --> 00:02:48.487
A quick answer for one person does

73
00:02:48.534 --> 00:02:50.554
not guarantee a responsive service for a

74
00:02:50.589 --> 00:02:51.355
whole team.

75
00:02:51.796 --> 00:02:53.712
Test at your actual concurrency.

76
00:02:55.060 --> 00:02:57.602
Would an NVIDIA GB10 box be faster?

77
00:02:57.776 --> 00:02:58.786
Pause and guess.

78
00:03:01.497 --> 00:03:05.143
We ran the same Qwen3.6-35B model file,

79
00:03:05.294 --> 00:03:08.150
with the same llama.cpp runtime, on both

80
00:03:08.208 --> 00:03:08.904
machines.

81
00:03:09.346 --> 00:03:11.877
For one user, Strix Halo delivered fifty-three

82
00:03:11.923 --> 00:03:13.049
tokens per second.

83
00:03:13.374 --> 00:03:15.092
GB10 delivered seventy.

84
00:03:15.301 --> 00:03:17.275
About one point three times faster.

85
00:03:17.716 --> 00:03:19.887
At sixteen users, they were about level,

86
00:03:19.910 --> 00:03:22.070
around one hundred fifty total tokens per

87
00:03:22.128 --> 00:03:23.637
second for chat and code.

88
00:03:24.078 --> 00:03:25.889
Quality was equal within noise.

89
00:03:26.400 --> 00:03:28.420
The roughly three-times gap came from

90
00:03:28.420 --> 00:03:32.612
GB10's NVFP4 format and vLLM engine, not

91
00:03:32.658 --> 00:03:34.191
this like-for-like setup.

92
00:03:34.701 --> 00:03:37.163
Software sets speed more than silicon here.

93
00:03:37.372 --> 00:03:39.183
Compare the whole stack, not just the

94
00:03:39.241 --> 00:03:39.647
chip.

95
00:03:40.938 --> 00:03:42.830
Suppose the fastest setup is easy to

96
00:03:42.888 --> 00:03:43.434
spot.

97
00:03:43.875 --> 00:03:45.222
Does that make it the best answer

98
00:03:45.268 --> 00:03:45.976
for your work?

99
00:03:46.092 --> 00:03:49.436
Pause and choose: speed, or chat quality?

100
00:03:52.019 --> 00:03:55.642
GPT-OSS-20B is the throughput leader here.

101
00:03:55.851 --> 00:03:57.673
On the chat test it scored nineteen

102
00:03:57.731 --> 00:03:58.579
out of fifty.

103
00:03:59.020 --> 00:04:01.470
Qwen three point six scored thirty-two.

104
00:04:02.073 --> 00:04:05.672
Gemma 4 31B led chat quality at forty-two out

105
00:04:05.719 --> 00:04:08.563
of fifty, while delivering sixty-six point seven

106
00:04:08.621 --> 00:04:11.361
total tokens per second to sixteen users.

107
00:04:12.058 --> 00:04:13.835
They show a trade-off between one measured

108
00:04:13.870 --> 00:04:15.611
chat score and team throughput.

109
00:04:16.122 --> 00:04:17.829
Your use case decides where you want

110
00:04:17.863 --> 00:04:18.421
to sit.

111
00:04:18.804 --> 00:04:20.859
Choose the fast point when throughput is

112
00:04:20.894 --> 00:04:21.880
your priority.

113
00:04:22.322 --> 00:04:24.144
Move toward Gemma when the quality of

114
00:04:24.202 --> 00:04:25.909
chat answers matters more.

115
00:04:26.350 --> 00:04:28.127
Qwen is a balance between the two.

116
00:04:29.358 --> 00:04:30.589
Now a different question.

117
00:04:30.763 --> 00:04:32.156
How much output do you get for

118
00:04:32.191 --> 00:04:33.816
the energy used at the socket?

119
00:04:34.257 --> 00:04:35.778
Pause and guess which of the two

120
00:04:35.860 --> 00:04:38.507
fast picks makes more tokens per joule.

121
00:04:41.310 --> 00:04:45.304
At thirty-two users, GPT-OSS-20B was both the

122
00:04:45.362 --> 00:04:47.081
fastest and the most efficient of the

123
00:04:47.115 --> 00:04:48.195
two fast picks.

124
00:04:48.578 --> 00:04:50.540
It produced four point seven tokens per

125
00:04:50.622 --> 00:04:51.121
joule.

126
00:04:51.562 --> 00:04:53.629
Qwen three point six produced three point

127
00:04:53.687 --> 00:04:54.221
three.

128
00:04:55.033 --> 00:04:56.705
It's a way to see output relative

129
00:04:56.740 --> 00:04:58.865
to the power of the system-on-chip socket.

130
00:04:59.306 --> 00:05:01.292
The same stream of generated tokens is

131
00:05:01.350 --> 00:05:03.347
now being viewed through a different lens:

132
00:05:03.788 --> 00:05:05.912
throughput divided by socket power.

133
00:05:06.296 --> 00:05:08.594
More tokens per joule means more output

134
00:05:08.652 --> 00:05:10.347
for that measured socket energy.

135
00:05:10.951 --> 00:05:13.053
Match the efficiency number to the workload

136
00:05:13.227 --> 00:05:15.015
and load you actually expect.

137
00:05:15.456 --> 00:05:17.778
A fast result and an efficient result

138
00:05:17.952 --> 00:05:19.810
can point to different choices for different

139
00:05:19.845 --> 00:05:20.448
teams.

140
00:05:21.738 --> 00:05:22.829
You have the trade-offs.

141
00:05:23.038 --> 00:05:24.083
What would you put in front of

142
00:05:24.130 --> 00:05:24.722
your team?

143
00:05:27.433 --> 00:05:30.591
For speed, choose GPT-OSS-20B.

144
00:05:30.916 --> 00:05:32.669
For a balance, look at Qwen three

145
00:05:32.727 --> 00:05:33.528
point six.

146
00:05:34.039 --> 00:05:36.326
For chat quality, try Gemma four.

147
00:05:36.535 --> 00:05:38.485
And if model size is the priority,

148
00:05:38.729 --> 00:05:43.210
Qwen3.5-122B fits on this system and delivered

149
00:05:43.257 --> 00:05:45.846
fifty-seven total tokens per second at sixteen

150
00:05:45.927 --> 00:05:46.542
users.

151
00:05:46.983 --> 00:05:48.307
These are starting points.

152
00:05:48.748 --> 00:05:50.037
The best choice is the one that

153
00:05:50.072 --> 00:05:51.674
meets your team's needs on its own

154
00:05:51.732 --> 00:05:52.463
tasks.

155
00:05:52.904 --> 00:05:55.377
Beyond the desk, Dell PowerEdge systems are

156
00:05:55.400 --> 00:05:58.013
available with AMD Instinct accelerators.

157
00:05:58.524 --> 00:06:01.380
Also from Metrum AI, Metrum AI Router

158
00:06:01.438 --> 00:06:02.401
is open source.

159
00:06:03.669 --> 00:06:07.570
Metrum AI Bench-CLI measured speed, promptfoo assessed

160
00:06:07.628 --> 00:06:10.333
quality, and all-smi recorded socket power.

161
00:06:10.774 --> 00:06:12.864
Knowing this, what would your team run?
