Chapter 4 disassembles every byte of Q4_0 and looks at it. This chapter stands next to it, completing the complete genealogy of “simple block”: Q8_0, Q4_1, Q5_0, Q5_1. They have one thing in common –Each chunk is “a block of weights + one or two shared parameters”, 32 values packed into compact bytesno super-blocks, no sub-blocks, no two-level scaling.
You may ask: Is it still necessary to learn these “old formats”? Doesn’t K-quants surpass them all? The answer is: It is necessary, and there are three reasons.
First, Q8_0 is the “ruler” for quality verification. Its error is extremely small (PPL degradation is about +0.1%), so it is often used as a reference for “quantification correctness verification”: if a new format has abnormal errors under the Q8_0 reference, the problem is mostly in the format implementation rather than the quantization itself.
Second, Q4_1/Q5_0/Q5_1 is the vehicle for understanding “explicit minimum”. Q4_0 of q-8 It is the storage offset and does not occupy bytes; while Q4_1’s m It is really 2 bytes stored. Once you understand the difference between the two, you will understand what the trichotomy of “symmetric, asymmetric, pseudo-symmetric” in Chapter 1, Section 1.2 looks like at the byte level.
Third, simple blocks are still the optimal solution in certain scenarios. Minimal models, embedded extreme constraints, debugging, unit test benchmarks, old file compatibility – these scenarios require “the simplest implementation, the most complete back-end coverage, and the most predictable numerical behavior”, and this happens to be the strength of simple blocks.
Let’s unify a few terms first.Near-lossless: The quantified error is so small that it cannot be detected in most evaluation indicators.explicit min: Store the approximate value of the minimum value in the block directly into the file as a parameter, and add it back during dequantization.int8: 8-bit signed integer, value range -128 to +127.
5.1 Q8_0: Near lossless baseline
5.1.1 34-byte/32-value layout and (y_i = d \cdot q_i)
The structure of Q8_0 is almost an “enlarged version” of Q4_0: each 32-value block still shares an FP16 scale dbut this time each weight no longer uses 4 bits, but a complete 8-bit signed integer int8。
block_q8_0 的 34 字节布局（磁盘/内存顺序）
偏移 0 1 32 33
┌────────┬───────────────────────────────────────────┐
│ d │ qs[0..31] │
│ 2 字节 │ 32 字节 │
│ FP16 │ 32 个 int8 码（-128..+127） │
└────────┴───────────────────────────────────────────┘
总计 = 2 + 32 = 34 字节 / 32 个权重
有效位宽 = 34 × 8 / 32 = 8.5 bit/weight
The structure definitions in the three languages are as follows.
C++ (with llama.cpp ggml-common.h consistent):
#define QK8_0 32
typedef struct // j = 0..15
const uint8_t xh_0 = ((qh >> (j + 0)) << 4) & 0x10; // 权重 j 的高位：qh 的 bit j
const uint8_t xh_1 = ((qh >> (j + 12)) ) & 0x10; // 权重 j+16 的高位：qh 的 bit j+12
const int32_t x0 = ((x[i].qs[j] & 0x0F) block_q8_0;
static_assert(sizeof(block_q8_0) == sizeof(ggml_half) + QK8_0, "wrong q8_0 block size/padding");
C# (.NET 5+):
using System.Runtime.InteropServices;
public static class Q8_0
xh_0) - 16;
const int32_t x1 = ((x[i].qs[j] >> 4)
fixed sbyte qs[32] is a fixed size buffer in C#,sbyte Corresponding to C++ int8_t,need unsafe context and
Python：
QK8_0 = 32
def block_layout_q8_0() -> dict:
return xh_1) - 16;
y[i*qk + j + 0 ] = x0 * d;
y[i*qk + j + qk/2] = x1 * d;
# 总字节 = 2 + 32 = 34
Key difference: Why does Q8_0 not need offset reduction?
The inverse quantization of Q4_0 is (y_i = d \cdot (q_i – 8)), which needs to be subtracted by 8; the inverse quantization of Q8_0 is:
[
y_i = d \cdot q_i
]
The reason is that the code of Q4_0 isunsigned 4-bit（q ∈ [0,15]), negative numbers cannot be stored, and must be translated into an approximately symmetrical quantity by subtracting 8; and the code of Q8_0 isSigned you8（q ∈ [-128,127]), naturally covers positive and negative, and does not require any offset. Compare with pictures:
Q4_0 vs Q8_0：为什么一个有减偏移、一个没有
Q4_0: 无符号码 q ∈ [0, 15] Q8_0: 有符号码 q ∈ [-128, 127]
需要 q - 8 变对称 天然对称，直接 d·q
┌──────┬──────┐ ┌────────────────────────┐
│ 0..7 │ 8..15│ │ -128 ... 0 ... +127 │
│ 负半 │ 正半 │ │ 对称于 0 │
└──────┴──────┘ └────────────────────────┘
减 8 免费获得对称性 i8 本身就带符号
5.1.2 Quantitative end: how to choose the scale
llama.cpp quantize_row_q8_0_ref useThe value with the largest absolute value within the blockTo determine the scale:
void quantize_row_q8_0_ref(const float * x, block_q8_0 * y, int64_t k) {
assert(k % QK8_0 == 0);
const int nb = k / QK8_0;
for (int i = 0; i < nb; i++) {
float amax = 0.0f;
for (int j = 0; j < QK8_0; j++) {
const float v = x[i*QK8_0 + j];
amax = MAX(amax, fabsf(v)); // 只取绝对值，不管符号
}
const float d = amax / ((1 << 7) - 1); // 除以 127
const float id = d ? 1.0f/d : 0.0f;
y[i].d = GGML_FP32_TO_FP16(d);
for (int j = 0; j < QK8_0; ++j) {
const float x0 = x[i*QK8_0 + j]*id;
y[i].qs[j] = roundf(x0); // 四舍五入，不夹取
}
}
}
Note two differences from Q4_0:
- No symbols are preserved. Q4_0 takes
d = max / -8deliberately letdSigned to allow extreme values to hit code range endpoints; Q8_0 directlyd = amax / 127because the code range of int8 is symmetric,±amaxall fall on±127on, no symbolic skills required. - No clamping is required. The capacity of int8 far exceeds the actual range requirements of 8-bit code.
amax/127ensure|x·id| ≤ 127there will be no overflow after round. (Q4_0 milesmin(15, ...)That step is unnecessary here. )
C#：
public static BlockQ8_0[] QuantizeRow(float[] x)
{
int k = x.Length;
if (k % QK8_0 != 0) throw new ArgumentException("k must be multiple of 32");
var result = new BlockQ8_0[x.Length / QK8_0];
for (int i = 0; i < result.Length; i++)
{
float amax = 0f;
for (int j = 0; j < QK8_0; j++)
amax = Math.Max(amax, Math.Abs(x[i * QK8_0 + j]));
float d = amax / 127f;
float id = d != 0 ? 1f / d : 0f;
var block = new BlockQ8_0 { d = BitConverter.HalfToUInt16Bits((Half)d) };
unsafe
{
for (int j = 0; j < QK8_0; j++)
block.qs[j] = (sbyte)MathF.Round(x[i * QK8_0 + j] * id);
}
result[i] = block;
}
return result;
}
Python：
import struct
QK8_0 = 32
def quantize_row_q8_0(x: list[float]) -> bytes:
assert len(x) % QK8_0 == 0
out = bytearray()
for i in range(0, len(x), QK8_0):
block = x[i:i + QK8_0]
amax = max(abs(v) for v in block)
d = amax / 127.0
id_ = 1.0 / d if d != 0 else 0.0
out += struct.pack('5.1.3 Why Q8_0 is the reference for “Quantitative Correctness Verification”
Degradation of Q8_0 at 8.5 bpw on PPL is typically only about +0.1%so small that it is almost unnoticeable in actual use. This makes it a part of the entire quantitative systemanchor point：
- Implement verification: After writing the reference dequantization of any new format, the first step is to compare it with the same process of Q8_0. If the new format has abnormal errors under the Q8_0 reference (such as a much larger PPL increment), it is probably due to a bug in the format implementation (bit order, packaging), rather than a failure of the quantization principle.
- Error attribution: Q8_0 almost only leaves the error of “insufficient code bit width” (still close to negligible), so using it as the denominator can separate “format implementation bugs” and “loss of quantization itself”.
Chapter 18 will explain in detail how to make “reference implementation cross-validation” into a runnable workflow. Just remember the conclusion here:When encountering any suspicious format, first run the same process of Q8_0 as a reference。
5.2 Asymmetric variant: Q4_1 / Q5_1 with explicit zeros
5.2.1 Two-parameter layout of Q4_1
Q4_1 at the scale of Q4_0 d In addition, store an additional minimum value m. The inverse quantization formula becomes:
[
y_i = d \cdot q_i + m
]
The byte layout is increased from 18 bytes in Q4_0 to 20 bytes:
block_q4_1 的 20 字节布局
偏移 0 1 2 3 19
┌────────┬────────┬────────────────────────────────────────┐
│ d │ m │ qs[0..15] │
│ 2 字节 │ 2 字节 │ 16 字节 │
│ FP16 │ FP16 │ 32 个 4-bit 码，打包 │
└────────┴────────┴────────────────────────────────────────┘
总计 = 2 + 2 + 16 = 20 字节 / 32 个权重
有效位宽 = 20 × 8 / 32 = 5.0 bit/weight
C++：
#define QK4_1 32
typedef struct {
ggml_half d; // 尺度，2 字节
ggml_half m; // 最小值，2 字节
uint8_t qs[QK4_1 / 2]; // 16 字节，32 个 4-bit 码
} block_q4_1;
static_assert(sizeof(block_q4_1) == 2 * sizeof(ggml_half) + QK4_1 / 2, "wrong q4_1 block size/padding");
(in llama.cpp d and m use a consortium ggml_half2 dm Packed, but with equivalent layout:d in front,m After. )
C#：
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public unsafe struct BlockQ4_1
{
public ushort d; // FP16，2 字节
public ushort m; // FP16 最小值，2 字节
public fixed byte qs[QK4_1 / 2]; // 16 字节
}
Python：
QK4_1 = 32
def layout_q4_1() -> tuple[int, int, int]:
return (2, 2, QK4_1 // 2) # (d, m, qs)，共 20 字节
Quantitative side m How was it decided? Reference implementation quantize_row_q4_1_ref The approach is:
float min = FLT_MAX, max = -FLT_MAX;
for (int j = 0; j < qk; j++) {
const float v = x[i*qk + j];
if (v < min) min = v;
if (v > max) max = v;
}
const float d = (max - min) / ((1 << 4) - 1); // 15
const float id = d ? 1.0f/d : 0.0f;
y[i].d = GGML_FP32_TO_FP16(d);
y[i].m = GGML_FP32_TO_FP16(min); // 最小值直接存
// 码： (x - min) * id，再四舍五入
You can see Q4_1 m yesTrue intra-block minimumnot the fixed one of Q4_0 -8 offset. This is the “explicit zero point” mentioned in Section 1.2: itConsumes 2 real bytesin exchange for the ability to adapt to asymmetric distribution (the minimum value is far away from 0).
5.2.2 Q5_0 / Q5_1: First attempt at 5-bit
Before the emergence of K-quants, the most direct way to break through the quality bottleneck of 4.5 bpw was to upgrade the code from 4-bit to 5-bit. But 5-bit is not an integer multiple of bytes, how to store it? The answer is Q5_0’s “4-byte high plane” scheme: 32 5-bit codes, the lower 4 bits are packed in nibble (16 bytes), and the 5th bit is stored separately in 4 bytes qh。
block_q5_0 的 22 字节布局
偏移 0 1 2 5 21
┌────────┬────────┬────────┬──────────────────────────────────┐
│ d │ qh │ qs[0..15] │
│ 2 字节 │ 4 字节 │ 16 字节 │
│ FP16 │第5位平面│ 32 个 4-bit 低码，打包 │
└────────┴────────┴──────────────────────────────────────────┘
总计 = 2 + 4 + 16 = 22 字节 / 32 个权重
有效位宽 = 22 × 8 / 32 = 5.5 bit/weight
The high-endian order of Q5_0 (the most common place to write errors):
When decoding, the j A 4-bit low code (j = 0..15) corresponding weight j and j+16：
uint32_t qh; // 把 4 字节 qh 按小端读成 uint32
memcpy(&qh, x[i].qh, sizeof(qh));
for (int j = 0; j < qk/2; ++j) { // j = 0..15
const uint8_t xh_0 = ((qh >> (j + 0)) << 4) & 0x10; // 权重 j 的高位：qh 的 bit j
const uint8_t xh_1 = ((qh >> (j + 12)) ) & 0x10; // 权重 j+16 的高位：qh 的 bit j+12
const int32_t x0 = ((x[i].qs[j] & 0x0F) | xh_0) - 16;
const int32_t x1 = ((x[i].qs[j] >> 4) | xh_1) - 16;
y[i*qk + j + 0 ] = x0 * d;
y[i*qk + j + qk/2] = x1 * d;
}
Look at it bit by bit:
- The 5th bit of the low nibble (the 0x10 bit) comes from
qhof bit j(j = 0..15 corresponds to a low nibble with a weight of 0..15); - The 5th bit of the high nibble comes from
qhof bit j+12(j = 0..15 corresponds to high nibbles with weights 16..31).
The reason why is not bit j+16because the high nibble of each byte only requires 16 bits, and qh Bits 0..15 have been given to the high bits of the low nibble – the high bits of the high nibble start from bit 12 (j+12), so that 32 bits of 4 bytes are fully used.If you read according to “bit j and bit j+16” or “8 bits per byte” during implementation, the decoding will be wrong.。
Q5_1 adds an explicit minimum value to Q5_0 mthe layout becomes 24 bytes, 6.0 bpw:
block_q5_1 的 24 字节布局
偏移 0 1 2 3 4 7 23
┌────────┬────────┬────────┬────────┬────────┬──────────────────┐
│ d │ m │ qh │ qs[0..15] │
│ 2 字节 │ 2 字节 │ 4 字节 │ 16 字节 │
└────────┴────────┴────────┴──────────────────────────────────┘
总计 = 2 + 2 + 4 + 16 = 24 字节 / 32 个权重
有效位宽 = 24 × 8 / 32 = 6.0 bit/weight
公式 (y_i = d \cdot (q_i^5 – 16) + m)，-16 Translate the 5-bit unsigned code into a symmetrical quantity (0..31 → -16..+15), and then add back the minimum value m。
5.2.3 Why Q4_1 has not become mainstream
Theoretically, Q4_1 has more minimum parameters than Q4_0, which is more friendly to blocks with asymmetric distribution. But in practice it has not caught on for three reasons:
-
The price of 2 bytes is not small in the face of 4-bit. From 18 to 20 bytes, bpw increased from 4.5 to 5.0. The same number of bytes, later Q4_K (4.5 bpw) usessub-block minimumA finer-grained asymmetry is achieved, and the error improvement is much greater than the 2 bytes of Q4_1.
-
The explicit minimum is too coarse-grained for “32 values per block”. Within a 32-value block, the minimum value may fall locally rather than globally; Q4_1 can only record the “whole block minimum” and cannot distinguish the offsets of different areas within the block. K-quants cuts the 256 values into 8 sub-blocks, each sub-block has a minimum value, and the granularity is much finer.
-
ecological inertia. Q4_1 is an early format. After the emergence of Q4_K, it became the de facto standard (Chapter 7). Most of the files quantified by the community are in the Q4_K series, and Q4_1 only remains in the compatibility list.
One sentence summary:The idea of explicit zero points is correct, but the granularity of “one m for the whole block” is not enough. Q4_K uses sub-block scale + sub-block minimum value to refine the same idea. This is the motivation for Chapter 6 K-quants.
5.3 Simple block quantification of family residual value
After talking about “why it was surpassed”, we still need to talk about “why it is still useful”. Simple blocks remain irreplaceable in four scenarios.
5.3.1 Minimal model and debugging scenarios
- minimal model: When the model has only a few million parameters, the proportion of scale overhead becomes important, but more importantlyImplementation must not have bugs. Q8_0/Q4_0 has the simplest structure and the lowest probability of bit sequence errors, making it suitable as a starting point for embedded/microcontrollers.
- debug: When you suspect that there is a problem with the decoding of a certain K-quant or IQ format, the fastest troubleshooting method is toRun Q4_0/Q8_0 on the same tensor——Their decoding logic is simple. If they are normal but K-quant is abnormal, the problem lies in the packing/bit order of K-quant, not the entire quantization system.
5.3.2 Compatibility and certainty
- The most complete coverage across backends: Q4_0/Q8_0 is the earliest and most stable format implemented by all llama.cpp backends (CPU, Metal, CUDA, Vulkan, OpenCL…). A model that has only been verified on Q8_0/Q4_0, but can run on almost any build and any device.
- Numerical behavior is predictable:Q4_0 of
y = d(q-8)Q8_0’sy = dqThere are no conditional branches, no table lookups, and no interleaved bit streams. The inverse quantization result is onlydand code decisions – this makes unit testing and cross-language cross-validation easy (reference implementations can be directly compared byte-for-byte). - Old file compatibility: A large number of historical GGUF files are Q4_0/Q8_0 quantization, and the new version of llama.cpp always needs to read them correctly.
The “imperfection” of simple blocks is precisely the starting point for the design motivation of K-quants: when 32-value blocks share a single scale, intra-block heterogeneity will concentrate errors to small values (Chapter 4, Section 4.5); when lower bit width or higher fidelity is required, simple blocks can neither be pressed (4-bit at the end) nor maintained (8-bit is too expensive). The solution to these two ends is the super block architecture starting in Chapter 6.
Summary of this chapter and the bridge to “Super Block”
This chapter can be condensed into four facts.
First, Q8_0 is a nearly lossless 8-bit baseline:d + 32 int8, 34 bytes = 8.5 bpw, formula y = d·q。 Because int8 has its own sign, there is no need to reduce the offset; the quantization end is used amax/127 Set the scale without clamping. PPL degradation is about +0.1%, which is the “ruler” for all format verifications.
Second, Q4_1 introduces an explicit minimum:d + m + 16 bytecode, 20 bytes = 5.0 bpw, formula y = d·q + m。 m is the true intra-block minimum, consuming 2 real bytes – this is consistent with Q4_0’s q-8(Storage offset, not occupying bytes) are two different things.
Third, Q5_0/Q5_1 uses the “high bit plane” to push 4-bit to 5-bit: 22/24 bytes = 5.5/6.0 bpw. The 5th bit is stored centrally qhlow nibble configuration qh bit j, high nibble matching qh bit j+12 – This is the most common place to make mistakes.
Fourth, the remaining value of simple blocks lies in “certainty” and “compatibility”. It has the simplest implementation, the most comprehensive backend coverage, and the most predictable numerical behavior, so it remains the first choice for minimal models, debugging, test benchmarks, and legacy file compatibility.
Together, these four facts point to a key conclusion:The two shortcomings of simple blocks – the problem of heterogeneity at a single scale of 32 values, and the need to sacrifice bytes to accommodate distribution asymmetry – come from the same source: each block has only one “global” parameter. If you add a layer of structure to the block and let “global” and “local” be managed separately, these two problems can be alleviated at the same time.
The next chapter will enter this new structure.The super-block groups 256 weights into a group and shares two global scales d and d_min; Internally cut into 8 32-value sub-blocks, each sub-block has its own 6-bit scale and 6-bit minimum value. This is K-quants – from this chapter on,block_q4_0 The 18 bytes will become block_q4_K 144 bytes, and the answer to “Why are these extra bytes worth spending?” is hidden in Section 6.3 of the next chapter.
