Redis 数据类型全解 – HyperLogLog：12 KB 估出亿级 UV（八） – IT规划师

Applicable: It is necessary to implement UV/duplication counting, capacity planning or troubleshooting backend and operation and maintenance of online problems in the production environment.

This article only answers one thing: can this 12 KB estimate fit into your production link, and where are the boundaries. The reading path is “is it worth it → what does it look like → how to write it → how to implement it → how not to be beaten to death → pitfalls that have been stepped on → online verification”, which does not involve estimator derivation and source code step-by-step reading.

1.1 0.81% What does it mean?

标准误差 = 1.04 / sqrt(m) = 1.04 / sqrt(16384) ≈ 0.008125，即 0.81%

If one of the three qualifications is missing, it will be misused.

1. Relative error, not absolute error: The base number of 100,000 is about ±810, and the base number of 100,000 is about ±810,000. A difference of several hundred thousand on the kanban board is not a bug.
2. It is 1 sigma, not the upper limit: About 68% falls within ±0.81%, 95% falls within ±1.62%, 99.7% falls within ±2.43% – about 32% of the probability will exceed ±0.81%, and occasionally ±1.5% is normal.
3. Small cardinality is not applicable: when it comes to hundreds to thousands of times, the estimator approximates linear counting, and the deviation is often single digits. Don’t use the 0.81% inverse interval. The estimated value of the same fixed set is certain, and there is no such thing as “checking it several times and taking the average to make it more accurate”.

What should be asked in engineering is not “Is the error big?” butWill the error cause an action to cross the threshold?: The kanban difference is acceptable if it is hundreds of thousands; if the fuse threshold is 500 and the actual 498 is read as 502, it will be accidentally fuse.The only thing that can change the accuracy is the number of barrels mit is hard-coded at compile time。

Going back to the question at the beginning: The only requirement that is really worth going to HLL is “a massive deduplication total that can tolerate an error of about 1%”; if you want accuracy, detail, intersection, and judgment, change the structure in 1.4.

1.2 The third generation solution and a designed transaction

Set / ZSet The ID is stored directly, and the memory increases linearly with the base, from hundreds of millions to GB.Bitmap To map ID to offset, the memory depends on the maximum ID rather than the base. The premise is that dense non-negative bounded integers – UUID and snowflake ID are directly invalid.

HLL is the third way: it does not store members, only statistical summaries, and is capped at about 12 KB when dense; the price is about 0.81% error and four “can’ts” of 1.3. Its advantage over Bitmap is not that it is “more economical”, but that it is not picky about the input form.

This deal was designed: Error 1.04/sqrt(16384) and memory 16384 * 6 / 8 + 16 = 12304 Bytes, from the same set of registers, that is, the same constant P = 14。HLL_P Unable to configure, you need to change the source code to change the configuration (P = 16 About 0.41% / 48 KB); the project either accepts “12 KB / 0.81%” or changes the structure.

1.3 The four cannots and when not to use them

PFADD The elements are not saved, only the maximum value of the register is updated after hashing. It follows from this:

In one sentence:HLL only answers “how many”, not “which ones” or “is this one”.

1.4 Compared with other four structures

Bloom that 114 MB by m = -n*ln(p)/(ln2)^2 It is estimated that writing “10 MB” is a common order of magnitude error.

Selection order: accurate/detailed/save/intersect/delete single elements → Set or Bitmap; as long as “presence or absence” and tolerance of misjudgment → Bloom; “how many times appears” → CMS; the base number is only a few hundred → small Set; the rest only need the total number → HLL. Do not use inclusion and exclusion for intersection: each of the three quantities has an error of about 0.81%. When the intersection is much smaller than the union, the error will be greater than the intersection itself. Use SINTER / BITOP AND。

1.5 version differences at a glance

Before reading the HLL information, first confirm which version it is written in; the conclusion of the unmarked version holds true for 2.8.9 so far.

Two more points. The first two generations have three commands, dual encoding and hll-sparse-max-bytes，PFCOUNT Single keys will be written back to the cache; the ACL in 8.0 is a destructive change.@read / @write Probabilistic structure commands are henceforth included;PFCOUNT The whole strip is @read @hyperloglog @slow. See version 5.5 for security fixes.

The “untested” grid is estimated according to the source code structure: run once PFMERGE dest src Look again STRLEN dest You can judge by yourself – 12304 is dense, tens to several kilobytes is sparse. The output of the three generations of estimators is almost the same (the new implementation is about 20% faster),Changing the version does not change “whether to use it or not”, only the actual deviation of the small base is changed.。

2.1 How does one hash become 16384 registers?

Hash the elements into a uniformly random 64-bit string, counting from the low bit the length of “consecutive 0s until the first 1” rho。rho = 20 means that this bucket has been fed approximately 2^20 Magnitude elements. Only the global maximum variance is too large, so after bucketing, the harmonic average of 16384 registers is taken (not sensitive to extreme values), and the error is reduced to 1.04/sqrt(m)。

元素（任意字符串）
  |  MurmurHash64A（64 bit，seed 0xadc83b19，与字节序无关）
  v
64 位哈希值
  |-- 低 14 位（bit 0-13）: index = hash & HLL_P_MASK，桶编号 0-16383
  |-- 高 50 位（bit 14-63）: 右移后找 000...1 模式长度 rho，范围 1-51
  v
M[index] = max(M[index], rho)      只留最大值，天然去重、只增不减
  v
16384 个寄存器的直方图 + 估计器 -> 基数估计 E

A lot of information says “the first 14 digits are fixed bucket, the last 50 digits are leading zeros”, but the direction is reversed: source code hllPatLen() It is the lower 14-bit fixed bucket,hash >>= HLL_P Then use the remaining high positions to find 000...1;The hash is MurmurHash64A, not MurmurHash3.

Officially, 64-bit hashing brings the base “closer to 2^64 There were no practical restrictions before”, butThe available quantum range determined by hash collisions (birthday paradox) is much lower than 2^64。1.04 Nor the fitting coefficients: asymptotic constants α∞ ≈ 0.7213(source code HLL_ALPHA_INF) launched.

2.2 Where does 12 KB come from?

+--------------------- 16 字节头部 ----------------------+------- 寄存器区 -------+
| HYLL(4B) | encoding(1B) | notused(3B) | card(8B)      | 16384 * 6 bit        |
| 魔数     | 稀疏/稠密     | 保留        | 基数缓存       | 稠密 12288B / 稀疏可变 |
+-------------------------------------------------------+----------------------+

The header is fixed at 16 bytes,card Those 8 bytes are for single keys PFCOUNT Cache field that produces write side effects (highest bit 1 indicates invalidation). dense 16 + 12288 = 12304 Bytes, regardless of base; when sparse is first created, the register area only needs 2 bytes (one XZERO represents 16384 all 0s),STRLEN is 18.MEMORY USAGE Contains key name,dictEntryexpiration dictionary, usually 2 – 3 times larger than pure data.

2.3 Sparse vs. dense: when to double

If any condition is met, it becomes dense, andOne-way irreversible: The value to be written to a certain register > 32 (HLL_SPARSE_VAL_MAX_VALUE); or sparse means that the total length exceeds hll-sparse-max-bytes(Default 3000 bytes, including 16-byte header).

# redis.conf / ADVANCED CONFIG 段：稀疏表示的长度上限
hll-sparse-max-bytes 3000

Official trade-off: 3000 takes into account memory saving and writing performance; when the CPU is sufficient, the memory is tight, and the base is mostly between 0 and 15000, it can be raised to about 10000;More than 16000 uselessdenser but more economical. You can’t blindly increase the size because it’s sparse. PFADD yes O(N)。

The measured average value of source code comments: when the base is 1,000, the sparseness is about 1,882 bytes, when the base is 10,000, it is about 10,591, and the denseness is always 12,288; the original sentence “The sparseness is obviously more economical when the base is about 2,000 – 3,000 or less” is more useful than the default value of 3000 – the real profit range of sparseness is within 3,000. It will not return to sparse after becoming dense.Capacity is estimated based on worst case scenario。

2.4 How to determine whether it is sparse or dense now?

Only recognize STRLEN: Between 18 and 3000 is sparse, and 12304 is dense.

# 类型永远是 string：Redis 侧无法区分 HLL 与普通字符串
TYPE uv:demo
# 这是 String 对象的编码（raw / embstr），不是 HLL 的 sparse / dense
OBJECT ENCODING uv:demo
# 整只为二进制，前 4 字节是 HYLL 魔数
GET uv:demo
# 判断稀疏 / 稠密只认它
STRLEN uv:demo

OBJECT ENCODING Can’t give it sparse / dense，Cannot be used to determine HLL encoding. It even raw Neither necessarily gives:createHLLObject() Is to take 18 bytes sds directly createObject(OBJ_STRING, sds)not explicitly specified OBJ_ENCODING_RAWshort objects are displayed in some versions embstr;First time with element PFADD(internal dbUnshareStringValue Copy it once) before it becomes stable. raw. You will know by running two of your own versions:PFADD uv:probe Watch it once,PFADD uv:probe a Watch it again.

Two inferences: The Redis side cannot identify “which keys are HLL”, and monitoring and cleaning can only rely on the prefix + SCAN;HLL is an ordinary String,EXPIRE / DEL / UNLINK It can be used, and it can be used GET Read out the wholeSET Write back for migration; vice versa PFADD Report as ordinary String WRONGTYPEmust be isolated with independent prefixes.

3.1 Quick check of three commands

official PFCOUNT The sentence “Hundreds of operations per second are possible” in the single-key performance section coexists with “performances are excellent” in the same section. It is a conservative statement without hardware conditions:Don’t use it as a throughput limit, and don’t use it to kick single-key queries off the link.。

3.2 60 seconds minimum example

# 11/7 写入 3 个用户：同一天可多次调用，重复元素返回 0
PFADD uv:page:home:20261107 user:1001 user:1002 user:1003
# 单键查询：返回近似基数 3
PFCOUNT uv:page:home:20261107
# 合并两天到周 key：尚不存在，以稀疏编码创建
PFMERGE uv:page:home:week:2026W45 uv:page:home:20261107 uv:page:home:20261106
# 并集是 5 个不同用户，不是 3 + 3 = 6
PFCOUNT uv:page:home:week:2026W45
# 日 key 保留 35 天；周汇总保留 90 天，PFMERGE 不继承源 key 的 TTL
EXPIRE uv:page:home:20261107 3024000
EXPIRE uv:page:home:20261106 3024000
EXPIRE uv:page:home:week:2026W45 7776000

Seven items return in turn 1 / 3 / OK / 5 / 1 / 1 / 1. 3 + 2 writes, 5 different users in two days,user:1003 Only counts once.

3.3 PFADD: Returns 0 if it is not equal to the old element

# 3 个新元素，至少一个寄存器被改写，返回 1
PFADD uv:demo user:1001 user:1002 user:1003
# 重复元素：落在同一个桶且 rho 不更大，返回 0
PFADD uv:demo user:1001
# 不带元素：key 不存在时创建空 HLL，返回 1
PFADD uv:empty
# 已存在的空 HLL 再做一次空操作，返回 0
PFADD uv:empty

return 1 means “at least one register has been changed”,0 means “no”. The update rule is M[index] = max(M[index], rho)：

in conclusion:Return 0, which may be an old element or a new element.can only be used as a signal that “the estimated value may have changed”; please use it if it is new. SETNXSet, Bloom or unique constraint. Accident prototype: Use return 1 to issue novice coupons. The same user occasionally returns 1, and coupons are issued repeatedly.

3.4 PFCOUNT: Two forms of the same command

The official saying “PFCOUNT “Technically a write command” refers to single-key cache writeback; the command flag is still readonlystarting from 7.0 with additional may-replicate. Three engineering impacts: Cache invalidation will modify the value and propagate to the replica/AOF, generating replication traffic, even if it is just to check the number; starting from 7.0, it is prohibited to use read-only scripts (EVAL_RO / FCALL_RO); called in replica-read-only yes on the slave database PFCOUNT Can be executed normally (source code uses lookupKeyRead()), the query can be routed to the replica, butDon’t write “the copy will definitely report an error” into the plan. When checking the same union repeatedly, first PFMERGE Complete the result key and check it with one click.

3.5 PFMERGE: Three semantics that must be written accurately

# 先清历史数据，保证结果可复现、可反复执行
DEL uv:demo:d1 uv:demo:d2 uv:demo:week
# 两个日集合，c 两边都有，用来验证并集去重
PFADD uv:demo:d1 a b c
PFADD uv:demo:d2 c d e
# dest 不存在时以稀疏创建；若 dest 已存在，它会被当成额外的源一起合并
PFMERGE uv:demo:week uv:demo:d1 uv:demo:d2
# 并集基数 5：a b c d e，c 只算一次
PFCOUNT uv:demo:week

one,destkey Participate in the merge if it already exists. The official original text “If the destination variable exists, it is treated as one of the source sets”——not overwriting, the source code merge loop starts from argv[1](destkey itself) starts. A clean rebuild must first DEL Or use temporary key + RENAME;To add a new day to the existing week HLL, do not delete it, just PFMERGE weekKey dayKey. The syntax allows zero source keys, which is equivalent to merging with itself.

2. The target encoding is not “certainly dense”. dest is created sparsely when it does not exist, and is converted to dense only when any HLL participating in the merge is dense (use_dense). 2.8.9 has been unconditionally converted to dense, which is where the early saying “the result must be dense” comes from; daily/weekly scene sources are usually dense, and the actual results are basically 12 KB.

3. Do not inherit TTL. dest cannot get the expiration time of any source and must explicitly EXPIRE. In addition: if the source key does not exist, it will be treated as an empty HLL; if the source key is not a legal HLL, it will be reported. WRONGTYPE Key is not a valid HyperLogLog string value.; dest and all sources in the cluster must be in the same slot.

3.6 Cluster same slot and hash tag

PFCOUNT Multiple keys and PFMERGE All slot verification is performed, and cross-slot reporting is performed. CROSSSLOT Keys in request don't hash to the same slot。

#  是 hash tag：下面所有 key 都落在这一个 slot 上
PFADD uv:
    byte[] bytes = (byte[])blob;

    // 16 字节头 + 至少 2 字节数据，短于此不可能合法
    if (bytes == null :20261107 user:1001
PFADD uv: bytes.Length == 12304;
:20261108 user:1002
# destkey 也必须带同一个 tag，否则多键命令直接 CROSSSLOT
PFMERGE uv: bytes[3] != 'L') return false;

    // encoding 只允许 0（稠密）/ 1（稀疏）；稠密长度必须正好 12304
    byte encoding = bytes[4];
    if (encoding > 1) return false;
    return encoding == 1 :week:2026W45 uv:{home}:20261107 uv:{home}:20261108
# 返回的是并集基数，不再区分是哪一天的用户
PFCOUNT uv:{home}:week:2026W45

“The client takes the numbers separately and then adds them up” is not an alternative: multi-key PFCOUNT It is a union semantic, and adding them separately will repeatedly count overlapping users. When they cannot be in the same slot, each HLL byte can only be moved to the same instance and then merged (4.6); see 5.3 for the cost of hash tags.

3.7 .NET calls and four pitfalls

using StackExchange.Redis;

// redis 为 IConnectionMultiplexer 实例，例如 ConnectionMultiplexer.Connect("redis://127.0.0.1:6379")
IDatabase db = redis.GetDatabase();

// PFADD：单元素 / 批量（批量减少 RTT），均返回 Task
bool changed = await db.HyperLogLogAddAsync("uv:page:home:20261107", "user:1001");
bool changed2 = await db.HyperLogLogAddAsync("uv:page:home:20261107",
    new RedisValue[] { "user:1002", "user:1003" });

// PFCOUNT：单键与多键重载都是 Task，但集群下多键需同 slot
long uv = await db.HyperLogLogLengthAsync("uv:page:home:20261107");
long unionUv = await db.HyperLogLogLengthAsync(new RedisKey[]
    { "uv:page:home:20261107", "uv:page:home:20261106" });

// PFMERGE：Task 重载；不会帮你设置 destkey 的 TTL
await db.HyperLogLogMergeAsync("uv:page:home:week:2026W45",
    new RedisKey[] { "uv:page:home:20261107", "uv:page:home:20261106" });
await db.KeyExpireAsync("uv:page:home:week:2026W45", TimeSpan.FromDays(90));

// 客户端未封装的命令用原生执行兜底，返回 RedisResult
RedisResult raw = await db.ExecuteAsync("PFCOUNT", "uv:a", "uv:b");

// raw 为 nil 或 error 时强转会抛 InvalidCastException，必须先判空
long unionFromRaw = raw.IsNull ? 0 : (long)raw;

There are four pitfalls on the client side.HyperLogLogAddAsync return booldon’t use it to judge new or old; multi-key HyperLogLogLengthAsync There will be no automatic routing under the cluster, cross-slot is runtime CROSSSLOT；HyperLogLogMergeAsync If TTL is not set, the business layer must make up;(long)RedisResult A judgment must be made before forced transfer IsNullotherwise it is thrown when nil or error InvalidCastException。

4.1 Caliber first: more error-prone than commands

Most of the root causes of the “inexplicable decline” of UV boards are here, not the algorithm. Four things must be done in writing before going online:

PFADD Naturally idempotent, it is safe to bury the key and retry; when marking rule changes, you must create a new batch of keys, and do not mix old keys.

4.2 key specification and TTL

uv:{tag}:              例：uv:{home}:20261107
uv:{tag}:week:            例：uv:{home}:week:2026W45
uv:{tag}:month:          例：uv:{home}:month:202611
uv:{tag}:src:<渠道>:    例：uv:{home}:src:wechat:20261107

Four norms. independent prefix uv:(isolation WRONGTYPEis also the only starting point for monitoring/cleaning); the cycle uses a fixed-length format; keys that need to be merged together (Contains destkey) reuse the same tag, and the tag is only bound to the key that really needs to be merged.

The most typical accident: the Japanese key was set EXPIRE 86400the weekly/monthly merge has not been completed in the early morning, the earliest few days have expired, and the results are too small. Unified caliber:

TTL is the bottom line of defense, and key report links are cleaned offline at the same time (low peak UNLINK) and do “day to month” downsampling.EXPIRE It has overwriting semantics. Repeated calls will renew the key indefinitely (Lua 4.3 only TTL < 0 time settings).

4.3 days/week/month

# 日 key 写入：埋点链路每次访问调用一次
PFADD uv:{home}:20261107 user:1001
# 日 UV：单键 O(1)，可以放在请求链路上
PFCOUNT uv:{home}:20261107
# 周汇总：7 个日 key 覆盖 11/2 - 11/8，全部带 {home} 保证同 slot
PFMERGE uv:{home}:week:2026W45 uv:{home}:20261102 uv:{home}:20261103 uv:{home}:20261104 uv:{home}:20261105 uv:{home}:20261106 uv:{home}:20261107 uv:{home}:20261108
# 显式设 TTL（90 天）：destkey 不继承任何源 key 的过期时间
EXPIRE uv:{home}:week:2026W45 7776000
# 周 UV：结果 key 是单键，同样可进链路
PFCOUNT uv:{home}:week:2026W45

The week number and date must be aligned: 2026W45 is 11/2 (Monday) to 11/8 (Sunday). Bundle 20261109 W45 is the most common error. It belongs to W46, which will make the weekly UV and monthly summary too large at the same time; scheduled tasks use ISO week numbers to reverse 7 dates, do not fill them in by hand.

Same as month UV: offline merge the day key of the current month to uv:{home}:month:202611the interface only reads the result key. Weekly merging of 7 keys is acceptable, and monthly merging of 30 keys must be done offline and off-peak.

.NET two key points:TTL is only set the first time the key appears(Every visit EXPIRE equal to indefinite renewal);Summarize using “temporary key + RENAME Atomic switching(Avoid dest from participating in the merge, and avoid the empty Kanban window after the merge fails). See 4.4 for the writing method. RebuildRollingWindowAsync。

using System;
using System.Linq;
using System.Threading.Tasks;
using StackExchange.Redis;

// UV 读写服务：日 key 写入、单键查询、周 / 月汇总重建
public sealed class UvService
{
    private readonly IDatabase _db;

    // 复用同一个 multiplexer，不要在每次请求里新建连接
    public UvService(IConnectionMultiplexer redis) => _db = redis.GetDatabase();

    // hash tag 统一由这里生成，避免某个调用点漏写花括号导致 CROSSSLOT
    private static string Tag(string page) => $"{{{page}}}";
    private static string DayKey(string page, DateTime d) => $"uv:{Tag(page)}:{d:yyyyMMdd}";

    // 一段 Lua 原子完成 PFADD 与「TTL 只在首次出现时设置」：拆两条命令会有并发续期
    private const string TrackScript = @"
        local ttl = redis.call('TTL', KEYS[1])
        redis.call('PFADD', KEYS[1], ARGV[1])
        if ttl < 0 then redis.call('EXPIRE', KEYS[1], ARGV[2]) end
        return 1";

    // day 由调用方换算成业务时区的自然日（见 4.4）；dayTtl 建议 35 天
    public async Task TrackVisitAsync(string page, string userId, DateTime day, TimeSpan dayTtl)
    {
        // KEYS[1] 日 key；ARGV[1] 用户标识；ARGV[2] TTL 秒数
        await _db.ScriptEvaluateAsync(TrackScript,
            new RedisKey[] { DayKey(page, day) },
            new RedisValue[] { userId, (long)dayTtl.TotalSeconds });
    }

    // 日 UV 查询：单键 O(1)，可以放进请求链路
    public Task DayUvAsync(string page, DateTime day)
        => _db.HyperLogLogLengthAsync(DayKey(page, day));
}

4.4 Rolling window (last N days)

The window moves every day, HLL cannot be deleted,PFMERGE It can only be added, so the entire window must be rebuilt. First determine the “including and excluding the current day”: if the current day is included, the UV of N days “up to this moment” will be checked during the day. To complete the natural day, start from today.AddDays(-1) Starting calculation; both are acceptable, but the dashboard, alarm, and report must be consistent.

The following is 4.3 UvService Supplementary members in the class:

// 业务时区固定为 Asia/Shanghai；时区 ID 在 Windows 与 Linux 上由 .NET 统一映射
private static readonly TimeZoneInfo BusinessTz =
    TimeZoneInfo.FindSystemTimeZoneById("Asia/Shanghai");

// 「今天」必须由业务时区换算，不能用 DateTime.Today（那是服务器本地时区，见 4.1）
private static DateTime BusinessToday() =>
    TimeZoneInfo.ConvertTime(DateTimeOffset.UtcNow, BusinessTz).Date;

// ttl 不传时按 4.2 的 2 天口径，避免每个调用方各自拍一个值
public async Task RebuildRollingWindowAsync(
    string page, int windowDays, TimeSpan? ttl = null)
{
    if (windowDays <= 0) throw new ArgumentOutOfRangeException(nameof(windowDays));

    // 窗口 key 不含日期：它是「最近的 N 天」，靠 TTL 保证不会长期滞后
    var windowTtl = ttl ?? TimeSpan.FromDays(2);
    var today = BusinessToday();
    var windowKey = $"uv:{{{page}}}:rolling{windowDays}d";
    var tmp = $"uv:{{{page}}}:tmp:{Guid.NewGuid():N}";

    // 含当日：白天重建得到「截至此刻」的 N 天 UV；要完整自然日请从 today.AddDays(-1) 起算
    var keys = Enumerable.Range(0, windowDays)
        .Select(i => (RedisKey)$"uv:{{{page}}}:{today.AddDays(-i):yyyyMMdd}")
        .ToArray();

    try
    {
        // 全新临时 key，结果不含任何历史窗口数据
        await _db.HyperLogLogMergeAsync(tmp, keys);

        // tmp 的 TTL 会随 RENAME 迁移给 windowKey，不必重建后再补 EXPIRE
        await _db.KeyExpireAsync(tmp, windowTtl);

        long uv = await _db.HyperLogLogLengthAsync(tmp);
        await _db.KeyRenameAsync(tmp, windowKey);
        return uv;
    }
    finally
    {
        // 失败时清理临时 key，避免重建反复失败时堆积
        _ = _db.KeyDeleteAsync(tmp, CommandFlags.FireAndForget);
    }
}

The window key does not contain a date, so “reconstruction failed” and “the window contains yesterday’s data” cannot be seen in the key name. Use TTL to cover the whole situation: it will disappear automatically after 2 days. I would rather not have it than give the old data; save it when stronger observability is needed. built Timestamp key, compare before taking the value.

4.5 Multi-dimensional: PV and UV dual aperture

# 活动开始前清一次：dest 若已存在会被当成源，不清会带上上一轮数据
DEL uv:{activity}:all:20261107 uv:{activity}:src:wechat:20261107 uv:{activity}:src:search:20261107
# PV 用 INCR 精确计数，不要拿 PFADD 返回值凑
INCR pv:{activity}:20261107
# 同一用户从两个渠道进来各写一次；跨渠道重叠由并集自动去重
PFADD uv:{activity}:src:wechat:20261107 user:1001
PFADD uv:{activity}:src:search:20261107 user:1001
PFADD uv:{activity}:src:search:20261107 user:1002
# 渠道并集就是全站 UV，不要再单独维护一个「全站」HLL
PFMERGE uv:{activity}:all:20261107 uv:{activity}:src:wechat:20261107 uv:{activity}:src:search:20261107
PFCOUNT uv:{activity}:all:20261107

The premise is that the channel enumeration is complete; if there is “direct access”, build the HLL and merge it together. Do not mix PV accurate + UV estimation: deduplication rate = UV / PV, the numerator is estimated, the denominator is accurate, and a margin is left for boundary scenes. For writing side CreateBatch Bundle INCR and PFADD Synthesize an RTT and it is necessary to complete the modification. await Two Tasks – only queued without await,WRONGTYPE / CROSSSLOT Not visible at all. For the complete writing method, see the batch example in 4.4.

4.6 Multiple computer rooms/cross-instances

PFMERGE Can only be mergedwithin the same instancekey. The correct approach is to move the HLL bytes (about 12 KB each) of each computer room to the central instance and write them as temporary keys, and then merge them in the center to automatically deduplicate overlapping users.

using System.Collections.Generic;

// 中心节点定时任务：各机房取字节 -> 中心临时 key -> 合并 -> RENAME；_db 为中心实例
public async Task MergeAcrossRegionsAsync(
    DateTime day, IReadOnlyDictionary regionDbs,
    string srcKey, TimeSpan destTtl)
{
    // 所有中转 key 与 dest 共用同一个 tag，否则中心侧合并会 CROSSSLOT
    var tag = $"global:{day:yyyyMMdd}";
    var dest = $"uv:global:{{{tag}}}";
    var tempKeys = new List();

    try
    {
        // 不用 KeyValuePair 解构，兼容 netstandard2.0 / net472
        foreach (var kv in regionDbs)
        {
            // 只取整只二进制，不做任何解析或改写
            RedisValue blob = await kv.Value.StringGetAsync(srcKey);
            if (blob.IsNullOrEmpty || !LooksLikeHll(blob)) continue;

            // 2 小时自动过期：搬运成功与否都不会留下垃圾
            var tmp = (RedisKey)$"uv:tmp:{{{tag}}}:{kv.Key}";
            await _db.StringSetAsync(tmp, blob, TimeSpan.FromHours(2));
            tempKeys.Add(tmp);
        }

        // 所有机房当天都没数据，不必建空 HLL
        if (tempKeys.Count == 0) return 0;

        // 全新临时 key：dest 不参与合并，机房数据晚到只需重跑同一天
        var staged = (RedisKey)$"uv:global:{{{tag}}}:tmp:{Guid.NewGuid():N}";
        await _db.HyperLogLogMergeAsync(staged, tempKeys.ToArray());

        // 先设 TTL：RENAME 之后 dest 继承它，不必事后补
        await _db.KeyExpireAsync(staged, destTtl);

        long uv = await _db.HyperLogLogLengthAsync(staged);
        await _db.KeyRenameAsync(staged, dest);
        return uv;
    }
    finally
    {
        // 清理中转 key；FireAndForget 不阻塞
        foreach (var k in tempKeys)
            _ = _db.KeyDeleteAsync(k, CommandFlags.FireAndForget);
    }
}

// 搬运前粗筛：魔数 + encoding + 长度自洽。只挡误传，不是安全边界
private static bool LooksLikeHll(RedisValue blob)
{
    byte[] bytes = (byte[])blob;

    // 16 字节头 + 至少 2 字节数据，短于此不可能合法
    if (bytes == null || bytes.Length < 18) return false;

    // 魔数必须是 HYLL：这里是唯一能挡住「拿错 key」的检查
    if (bytes[0] != 'H' || bytes[1] != 'Y' || bytes[2] != 'L' || bytes[3] != 'L') return false;

    // encoding 只允许 0（稠密）/ 1（稀疏）；稠密长度必须正好 12304
    byte encoding = bytes[4];
    if (encoding > 1) return false;
    return encoding == 1 || bytes.Length == 12304;
}

Three key points. Only GET / SET Moved in its entirety, never parsed or rewritten (also DUMP / RESTORE）；LooksLikeHll It cannot stop malicious structures – the length, magic number, and encoding are all correct, but strings with malformed registers still pass, and the source must be trustworthy (hllMerge() It is the trigger point of CVE-2025-32023); the source key TTL covers the maximum delay of the computer room + the backfill window, and cross-version transfer is first used in the test environment PFCOUNT acceptance.

5.1 Capacity: Estimate based on 12,304 bytes, then multiply by 2-3 times

Converted according to 1 GB = 10^9 bytes; pure data columns are only used for magnitude intuition.Budget and alarm thresholds include expense columnsand use used_memory check.

The real cost lies in the number of keys: each additional dimension is “about 12 KB × number of days”, so it is necessary to set TTL, do “day to month” downsampling, export the cold dimensions to OLAP and clean them up.

The 1.2 GB HLL data set will also be enlarged in three places: persistence (RDB will follow the increase, AOF each PFADD All are written once); fork and COW (merging is concentrated in the early morning, hitting the backup window); full synchronization (multiplied by the number of computer rooms equals bandwidth and time). So the capacity requires two numbers:used_memory Peak value + RDB volume/full synchronization time.

5.2 Hot path restricted area

A widely circulated erroneous suggestion is that “large merges are executed first in the slave database, and the results are synchronized back to the master database.”PFMERGE yes @write,exist replica-read-only yes The slave database will be rejected, and an independent analysis instance should be placed or the batch should be run in the main database with staggered peaks;PFCOUNT yes readonlycan be executed on the copy. On the contrary, when the cardinality is only a few hundred, the small Set is more compact and enumerable, and there is no need for HLL to play.

5.3 Cost of hash tag

All keys with the same tag are fixed in the same slot. according to {page} Binding, the day/week/month/channel key of this page will always be on the same shard:Hotspot sharding(all PFADD Pressed on single thread), slot tilt (reshard cannot open this tag),single shard maxmemory Become a hard cap.

Three mitigations: tag plus time bucket (such as {home:2026W45}cross-cycle merging requires moving bytes first); put independent instances in the hotspot dimension; reduce the set of the same tag. Principle: First determine “which keys must be in the same slot”, then design the tag; use it after going online CLUSTER COUNTKEYSINSLOT with each shard used_memory Look at the distribution.

5.4 Obsolescence, Hotspots and Availability

In addition: you need to use “temporary key + RENAME“, don’t “first DEL “Re-merge” leaves an empty window; if the computer room data is late, you can only rely on the source key TTL to cover the delay + re-run on the same day.

5.5 Security and Upgrades: This section is more important than performance

official PFCOUNT The sentence on the page “Executing on a corrupted HLL PFADD / PFCOUNT / PFMERGE “Does not affect server stability” is an old expression in the context of normal misoperation, not a security promise:

• CVE-2019-10192/10193 (2019): Corrupted HLL can trigger heap/stack out-of-bounds writes; affects 3.x < 3.2.13, 4.x < 4.0.14, 5.x < 5.0.4.
• CVE-2025-32023/GHSA-rp2m-q4j6-gr43 (2025, High Risk, CVSS 7.0): Integer overflow occurs in sparsely encoded run length accumulation, specially crafted string can be used in hllMerge()(stack) or hllSparseToDense()(Heap) causes out-of-bounds writing, which can lead to remote code execution; impact >= 2.8, fixed version 8.0.3 / 7.4.5 / 7.2.10 / 6.2.19,No patches for 5.x and earlier。

Four project requirements.

1. Upgrade to one of the repair versions and incorporate the HLL security bulletin into the upgrade process; 5.x that cannot be upgraded can onlyDisable HLL commands using ACL。
2. Use ACL to @hyperloglog and SET / SETRANGE / RESTORE Grant them separately according to the minimum permissions, and do not open them to untrusted clients at the same time. PF* and the ability to write String.
3. Cross-instance transfer only accepts trusted sources, and external bytes are verified first. HYLL magic number and encoding；Don’t let untrusted input go directly to PFMERGE——PFMERGE Not a legality checker.
4. Member inference side channel: the hash is MurmurHash64Athe seed is fixed and does not contain the key, and the register can be recalculated. Get the complete bytes (GET / DUMP / RDB leak), the candidate uid Count one by one index and rho By comparison, it can be determined that “this person has probably been here” – there are false positives, but it is enough to narrow the candidate set. Therefore, HLL key export and backup are managed as sensitive data; when anti-inference is really needed, the identifier is salted, at the cost of being unable to merge across instances.

5.6 Monitoring and Error Sampling

IServer.Keys() The inside is long SCAN loop, single SCAN 0 Only one batch is returned, and taking that result as the full number of keys will lead to misjudgment; this statistic is only suitable for low-peak runs or sampling, and do not hang on to monitoring that is executed every minute.

Error verification must be done offline (do not double-write online): use a Set or offline details of the same caliber as a benchmark, and select a sample base of 10,000-100,000. 30 keys are drawn from each batch. abs(estimate - exact) / exact Compared with 0.81%; single key exceeds 3 sigma (about 2.43%)Check the engineering links first (missing points, mixed identifiers, key drift, premature TTL expiration, missing summary merges), and then suspect the algorithm; systematic same-direction shift of the same batch of keys is almost certainly a caliber problem.

HLL’s errors are controllable, and there is no bottom line for engineering errors.

Sort by severity: high = money difference, data error or unrecoverable; medium = performance, cluster, life cycle; low = cognitive bias.

---

Caliber and modeling

1. Written confirmation of time zone/calendar day, event time vs arrival time, UV identification definition and governance rules (4.1)
2. “Today” is converted from the business time zone and does not depend on the server’s local time zone (4.1 / 4.4)
3. HLL uses independent prefix uv:the period uses a fixed-length format, and the entire link is uniformly identified (4.2)
4. All keys that need to be merged under the cluster (including destkey) with the same tag, and the tilt has been evaluated (3.6 / 5.3)

Precision and bounds

5. Confirm in writing with product/operation that “0.81% is 1 sigma, not the upper limit” (1.1)
6. HLL is disabled in threshold/amount scenarios such as circuit breaker, settlement, lottery and new judgment (1.3)
7. No PFADD return value,PFCOUNT The results are used in business branches (3.3)

Commands and Performance

8. Live links only allow PFADD with single key PFCOUNT;Multikey PFCOUNT / PFMERGE Always offline (5.2)
9. Union query first PFMERGE Drop the result key and query with a single key (3.4)
10. Rebuild class merge using “temporary key + RENAME”, the additional class is reserved destkey（3.5 / 4.3）
11. Complete the actual measurement: single-key cache hit throughput, cache failure P99, combined time of 7 keys and 30 keys, set the threshold accordingly after filling in the form (5.2)
12. Buried writing is batched; if atomicity is required, use Lua instead MULTI/EXEC（4.5 / 5.4）

life cycle and capacity

13. Explicit TTL for each key: 35 days per day / 90 days per week / month / 2 days window / temporary 1 – 2 hours (4.2)
14. destkey Manual compensation EXPIRE; Key reports are equipped with offline cleaning and downsampling (4.2)
15. Capacity pressIncluding key overheadCaliber estimate, and give used_memory Peak and RDB volume/full volume synchronization time (5.1)

Safety and Observation

16. The version has been upgraded to the CVE-2025-32023 repair version; if it cannot be upgraded, use ACL to disable it. PF*and isolate String write commands by least privilege (5.5)
17. Foreign bytes only come from trusted sources,LooksLikeHll Only when the filter is not suitable for security boundaries; key export and backup are managed as sensitive data (5.5)
18. Monitor the total number of keys, no TTL key, close to 12 KB, slow logs,used_memoryslot distribution, and record offline sampling according to 5.6 (5.6)