Redis 数据类型全解 – 数据类型选型矩阵：一张表、一棵树、四条线（十二） – IT规划师

a word: The real winner in type selection is not the type itself, but the four lines of “scale × access method × operation and maintenance capabilities × combination constraints”. The first reaction gives you a starting point, and these four lines determine whether this starting point should be overturned.

Selection errors are not like coding bugs: they do not report errors and only become more expensive as the data grows. The cost of repair is magnified by layer – changing the way a field is written changes a few lines of code; changing the way a command is written changes a function;Changing the structure of a Key requires changing the data model, migrating storage, and changing all reading and writing methods.. Therefore, the judgment must be made “before the key is created” rather than “after the stress test”.

There are four types of constraints, and the conclusion will not be established without one of them:

The type is just the first response, and the constraints are the overturning conditions. This sentence will appear repeatedly in this article: Chapter 2 gives the first reaction, Chapter 3 gives the conditions for overturning, and Chapters 5 and 6 give the evidence (cost and boundary) required for overturning.

It is more intuitive to convert the cost into man-hours: changing the reading and writing of a field takes a few minutes; changing the implementation of a command takes half a day (including regression);The structure of changing a Key is “data model + migration script + double writing + cut and read + inventory verification”, usually measured in weeks, and also requires a downtime window or double writing period.. Spending two hours on the selection discussion can save the next two weeks – this is the only reason why this article is willing to spend more time on “judgment”.

Let’s look at the first reaction first, and then look at why we can’t write in the “Common Wrong Choices” column.

Three interpretation reminders:

1. “Not recommended” does not mean “definitely not allowed to write”. It is completely reasonable to use List for preheating tasks that allow message loss. Using List for payment callbacks is an anti-pattern – the definition of anti-pattern is “requiring semantics that it cannot do”.

Second, the same type name does not mean the same capabilities. Bitmap TYPE return stringit is not an independent type, but “bit operation on String”; GEO is not an independent type either, the bottom layer is ZSet,score is a 52-bit geohash integer.

Third, the mapping table gives “first reaction”, not “conclusion”. All issues that can advance to Article 2 (cost) and Article 4 (combination) of Chapter 1 must go through the overturning conditions in Chapter 3 again.

Four,The difference between Stream and List is not “new or old”, but “whether there is confirmation semantics”. List is “disappeared when taken away”, and Stream is “read into PEL and removed only after ACK”. Therefore, if you answer “Can this message be lost” incorrectly, all subsequent optimizations will be in vain; this is also the only prerequisite for the mapping table to write “allowed to be lost” as List.

The first question: Should this piece of data be a “whole piece” or a “small piece”?
Whole block → String (including String+JSON). Want a small piece → Go to the second question.

Second question: Is that small piece a “field” or a “member/element”?
Field → Hash. Members and to be sorted → ZSet. Members only need to remove duplicates/intersect → Set. member and is just a boolean bit → Bitmap.

Question 3: Do I need to “sort by value” or “replay by time”?
Sort by value → ZSet. Replay by time, ACK → Stream required. Just first in, first out, allowed to lose → List.

In addition to the three exits, there is also a secret:When the scale or precision exceeds the capabilities of a single key, it is not a change of type but a change of structure.——Top N are put into Redis, and the rest are stored in the database; sharding according to business dimensions; precomputation replaces real-time aggregation. Changing the type cannot solve the physical problem of “one Key is too big”.

Three conditions for overturning(If either article is true, the first response is invalid):

one,A single Key will exceed the encoding threshold. Hash super 512 field translation hashtableZSet exceeds 128 members or single value exceeds 64 B translation skiplistSet more than 128 members or more than 64 B elements (starting from 7.2) hashtable. After flipping, the cost per element will increase by an order of magnitude. At this time, you must either control the scale or accept the cost, and you cannot pretend that it has not flipped.

two,Access method does not match type. Only change one field but use String to store the entire object (write amplification = full read, modify and write, and non-atomic); only read the Top 10 but use Set to store the entire object (no sorting capability).

three,Combination out of bounds. Two keys are involved and need to be atomic, but they are not in the same slot – the only way is to change the key design (add a hash tag), and the client cannot remedy it.

One walkthrough: 1 million user check-ins.
The first question: “Did this user sign on a certain day?” It’s a small piece. Second question: It is a Boolean bit → Bitmap, offset uses “user serial number × 1” or “user serial number”, the premise is to create it first id → offset mapping. Can I directly use the user ID as offset? No – ID is an 8-bit long integer, the maximum value may be 10^10,SETBIT GB-level memory will be allocated immediately, just step on Chapter 7, Item 5.
Override the condition check: 1 million bits = 125 KB, which does not exceed the encoding threshold and does not trigger operation and maintenance problems. It is established.
But if the requirement becomes “a check-in calendar for 30 consecutive days”, the answer changes: This is a “fixed-length segment of one bit per day”, using Bitfield or “one Key per day + BITCOUNT“More worry-free; if you also want to “check in this month”, then the sorting requirement arises, the main body is still Bitmap, and a ZSet is added for ranking – this is the combination of Chapter 6.For the same business word, add a requirement and change the answer.this is the meaning of the existence of decision trees.

Check it out again: a real-time list of 10 million members.
The first question: only take the first 100 → is a small piece. Second question: Members need to be sorted → ZSet. The third question: Sort by value → ZSet, the first reaction is true.
However, the three overriding conditions must be checked one by one: the encoding threshold does not pose a problem (this scale is already skiplist）；Operation and maintenance threshold exceeded——Single Key 10 million, backup, migration,SCANslot relocation began to time out; the scale threshold also exceeded the limit. The conclusion is:The type is correct, the structure is wrong. There are three reform methods, sorted by cost: Key by ranking period (rank:2026:11), only the Top N are retained, and the rest are dropped into the database; fragmented according to business dimensions (partitions, segments); precomputation replaces real-time aggregation.
Pay attention to the easiest step to go astray here: many people will switch to Set or Hash to “save memory”. As a result, they lose their sorting ability, which is equivalent to using the wrong second question.When the reason for overturning is scale, the answer is always to change the structure, not the type.

The same “user information”, three storage methods. The difference is not in the type name, but inHow much does it cost to change a field?。

Two boundaries that are easy to step on:

“JSON saves memory” is an illusion. Field names are also stored in JSON, with additional quotes, colons, curly braces and escapes. It is more economical only when “there are few fields and large values”; when there are many fields and small values, Hash’s listpack encoding compresses the length prefix of each field to 1-2 bytes, and the unit overhead is actually lower. The basis for judgment is not intuition, but the actual measured caliber of 5.3.

String+JSON Field-level atomic updates cannot be done. Native String does not have the ability to “change a certain path in JSON”. Without RedisJSON, every time a field is changed, it will be “read the full amount → change → write back”. If it is changed by others between two requests, the update will be lost. If you want field-level atoms, Hash is the native answer; if you want the nested structure of JSON and need local updates, then RedisJSON (JSON.SET $path), this is another selection branch and will not be discussed in this article.

Criteria for splitting multiple keys: There are less than 5 fields, the reading frequency of each field is very different, or a field needs to expire separately (EXPIRE Can only be applied to the entire Key). On the other hand, fields must always be read together, or must be updated transactionally, and splitting them apart is creating cross-Key consistency problems.

5.1 Complexity only depends on one thing:O(N) Who is the N in

O(1) The bottleneck is the single Key QPS and network round trip, not the CPU;O(log N + M) As the number of returned items increases,M Controlled by the caller;O(N) The danger is that N isThe size of this Keybut the Key cannot be opened.

Three common mistakes:SINTER yes O(N*M)N takes the cardinality of the smallest set, so “intersection of a small set and a large set” is cheaper than “intersection of two large sets”;PFCOUNT Multi-Key Yes O(N)N HLLs need to be temporarily merged;SCAN It is guaranteed that “all elements in the collection must be returned”, butMay return repeatedlythe caller must be idempotent.

“How long does it take to read once” can be roughly calculated: the RTT of a round trip plus the processing time of the command itself.O(1) Class commands are almost all RTT;O(N) The class command on the big key isMain thread serial processingmillions of elements often correspond to hundreds of milliseconds of blocking – during this time all requests on this node are queued. So “Is there any LIMIT” affects the online experience more than “which type to use”.

5.2 Three-layer thresholds, do not substitute each other

These three numbers answer three different questions:Can it save memory, can it be operated and maintained, can it last longer?. It is the most common misunderstanding to regard 512 as the “performance red line” – it is just a coding switching point, not a capability boundary; conversely, it is also a misunderstanding to regard 10 million as “the coding has not been changed yet, so it is fine”, which is already an operation and maintenance problem.

The threshold is bound to the version: Set’s listpack path is only available since 7.2 (all integers in 7.0 and before intset,otherwise hashtable); 512 of Hash and 128 of ZSet aredefault value,Can CONFIG SET Changed, but changed the assumption that it needs to be synchronized to all nodes and all clients. Cross-version coding, first CONFIG GET Confirm, do not use the “default value I remember” calculation.

5.3 “XX bytes per element” cannot be multiplied by production scale

Any per-element overhead that is accurate to bytes does not hold true: it varies with encoding state, key length, allocator, and real data distribution. There is only one correct approach:To sample from the target data distribution, use INFO memory of used_memory Calculate the difference before and after。

Three reproducible measured anchor points (Redis 7.4.2):

• Caliber misalignment is the norm: For the same ZSet, multiply “112 bytes/member” by 1 million to get 112 MB. If converted to the full Key size, the actual measurement is 145 MB, that is, 145 B per member. The difference is not that who calculated it wrong, but that the caliber is different: 112 B is the sample value of the element itself (including the prefix in the encoding), and 145 B also needs to add the Key name,redisObject The multi-level pointers of the header, skiplist and allocator are fragmented by size class. Therefore, the capacity should be calculated according to the “actual measurement of the whole key”, and 20%-30% redundancy is left by default.Disable “multiply quantity per element”。
• SETBIT bm 99999999 1 after MEMORY USAGE bm = 12500070,and 1e8/8 = 12,500,000 The difference is only 70 bytes. The capacity of Bitmap can be calculated because it is a pure bitmap; this also shows that the structural overhead is not negligible for small keys but can be ignored for large keys.
• MEMORY USAGE Stream is systematically underestimated by about 3 times: 1,000 three-field small messages,MEMORY USAGE About 39 KB,used_memory The difference is about 130 KB. It counts rax + listpack, excluding consumer group structure and PEL, so the Stream capacity can only be MEMORY USAGE whenNether。
• PEL about 170 B/item: 1 million unACKed messages occupy an additional approximately 170 MB, and MAXLEN ~ Approximate cropping ofMay crop unACKed messages. Use it if it’s reliable MINID warranty window, or regular XPENDING Inspection is left vacant; these two things should be done together. Setting up a window without inspection is equivalent to pushing the problem to the time of failure.

5.4 Two precision bounds

The score of ZSet is double: Integers can be accurately represented within 2^53. The millisecond timestamp is about 1.7×10^12, which is safe; the nanosecond is about 1.7×10^18, which has exceeded the limit. Adjacent tasks will fall to the same score, and the order will be invalid. The delay queue uses nanoseconds as the score, which is a choice that “seems more accurate, but is actually less reliable.”

GEO’s score is a 52-bit geohash integernot longitude and latitude, nor business score; the distance is calculated according to the spherical model, and the official label error is about 0.5%, so the result must be verified twice, and cannot be score Used when business sorting value.

Real requirements are almost all combinations: the main type stores the subject, and the auxiliary type stores the index. The pitfalls of the combination are all concentrated on the “border”, andIt cannot be measured in a stand-alone environment.。

① Cross-slot → hash tag, not client-side remedy.
Cluster uses Key as the minimum fragmentation unit, and the multi-Key command (MGETset operations,MULTILua) requires all keys to be in the same slot. When splitting multiple keys for the same business object, use {user:1001}:cart and {user:1001}:rank Pin them together. Note that the hash tag only solves the “same slot”, not the “same transaction” – the transaction also requires these keys to be on the same node, and the tag just solves the two things together.

② The boundary of a transaction is a connection, not a command sequence.
MULTI Subsequent commands only join the queue,EXEC are executed together; but the Redis transactionNo rollback——A runtime error only affects the one that made the error, and the rest will take effect as usual. More importantly:MULTI、QUEUED、EXEC Must be sent on the same connection. Use script pipeline to feed these lines at once redis-cliit will open a connection for each row,EXEC Report directly ERR EXEC without MULTI——The most common false failure in local verification scripts.

③ To be atomic across Keys, you can only use Lua or accept reconciliation.
Delay queue is a typical case:ZRANGEBYSCORE take out,ZREM Delete, any delay between the two commands will allow multiple consumers to fetch the same batch of tasks. Put two steps into one Lua call (the script only touches one Key, not under Cluster) CROSSSLOT) is truly “take it out and delete it”. The non-atomic version only allows a single consumer – this is not a suggestion, it is a semantic boundary.

Let me make one thing clear:The non-atomic version is not a “faster version”, it is a “one-consumer version”. It is simpler when running only one consumer; once it is expanded to two, it must be changed to the Lua version. Don’t use distributed locks to compensate – the lock granularity and retries will lower the throughput by an order of magnitude, and Lua executes on the server side with no extra round trip.

④ Double writing requires downgrading and reconciliation.
Ranking list = ZSet sort + Hash details. Only when both sides have the same slot can a transaction be entered; when the details are missing (user logs out, double-write is lost once), the reading side must give a placeholder value and cannot let the front end take it. null. Downgrade rules should be written into the code, not the documentation.

⑤ Reading and amplification need to be solved in one round trip.
“Take the Top 10 and check the details one by one” is N+1: the delay increases linearly with N, and N is determined by the length of the list. use CreateBatch put 10 times HGET Press into one hairdo. The order cannot be reversed: first Execute If you wait for the result, you will be waiting for an order that will never be issued.

The writing side and reading side should be designed separately:The writing side requires atoms and order(Transaction, Lua, same Key),Number of round trips required for reading(Batch, Pipeline, batch interface). Mix the two sides into a set of interfaces, and you must eventually compromise on one side – usually the reading side, because errors on the writing side are exposed earlier.

7.1 Twelve anti-patterns (ordered by consequences)

7.2 Fifteen inspections

① Double write the shadow structure. The new structure (Hash, or multiple keys after fragmentation) is written at the same time as the old structure, and the old structure is still read. This step does not change the reading path, has the lowest risk, and is also the easiest to roll back.

② Sampling comparison + full verification. First select 1%, use two sets of reading methods for the same batch of keys, compare the number of fields and compare the values; run another round SCAN Full comparison. The source of the difference is usually boundary writes (concurrent updates, expiration timing), which must be zeroed before proceeding to the next step.

③ Cut and read, keep and write, and then delete the old ones. Start reading (with second-level rollback) and observe a business cycle; then stop old writing; and finally delete the old key.The fallback point only exists in step ①——Once the writing of the old structure is stopped, rolling back is equivalent to a second migration, and the cost is higher than the first time.

Three hard constraints during the migration period: large key splitting SCAN + Write in batches, don’t KEYS; Practice slot migration before Cluster migration to match the copy. Don’t do it for the first time at the peak of business; the capacity of the double-write period is calculated as the sum of the two structures, otherwise the memory will be exhausted before the switch.

There are three numbers to keep an eye on during the migration period:Double write failure rate(Decide whether to continue),The number of differences between the two sets of structures(Decide whether it can be cut and read),Old structure QPS attenuation curve(Deciding when to stop writing). The migration is complete only when all three numbers are moving in a good direction; stopping the old writing just to see if “the new structure can be read through” is leaving the risk to the next failure.