1.1. Big Key
The Redis big key problem is not actually a big Key problem, but the value corresponding to the Key is too large, so technically it is a Big Value problem, Redis value is too large (key value is too large).

There is no unified standard for how large a value will cause a big key problem.

For example, for String values, sometimes more than 5M is a big key, sometimes more than 10K can be counted as a Bigey to be on the safe side.

Big Key can lead to what problems?

  1. Because of the large value, serialization and deserialization time is too long, and network latency is also long, resulting in a long time to operate Big Key, which reduces the performance of Redis.

  2. Load balancing is not possible in cluster mode, resulting in load skewing to a certain instance, which results in higher QPS and more memory usage for a single instance.

  3. Since Redis is single-threaded, if you want to perform a delete operation on this Big Key, the instance being operated on may be blocked, thus making it impossible to respond to the request.

How is the Big Key generated?

Generally, the Big Key is generated when the program designer does not anticipate the size of the data, or when design considerations are omitted.

In some business scenarios, it is easy to generate Big Key, for example, the fan list of KOL or traffic stars, the statistics of voting, the cache of large batch data, and so on.

1.2. Hot Key
Hot Key, also known as Hotspot Key, is a problem that occurs when a particular Key suddenly receives a large number of requests, and traffic is concentrated on a particular instance, causing the Redis server to go down because it has reached the physical NIC.

Hotspot Keys can cause many system problems.

  1. The traffic is overly concentrated and cannot take advantage of the cluster, and if it reaches the processing limit of that instance, it causes the node to go down, which in turn impacts the database and risks causing an avalanche of cache and leaving the entire system hanging.

  2. Since Redis is a single-threaded operation, hotkeys can affect the operation of other keys in the example.

2. How to discover BigKey and HotKey

  1. Query BigKey by Redis command.

The following command scans the entire Key space of Redis for the largest Key of different data types. -i 0.1 parameter allows the command to execute sleep 0.1 seconds every 100 times during the scan.

Redis’ own bigkeys command makes it easy to scan large keys online with minimal impact on service performance, with the single disadvantage of less information and only the largest key of each type.

1
$ redis-cli -p 999 --bigkeys -i 0.1

2、Query BigKey through open source tools.

Using open source tools, the advantage is that the obtained key information is detailed, optional parameters, support for customization needs, easy follow-up processing, the disadvantage is that offline operation is required, and the time to obtain the results is longer.

For example, redis-rdb-tools and so on.

git clonelink
1
2
3
$ cd redis-rdb-tools
$ sudo python setup.py install 
$ rdb -c memory dump-10030.rdb > memory.csv

2.2. Discovering HotKeys

  1. hotkeys parameters

Redis added the hotkeys (github.com/redis/redis…) lookup feature in version 4.0.3, so you can directly use redis-cli –hotkeys to get the hotkeys of the current keyspace, which is achieved by scan + object freq This is done by scan + object freq.

2、monitor command

The monitor command can capture the commands received by the Redis server in real time, and capture the data by redis-cli monitor, and combine with some ready-made analysis tools, such as redis-faina, to count the hot keys.

  1. Solution to BigKey Problem

To find and solve the BigKey problem, you can refer to the following ideas.

  1. At the beginning of the design process, estimate the size of the value, and avoid the appearance of too large value in the business design.

2、Discover BigKey as early as possible by monitoring.

3、If it is impossible to avoid large Key, then a Key can be split into multiple parts and stored in different Keys.

The following is an example of List type value to demonstrate the method of splitting to solve the big Key problem.

There is a list of User Id with 10 million data, if all of them are stored under one Key, it will be very large, so you can access the data by splitting it into pages.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
public void pushBigKey(List<Long> userIdList) {
   
    int pageSize = 1000;
    List<List<Long>> userIdLists = Lists.partition(userIdList, pageSize);

    Long index = 0L;
    for (List<Long> userIdListPart : userIdLists) {
        String pageDataKey = "user:ids:data:" + (index++);

        redisTemplate.executePipelined((RedisCallback<Long>) connection -> {
            for (Long userId : userIdListPart) {
                connection.lPush(pageDataKey.getBytes(), userId.toString().getBytes());
            }
            return null;
        });
        redisTemplate.expire(pageDataKey, 1, TimeUnit.DAYS);
    }

    String indexKey = "user:ids:index";
    redisTemplate.opsForValue().set(indexKey, index.toString());
    redisTemplate.expire(indexKey, 1, TimeUnit.DAYS);
}

public List<Long> popBigKey() {
    String indexKey = "user:ids:index";
    String indexStr = redisTemplate.opsForValue().get(indexKey);
    if (StringUtils.isEmpty(indexStr)) {
        return null;
    }

    List<Long> userIdList = new ArrayList<>();

    Long index = Long.parseLong(indexStr);
    for (Long i = 1L; i <= index; i++) {
        String pageDataKey = "user:ids:data:" + i;
        Long currentPageSize = redisTemplate.opsForList().size(pageDataKey);
        List<Object> dataListFromRedisOnePage = redisTemplate.executePipelined((RedisCallback<Long>) connection -> {
            for (int j = 0; j < currentPageSize; j++) {
                connection.rPop(pageDataKey.getBytes());
            }
            return null;
        });
        for (Object data : dataListFromRedisOnePage) {
            userIdList.add(Long.parseLong(data.toString()));
        }
    }

    return userIdList;
}

4. HotKey problem solving methods

If HotKey is present, consider the following solutions.

  1. Use local caching. For example, cache the hot data that needs to be requested at the server, so that you can avoid sending large traffic requests to Redis through load balancing of the server cluster.

However, local caching can introduce data consistency issues and waste server memory.

  1. HotKey will replicate multiple copies, break them up randomly, and use proxy requests.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

public void setHotKey(String key, String value) {
    int copyNum = 20;
    for (int i = 1; i <= copyNum; i++) {
        String indexKey = key + ":" + i;
        redisTemplate.opsForValue().set(indexKey, value);
        redisTemplate.expire(indexKey, 1, TimeUnit.DAYS);
    }
}

public String getHotKey(String key) {
    int startInclusive = 1;
    int endExclusive = 21;
    String randomKey = key + ":" + RandomUtils.nextInt(startInclusive, endExclusive);
    return redisTemplate.opsForValue().get(randomKey);
}