CLUSTERDOWN Hash slot not served问题复现

1、一主两从

172.31.4.18:6381> cluster nodes
f5e774bd5cd8f5bcba53f1297015f3245dd3f18c 172.31.4.20:6383@16383 master - 0 1726278342891 10 connected 5461-10922
9a79febbfb1d7a8f7a6ba3fb11e86b0f122f71e7 172.31.4.21:6386@16386 slave 401730ba6e1c2cd8b60243990f0965606edfc083 0 1726278344894 2 connected
23bb415a72cf125c8205be6b1de4f8f5a0a636c8 172.31.4.18:6381@16381 myself,master - 0 1726278336000 1 connected 0-5460
602fbfe65cf83178c0b897441013ec0bae590c96 172.31.4.20:6385@16385 slave 401730ba6e1c2cd8b60243990f0965606edfc083 0 1726278341890 2 connected
3dec7d5dd8139854f766417611026b07785c0a76 172.31.4.19:6382@16382 slave 23bb415a72cf125c8205be6b1de4f8f5a0a636c8 0 1726278342000 1 connected
83d609a16f5fc2605ff8fd698197533c6ad29334 172.31.4.18:6383@16383 master - 0 1726278343000 9 connected
401730ba6e1c2cd8b60243990f0965606edfc083 172.31.4.18:6385@16385 master - 0 1726278343000 2 connected 10923-16383
4e25e8f87029a22bbba7aa5f6fb2edba4a342da5 172.31.4.21:6384@16384 slave f5e774bd5cd8f5bcba53f1297015f3245dd3f18c 0 1726278343892 10 connected
172.31.4.18:6381> exit

2、从172.31.4.22上发4个并发

redis-benchmark -h 172.31.4.18 -p 6383 -a Redis_6.2.1_Sc -t set,get -e -c 4 -d 1 -n 100000000000000[cpcn@redis01 redis6383]$ netstat -anltp |grep 4.22
(Not all processes could be identified, non-owned process infowill not be shown, you would have to be root to see it all.)
tcp        0     31 172.31.4.18:6383        172.31.4.22:54100       ESTABLISHED 30548/redis-server  
tcp        0     31 172.31.4.18:6383        172.31.4.22:54098       ESTABLISHED 30548/redis-server  
tcp        0      7 172.31.4.18:6385        172.27.16.58:4422       ESTABLISHED 30559/redis-server  
tcp        0     31 172.31.4.18:6383        172.31.4.22:54102       ESTABLISHED 30548/redis-server  
tcp        0     31 172.31.4.18:6383        172.31.4.22:54096       ESTABLISHED 30548/redis-server  
tcp        0      0 172.31.4.18:16383       172.31.4.21:46224       ESTABLISHED 30548/redis-server

3、172.31.4.18 6383 切为从，老链接还在：

172.31.4.18:6383> cluster failover takeover
OK
172.31.4.18:6383> cluster nodes
83d609a16f5fc2605ff8fd698197533c6ad29334 172.31.4.18:6383@16383 myself,slave f5e774bd5cd8f5bcba53f1297015f3245dd3f18c 0 1726278372000 10 connected
401730ba6e1c2cd8b60243990f0965606edfc083 172.31.4.18:6385@16385 master - 0 1726278374588 2 connected 10923-16383
4e25e8f87029a22bbba7aa5f6fb2edba4a342da5 172.31.4.21:6384@16384 slave f5e774bd5cd8f5bcba53f1297015f3245dd3f18c 0 1726278373587 10 connected
f5e774bd5cd8f5bcba53f1297015f3245dd3f18c 172.31.4.20:6383@16383 master - 0 1726278372840 10 connected 5461-10922
23bb415a72cf125c8205be6b1de4f8f5a0a636c8 172.31.4.18:6381@16381 master - 0 1726278371000 1 connected 0-5460
9a79febbfb1d7a8f7a6ba3fb11e86b0f122f71e7 172.31.4.21:6386@16386 slave 401730ba6e1c2cd8b60243990f0965606edfc083 0 1726278369000 2 connected
602fbfe65cf83178c0b897441013ec0bae590c96 172.31.4.20:6385@16385 slave 401730ba6e1c2cd8b60243990f0965606edfc083 0 1726278371000 2 connected
3dec7d5dd8139854f766417611026b07785c0a76 172.31.4.19:6382@16382 slave 23bb415a72cf125c8205be6b1de4f8f5a0a636c8 0 1726278370582 1 connected
172.31.4.18:6383>

4、将 172.31.4.18 6383 下线，该进程还在，老链接还在。

[cpcn@redis01 redis6381]$ ./bin/redis-cli -a Redis_6.2.1_Sc --cluster del-node 172.31.4.18:6383 83d609a16f5fc2605ff8fd698197533c6ad29334
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Removing node 83d609a16f5fc2605ff8fd698197533c6ad29334 from cluster 172.31.4.18:6383
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.[cpcn@redis01 redis6383]$ ps -ef |grep redis
avahi     1161     1  0 Sep12 ?        00:00:00 avahi-daemon: running [redis01.local]
cpcn     30529     1  7 09:07 ?        00:03:50 /usr/local/redis6381/bin/redis-server 172.31.4.18:6381 [cluster]
cpcn     30559     1  8 09:07 ?        00:03:57 /usr/local/redis6385/bin/redis-server 172.31.4.18:6385 [cluster]
cpcn     31308     1  0 09:57 ?        00:00:00 /usr/local/redis6383/bin/redis-server 172.31.4.18:6383 [cluster]
cpcn     31314 29933  0 09:57 pts/1    00:00:00 grep --color=auto redis[cpcn@redis01 redis6383]$ netstat -anltp |grep 4.22
(Not all processes could be identified, non-owned process infowill not be shown, you would have to be root to see it all.)
tcp        0     31 172.31.4.18:6383        172.31.4.22:54100       ESTABLISHED 30548/redis-server  
tcp        0     31 172.31.4.18:6383        172.31.4.22:54098       ESTABLISHED 30548/redis-server  
tcp        0      7 172.31.4.18:6385        172.27.16.58:4422       ESTABLISHED 30559/redis-server  
tcp        0     31 172.31.4.18:6383        172.31.4.22:54102       ESTABLISHED 30548/redis-server  
tcp        0     31 172.31.4.18:6383        172.31.4.22:54096       ESTABLISHED 30548/redis-server  
tcp        0      0 172.31.4.18:16383       172.31.4.21:46224       ESTABLISHED 30548/redis-server  172.31.4.18:6383> cluster nodes
83d609a16f5fc2605ff8fd698197533c6ad29334 172.31.4.18:6383@16383 myself,master - 0 1726278605000 11 connected
172.31.4.18:6383>

5、下线 172.31.4.18 6383 集群信息：

172.31.4.20:6383> cluster nodes
401730ba6e1c2cd8b60243990f0965606edfc083 172.31.4.18:6385@16385 master - 0 1726277099000 2 connected 10923-16383
4e25e8f87029a22bbba7aa5f6fb2edba4a342da5 172.31.4.21:6384@16384 slave f5e774bd5cd8f5bcba53f1297015f3245dd3f18c 0 1726277098872 8 connected
23bb415a72cf125c8205be6b1de4f8f5a0a636c8 172.31.4.18:6381@16381 master - 0 1726277099874 1 connected 0-5460
602fbfe65cf83178c0b897441013ec0bae590c96 172.31.4.20:6385@16385 slave 401730ba6e1c2cd8b60243990f0965606edfc083 0 1726277098000 2 connected
9a79febbfb1d7a8f7a6ba3fb11e86b0f122f71e7 172.31.4.21:6386@16386 slave 401730ba6e1c2cd8b60243990f0965606edfc083 0 1726277097871 2 connected
f5e774bd5cd8f5bcba53f1297015f3245dd3f18c 172.31.4.20:6383@16383 myself,master - 0 1726277096000 8 connected 5461-10922
3dec7d5dd8139854f766417611026b07785c0a76 172.31.4.19:6382@16382 slave 23bb415a72cf125c8205be6b1de4f8f5a0a636c8 0 1726277097000 1 connected
172.31.4.20:6383>

此时的状态，下线172.31.4.18 6383，该进程还在，老链接还在。172.31.4.22上并发过的链接报槽位错误：

Error from server: CLUSTERDOWN Hash slot not served(overall: 0.244)
Error from server: CLUSTERDOWN Hash slot not served(overall: 0.244)
Error from server: CLUSTERDOWN Hash slot not served(overall: 0.244)

6、解决的办法，是尽快关闭下线的 redis 进程：

172.31.4.18:6383> cluster nodes
83d609a16f5fc2605ff8fd698197533c6ad29334 172.31.4.18:6383@16383 myself,master - 0 1726278605000 11 connected
172.31.4.18:6383> 
172.31.4.18:6383> 
172.31.4.18:6383> shutdown
not connected> exit
[cpcn@redis01 redis6383]$ 
[cpcn@redis01 redis6383]$ 
[cpcn@redis01 redis6383]$ ps -ef |grep redis
avahi     1161     1  0 Sep12 ?        00:00:00 avahi-daemon: running [redis01.local]
cpcn     30529     1  7 09:07 ?        00:03:49 /usr/local/redis6381/bin/redis-server 172.31.4.18:6381 [cluster]
cpcn     30559     1  8 09:07 ?        00:03:56 /usr/local/redis6385/bin/redis-server 172.31.4.18:6385 [cluster]
cpcn     31302 29933  0 09:57 pts/1    00:00:00 grep --color=auto redis

此时172.31.4.22上并发过的链接中断，4.22上的redis-benchmark进程中断，报：

报：

Error from server: CLUSTERDOWN Hash slot not served(overall: 0.237)
Error: Server closed the connection avg_msec=0.240 (overall: 0.238)
[cpcn@redis003 bin]$ [cpcn@redis003 bin]$ 
[cpcn@redis003 bin]$ ps -ef |grep redis
cpcn      67809  67655  0 10:08 pts/0    00:00:00 grep --color=auto redis
[cpcn@redis003 bin]$

附：连接日志：

[cpcn@redis003 bin]$ redis-benchmark -h 172.31.4.18 -p 6383 -a Redis_6.2.1_Sc -t set,get -e -c 4 -d 1 -n 100000000000000
Error from server: MOVED 13782 172.31.4.18:6385rall: -nan)
Error from server: MOVED 13782 172.31.4.18:6385
Error from server: MOVED 13782 172.31.4.18:6385242 (overall: 0.226)