es集群不中断实时数据更新损坏节点硬盘

一、部署三节点es集群并使用es-head查看集群状态二、新建测试索引并批量插入数据1.新建测试索引##人脸curl -XPUT http://192.168.127.171:9200/vids_data_face_20260601 -H Content-Type:application/json -d { settings:{number_of_shards:3,number_of_replicas:1}, mappings:{properties:{goods_id:{type:keyword}}}}##机动车curl -XPUT http://192.168.127.171:9200/vids_data_motorvehicle_20260601 -H Content-Type:application/json -d { settings:{number_of_shards:3,number_of_replicas:1}, mappings:{properties:{goods_id:{type:keyword}}}}##非机动车curl -XPUT http://192.168.127.171:9200/vids_data_nonmotorvehicle_20260601 -H Content-Type:application/json -d { settings:{number_of_shards:3,number_of_replicas:1}, mappings:{properties:{goods_id:{type:keyword}}}}##人体curl -XPUT http://192.168.127.171:9200/vids_data_person_20260601 -H Content-Type:application/json -d { settings:{number_of_shards:3,number_of_replicas:1}, mappings:{properties:{goods_id:{type:keyword}}}}2.脚本批量插入数据#!/bin/bash# 清空旧文件 bulk.json# 循环生成10000条for((i1;i10000;i))docat bulk.json EOF{index:{}}{goods_id:G2026$i,goods_name:测试商品$i,price:$((10i)).9}EOFdone# 导入EScurl -XPOST http://192.168.127.171:9200/vids_data_person_20260601/_bulk \-H Content-Type:application/json \--data-binary bulk.jsonecho 导入完成三、es集群不中断实时数据更换硬盘操作前准备工作1.查询集群状态##查询集群状态green为正常可写入状态curl http://192.168.127.171:9200/_cluster/health?pretty2.查看目标节点分片情况curl -X GET http://192.168.127.171:9200/_cat/shards?hindex,shard,prirep,state,nodev | grep node-92curl -X GET http://192.168.127.171:9200/_cat/shards?hindex,shard,prirep,state,nodev | grep node-92|wc -l该节点有18个分片四、执行禁用分片分配到待下线节点##exclude._ip为目标节点IPcurl -H Content-Type: application/json \ -XPUT http://192.168.127.172:9200/_cluster/settings?pretty \ -d { transient: { cluster.routing.allocation.exclude._ip: 192.168.127.171 } }五、查看分片迁移进度1.查看等待迁移的分片如果返回空数组则全部分片迁移完成curl -X GET 192.168.127.128:9200/_cluster/pending_tasks?pretty2.查看迁移进度为0时分片全部迁移完成watch -n 5 curl -s http://192.168.127.172:9200/_cat/shards | grep node-171 | wc -l3.确认要下线节点是否已清空无输出为已清空可安全下线curl -s http://192.168.127.172:9200/_cat/shards | grep node-1714.确认集群状态是否正常curl http://192.168.127.172:9200/_cluster/health?pretty页面查看成功迁移后要下线节点显示无分片分布五、停止es服务关闭该服务器进行硬盘更换##停止es服务docker-compose down 或者 docker stop elasticsearch-node-171##停止服务器poweroff查看es-head连接节点状态测试创建索引并写入数据curl -XPUT http://192.168.127.172:9200/vids_data_person_20260531 -H Content-Type:application/json -d { settings:{number_of_shards:3,number_of_replicas:1}, mappings:{properties:{goods_id:{type:keyword}}}}验证索引是否新建成功数据是否成功插入六、硬盘更换完成启用服务器、启用es服务##启用服务docker-compose up -d 或者 docker restart elasticsearch-node-171七、解除节点排除es集群自动恢复将该节点加入到集群中curl -H Content-Type: application/json \ -XPUT http://192.168.127.172:9200/_cluster/settings?pretty \ -d { transient: { cluster.routing.allocation.exclude._ip: null } }结果返回1.acknowledged: true ✅集群配置修改已成功确认生效2.transient: {}临时配置项已清空exclude._ip排除规则已被移除3.persistent: {}持久化配置无改动下面为返回内容{ acknowledged : true,persistent : { },transient : { }}##观察之前被排除节点是否开始接收分片、分片从UNASSIGNED变为STARTEDcurl http://192.168.127.172:9200/_cat/shards?vcurl http://192.168.127.172:9200/_cat/shards?v|grep node-171字段含义索引名vids_data_xxx_年月日按日期分表的业务索引人脸 / 人员 / 机动车 / 非机动车抓拍数据分片序号0/1/2分片编号0 主分片1、2 为副本分片业务索引默认 1 主 1 副p/rpprimary主分片、rreplica副本分片状态STARTED分片正常启动挂载读写可用文档数该分片存储的文档总量如6620条分片大小单分片磁盘占用如508.2kbIP 节点192.168.127.171 node-171分片所在节点八、分片数量验证1.查看分片数量有无减少curl http://192.168.127.172:9200/_cat/shards?v|grep node-171|wc -l2.查看集群状态是否正常curl http://192.168.127.172:9200/_cluster/health?pretty3.验证查看分片是否都均匀分配4.测试索引是否正常创建及数据是否正常写入curl -XPUT http://192.168.127.171:9200/vids_data_nonmotorvehicle_20260531 -H Content-Type:application/json -d { settings:{number_of_shards:3,number_of_replicas:1}, mappings:{properties:{goods_id:{type:keyword}}}}九、实施过程中更换硬盘出现系统损坏重装系统重装服务新加入集群恢复分片测试1.查看当前集群状态2.新建索引3.测试当前写入数据是否正常4.查看集群状态5.查看要下线索引分片数量6.执行节点下线操作7.查看下线节点数据迁移进度8.模拟es节点服务停止服务器关机下线9.模拟服务器故障重新安装服务查看是否正常加入集群先删除容器 模拟重装系统重新部署下线es节点重新启一个容器服务模拟重新部署并加入集群10.查看是否正常加入集群11.解除下线节点限制查看数据分片是否自动迁移到该下线节点12.查看分片数据自动迁移到下线节点进度13.查看集群状态十、如果节点上线迁移分片过程中服务器卡死执行以下命令先将分片迁回原节点先执行禁用命令curl -H Content-Type: application/json \ -XPUT http://192.168.127.172:9200/_cluster/settings?pretty \ -d { transient: { cluster.routing.allocation.exclude._ip: 192.168.127.171 } }如果执行禁用命令后没有响应还是RELOCATING或INITIALIZING再执行以下命令执行最后一步后观察分片是否正常迁回原节点# 1. 先停分配别让ES继续折腾 curl -X PUT http://任意正常节点IP:9200/_cluster/settings \ -H Content-Type: application/json \ -d {transient:{cluster.routing.allocation.enable:none}} # 2. 看当前卡住的分片 curl -s http://任意正常节点IP:9200/_cat/shards?v | grep -E RELOCATING|INITIALIZING # 3. 如果有卡住的用 reroute cancel 取消举例 curl -X POST http://任意正常节点IP:9200/_cluster/reroute \ -H Content-Type: application/json \ -d { commands: [ {cancel: {index: 卡住的索引名, shard: 0, node: 问题节点name}} ] } # 4. 然后再设 exclude最后重新 enable curl -X PUT http://任意正常节点IP:9200/_cluster/settings \ -H Content-Type: application/json \ -d {transient:{cluster.routing.allocation.enable:all}}十一、如果当前故障节点为*号主节点1.查看集群节点角色及查看es配置master为true注1master节点以上才可以执行下线命令停止节点后会自动选举迁移到其他master节点上面node.role节点的角色非常重要。用字母缩写表示m Master-eligible node有资格成为主节点d Data node存储数据的节点i Ingest node预处理文档的节点- 表示该角色不存在。例如 mdi表示同时是主节点候选、数据节点和摄取节点。#查看集群节点角色*号标识的为主节点curl -s 192.168.127.134:9200/_cat/nodes?vhname,ip,node.role,master2.执行下线命令curl -H Content-Type: application/json \ -XPUT http://192.168.127.135:9200/_cluster/settings?pretty \ -d { transient: { cluster.routing.allocation.exclude._ip: 192.168.127.134 } }3.停止故障主节点查看节点主节点会不会自动选举到其他master节点docker stop elasticsearch-node-1344.查看主节点是否自动选举到其他master节点curl -s 192.168.127.135:9200/_cat/nodes?vhname,ip,node.role,master5.故障节点上线docker restart elasticsearch-node-1346.解除节点限制curl -H Content-Type: application/json \ -XPUT http://192.168.127.134:9200/_cluster/settings?pretty \ -d { transient: { cluster.routing.allocation.exclude._ip: null } }7.查看分片迁移情况curl http://192.168.127.134:9200/_cat/shards?v|grep node-134|wc -l