wesdbtest1-wesdbtest4 4台机器搭建集群
OS版本：Ubuntu 24.04.1 LTS
ES版本：8.15
/etc/hosts文件信息如下
172.22.138.72 wesdbtest1
172.22.138.86 wesdbtest2
172.22.138.90 wesdbtest3
172.22.138.110 wesdbtest4
172.22.138.72 wesdbtest1.pan.com
172.22.138.86 wesdbtest2.pan.com
172.22.138.90 wesdbtest3.pan.com
172.22.138.110 wesdbtest4.pan.com

1、先在每个节点安装elasticsearch
https://www.elastic.co/downloads/elasticsearch
https://www.elastic.co/guide/en/elasticsearch/reference/8.15/deb.html

root@wesdbtest1:~# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
root@wesdbtest1:~# apt-get install apt-transport-https
root@wesdbtest1:~# echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
root@wesdbtest1:~# apt-get update && apt-get install elasticsearch

–如果不执行第一步的的话，apt-get update && apt-get install elasticsearch会有报错

W: GPG error: https://artifacts.elastic.co/packages/8.x/apt stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D27D666CD88E42B4
E: The repository 'https://artifacts.elastic.co/packages/8.x/apt stable InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

2、每个节点都启动elasticsearch，apt-get update && apt-get install elasticsearch后，/usr/lib/systemd/system/elasticsearch.service文件自动生成了

root@wesdbtest1:~# systemctl daemon-reload
root@wesdbtest1:~# systemctl enable elasticsearch.service
root@wesdbtest1:~# systemctl start elasticsearch.service

初始安装后的默认配置信息

root@wesdbtest1:~# cat /etc/elasticsearch/elasticsearch.yml |grep -v "#"
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["wesdbtest1"]
http.host: 0.0.0.0

3、每个节点都重设默认账号elastic的密码，apt-get update && apt-get install elasticsearch后，配置文件中/etc/elasticsearch/elasticsearch.yml安全认证方面的参数比如xpack.security.enabled等都已经自动配置好了，见官方文档https://www.elastic.co/guide/en/elasticsearch/reference/current/security-minimal-setup.html的描述In Elasticsearch 8.0 and later, security is enabled automatically when you start Elasticsearch for the first time.

root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
Failed to authenticate user 'elastic' against https://172.22.138.72:9200/_security/_authenticate?pretty
Possible causes include:
 * The password for the 'elastic' user has already been changed on this cluster
 * Your elasticsearch node is running against a different keystore
   This tool used the keystore at /etc/elasticsearch/elasticsearch.keystore
You can use the `elasticsearch-reset-password` CLI tool to reset the password of the 'elastic' user

–遇到如上报错，说明账号已经被设置了密码

设置elastic账号的密码

root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic

4、验证每个单节点是否安装完毕

https://wesdbtest1:9200
或
root@wesdbtest1:~# curl -XGET -uelastic:password "https://wesdbtest1:9200" -k

5、设置集群
https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html

2个节点的集群，生产环境不建议https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-small-clusters.html#high-availability-cluster-design-two-nodes
如果您有两个节点，我们建议它们都是数据节点。您还应该通过在每个不可搜索快照索引的索引上将 index.number_of_replicas 设置为 1 来确保每个分片在两个节点上冗余存储。这是默认行为，但可能会被索引模板覆盖。建议仅将两个节点之一设置为符合主节点资格，这样的话集群可以容忍从节点的丢失。如果将两个节点都设置为主节点资格，则需要两个节点才能选举master，那么如果任一节点不可用，选举就会失败，因此集群无法可靠地容忍任一节点的丢失。由于它无法适应故障，因此我们不建议在生产中部署两节点集群。

3个节点的集群https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-small-clusters.html#high-availability-cluster-design-three-nodes
建议三个节点全部是数据节点，并且每个不是可搜索快照索引的索引应该至少有一个副本,应该将每个节点配置为具有主节点资格，以便其中任何两个节点都可以进行主节点选举
应该避免仅将客户端请求发送到其中一个节点，应该负载均衡器来平衡所有三个节点上的客户端请求。

示例：4节点的Elasticsearch集群
配置方法：第一次引导先配置3个符合主节点资格的节点，即cluster.initial_master_nodes中配置3个节点，后面再把第四个节点加入集群

wesdbtest1-3的/etc/elasticsearch/elasticsearch.yml配置如下，其中节点node.name的为他们自己的名称
root@wesdbtest1:~# cat /etc/elasticsearch/elasticsearch.yml |grep -v "#"
cluster.name: wesdbtestcluster
node.name: wesdbtest1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host：0.0.0.0
http.host: 0.0.0.0
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["wesdbtest1","wesdbtest2","wesdbtest3"]
discovery.seed_hosts: ["wesdbtest1:9300","wesdbtest2:9300","wesdbtest3:9300"]
node.roles: [ master,data ]
transport.port: 9300
script.painless.regex.enabled: true

配置好后，重启wesdbtest1-3三个节点的ES服务systemctl restart elasticsearch
检查重启后的ES集群服务,发现每次执行都只能看到本节点的信息，看不到其他节点的信息

https://wesdbtest1:9200/_cat/health?v
curl -XGET -uelastic:password https://wesdbtest1:9200/_cat/health?v -k
https://wesdbtest1:9200/_cat/master?v
curl -XGET -uelastic:password https://wesdbtest1:9200/_cat/master?v -k
https://wesdbtest1:9200/_cluster/stats?pretty
curl -XGET -uelastic:password https://wesdbtest1:9200/_cluster/stats?pretty -k

而且wesdbtest1-3这3个节点的日志都有如下信息

[wesdbtest1] This node is a fully-formed single-node cluster with cluster UUID [yHThOX_0Sa-WyccipFtW7w], but it is configured as if to discover other nodes and form a multi-node cluster via the [discovery.seed_hosts=[wesdbtest1:9300, wesdbtest2:9300, wesdbtest3:9300]] setting. Fully-formed clusters do not attempt to discover other nodes, and nodes with different cluster UUIDs cannot belong to the same cluster. The cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path(s). Remove the discovery configuration to suppress this message.

尝试的但是失败了的解决方法：删除3个节点/var/lib/elasticsearch下面的目录，重启3个节点的ES服务，3个节点都正常启动，但是集群还是没能正常搭建成功，wesdbtest1-3都遇到问题如下

[wesdbtest1] failed to establish trust with server at [<unknown host>]; the server provided a certificate with subject name [CN=wesdbtest2], fingerprint [955800d548086eb4f83ed4bea52cd7efd9b5ab91], no keyUsage and no extendedKeyUsage; the certificate is valid between [2024-10-08T06:36:30Z] and [2123-09-15T06:36:30Z] (current time is [2024-10-17T03:52:02.821585809Z], certificate dates are valid); the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate does not have any subject alternative names; the certificate is issued by [CN=Elasticsearch security auto-configuration HTTP CA]; the certificate is signed by (subject [CN=Elasticsearch security auto-configuration HTTP CA] fingerprint [c333b85905842bb484548c023e8febcf08b846cb]) which is self-issued; the [CN=Elasticsearch security auto-configuration HTTP CA] certificate is not trusted in this ssl context ([xpack.security.transport.ssl (with trust configuration: StoreTrustConfig{path=certs/transport.p12, password=<non-empty>, type=PKCS12, algorithm=PKIX})]); this ssl context does trust a certificate with subject [CN=Elasticsearch security auto-configuration HTTP CA] but the trusted certificate has fingerprint [b3ca508a518abab4e15c3bc520fc26230056c2e9]

[wesdbtest1] address [172.22.138.90:9300], node [unknown] discovery result: [][172.22.138.90:9300] connect_exception: Failed execution: javax.net.ssl.SSLHandshakeException: (certificate_unknown) PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: (certificate_unknown) PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: Path does not chain with any of the trust anchors; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/discovery-troubleshooting.html
[2024-12-17T09:57:57,643][WARN ][o.e.t.TcpTransport] [wesdbtest1] exception caught on transport layer [Netty4TcpChannel{localAddress=/172.22.138.72:36338, remoteAddress=wesdbtest3/172.22.138.90:9300, profile=default}], closing connection

并且在节点wesdbtest1执行elasticsearch-reset-password修改密码的命令报错也提示无法检测cluster集群状态

root@wesdbtest1:/var/log/elasticsearch# /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic --url "https://wesdbtest1:9200"
ERROR: Failed to determine the health of the cluster. Unexpected http status [503], with exit code 65

思考报错原因：报错和certificate有关，可能安装的时候，每个节点/etc/elasticsearch/certs/下面的文件内容不一样，尝试着把wesdbtest2-3的/etc/elasticsearch/certs/下面的文件做备份，把wesdbtest1的/etc/elasticsearch/certs/下面的文件拷贝到wesdbtest2-3的路径/etc/elasticsearch/certs/，重启Elasticsearch服务，发现只有wesdbtest1的ES服务起来了，wesdbtest2-3的ES服务都起不来并且有如下报错

[wesdbtest2] fatal exception while booting Elasticsearch
org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl] - cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/transport.p12] - this is usually caused by an incorrect password; (a keystore password was provided)

[wesdbtest3] fatal exception while booting Elasticsearch
org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl] - cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/transport.p12] - this is usually caused by an incorrect password; (a keystore password was provided)

只有节点wesdbtest1的ES服务起来了，但是有如下报错

[wesdbtest1] address [172.22.138.86:9300], node [unknown] discovery result: [][172.22.138.86:9300] connect_exception: Failed execution: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: wesdbtest2/172.22.138.86:9300: Connection refused: wesdbtest2/172.22.138.86:9300: Connection refused; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/discovery-troubleshooting.html

[wesdbtest1] address [172.22.138.90:9300], node [unknown] discovery result: [][172.22.138.90:9300] connect_exception: Failed execution: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: wesdbtest3/172.22.138.90:9300: Connection refused: wesdbtest3/172.22.138.90:9300: Connection refused; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/discovery-troubleshooting.html

在节点wesdbtest1执行elasticsearch-reset-password修改密码的命令继续提示无法检测cluster集群状态

root@wesdbtest1:/var/log/elasticsearch# /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic --url "https://wesdbtest1:9200"
ERROR: Failed to determine the health of the cluster. Unexpected http status [503], with exit code 65

在节点wesdbtest1-3执行如下命令验证keystore，发现节点wesdbtest1-3显示的结果都不一样

root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-keystore list
root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password
root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-keystore show xpack.security.transport.ssl.truststore.secure_password
root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-keystore show xpack.security.transport.ssl.keystore.secure_password
root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-keystore show autoconfiguration.password_hash

最后成功的解决方法：在节点wesdbtest2-3执行如下命令重新配置keystore，密码和wesdbtest1一样，重启节点wesdbtest2-3，节点wesdbtest2-3启动成功，查看节点wesdbtest1-3的日志，发现集群已经建立好了

root@wesdbtest2:~# /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
root@wesdbtest2:~# /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
root@wesdbtest2:~# /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
root@wesdbtest2:~# /usr/share/elasticsearch/bin/elasticsearch-keystore add autoconfiguration.password_hash

再在节点wesdbtest1修改elastic账号的密码，即可使用新的elastic账号密码检查集群状态

/usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic --url "https://wesdbtest1:9200"
https://wesdbtest1:9200/_cat/health?v
curl -XGET -uelastic:password https://wesdbtest1:9200/_cat/health?v -k
https://wesdbtest1:9200/_cat/master?v
curl -XGET -uelastic:password https://wesdbtest1:9200/_cat/master?v -k
https://wesdbtest1:9200/_cluster/stats?pretty
curl -XGET -uelastic:password https://wesdbtest1:9200/_cluster/stats?pretty -k

ES cluster集群配置相关的部分参数的官方解释：
network.host：
设置此节点的HTTP traffic和transport traffic
备注：个人理解这个参数和postgresql的listen_addresses一样，elaseticsearch的network.host用于告诉elasticsearch将使用服务器中的哪个IP 进行绑定。服务器中运行的每个服务都需要绑定至少一个IP，因为服务器可以有多个IP，因此您可以使用0.0.0.0告诉elasticsearch服务绑定到了服务器上所有可用的IP。如果将network.host设置为127.0.0.1，您将只能允许与elasticsearch在同一服务器上服务使用127.0.0.1访问elasticsearch，您想从外部服务器访问elasticsearch的话，则不能设置network.host为127.0.0.1。

http.host:
设置此节点的HTTP traffic，如果network.host为默认值_local_，则http.host会覆盖network.host中的HTTP traffic的默认设置

cluster.initial_master_nodes
Starting an Elasticsearch cluster for the very first time requires the initial set of master-eligible nodes to be explicitly defined on one or more of the master-eligible nodes in the cluster. This is known as cluster bootstrapping. This is only required the first time a cluster starts up. Freshly-started nodes that are joining a running cluster obtain this information from the cluster’s elected master.After the cluster has formed, remove the cluster.initial_master_nodes setting from each node’s configuration and never set it again for this cluster. Do not configure this setting on nodes joining an existing cluster. Do not configure this setting on nodes which are restarting. Do not configure this setting when performing a full-cluster restart.You must set cluster.initial_master_nodes to the same list of nodes on each node on which it is set in order to be sure that only a single cluster forms during bootstrapping. If cluster.initial_master_nodes varies across the nodes on which it is set then you may bootstrap multiple clusters. It is usually not possible to recover from this situation without losing data.The simplest way to create a new cluster is for you to select one of your master-eligible nodes that will bootstrap itself into a single-node cluster, which all the other nodes will then join. This simple approach is not resilient to failures until the other master-eligible nodes have joined the cluster.https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html
When you start an Elasticsearch cluster for the first time, a cluster bootstrapping step determines the set of master-eligible nodes whose votes are counted in the first election. In development mode, with no discovery settings configured, this step is performed automatically by the nodes themselves.Because auto-bootstrapping is inherently unsafe, when starting a new cluster in production mode, you must explicitly list the master-eligible nodes whose votes should be counted in the very first election. You set this list using the cluster.initial_master_nodes setting on every master-eligible node. Do not configure this setting on master-ineligible nodes.After the cluster forms successfully for the first time, remove the cluster.initial_master_nodes setting from each node’s configuration and never set it again for this cluster. Do not configure this setting on nodes joining an existing cluster. Do not configure this setting on nodes which are restarting. Do not configure this setting when performing a full-cluster restart.https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#discovery-settings
Sets the initial set of master-eligible nodes in a brand-new cluster. By default this list is empty, meaning that this node expects to join a cluster that has already been bootstrapped. Remove this setting once the cluster has formed, and never set it again for this cluster. Do not configure this setting on master-ineligible nodes. Do not configure this setting on nodes joining an existing cluster. Do not configure this setting on nodes which are restarting. Do not configure this setting when performing a full-cluster restart.https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-settings.html
第一次启动ES集群需要在集群中的一个或多个符合主资格的节点上显式定义一组初始符合主资格的节点。这称为集群引导。仅在集群第一次启动时才需要这样做。新启动的节点加入正在运行的集群时，这个新启动的节点会从集群选出的主节点获取cluster.initial_master_nodes信息。集群形成后，从每个节点的配置中删除 cluster.initial_master_nodes 设置，并且不再为此集群设置它。不要在加入现有集群的节点上配置此设置。不要在正在重新启动的节点上配置此设置。执行整个集群重新启动时请勿配置此设置。你必须在每个设置cluster.initial_master_nodes参数的节点上设置相同节点列表，以确保在引导期间仅形成单个集群。如果 cluster.initial_master_nodes 在设置它的节点上有所不同，那么您可以引导多个集群。通常不可能在不丢失数据的情况下从这种情况中恢复。创建新集群的最简单方法是选择一个符合主节点资格的节点，该节点将自身引导到单节点集群中，然后所有其他节点都将加入该集群。但是在其他符合主节点资格的节点加入集群之前，这种简单的方法无法抵御故障(所以对于高可用集群的引导，应该在cluster.initial_master_nodes中配置所有符合主节点资格的节点，并且必须在每个设置cluster.initial_master_nodes参数的节点上设置相同节点列表)。
当您第一次启动 Elasticsearch 集群时，集群引导步骤会确定在第一次选举中计票的符合主节点资格的节点集。在开发模式下，如果没有配置发现设置，此步骤将由节点本身自动执行。由于自动引导本质上是不安全的，因此在生产模式下启动新集群时，您必须显式列出符合主节点资格的节点，这些节点的选票应在第一次选举中计入。您可以在每个符合主节点资格的节点上使用 cluster.initial_master_nodes 设置来设置此列表。不要在不符合主节点资格的节点上设置这个参数。集群首次成功形成后，从每个节点的配置中删除 cluster.initial_master_nodes 设置，并且不再为此集群设置它。不要在加入现有集群的节点上配置此设置。不要在正在重新启动的节点上配置此设置。执行整个集群重新启动时请勿配置此设置。
在全新集群中设置初始的符合主节点资格的节点集。默认情况下，此列表为空，这意味着该节点期望加入已引导的集群。一旦集群形成，请删除此设置，并且不再为此集群设置它。不要在不符合主节点资格的节点上配置此设置。不要在加入现有集群的节点上配置此设置。不要在正在重新启动的节点上配置此设置。执行整个集群重新启动时请勿配置此设置

discovery.seed_hosts:
Discovery is the process by which the cluster formation module finds other nodes with which to form a cluster. This process runs when you start an Elasticsearch node or when a node believes the master node failed and continues until the master node is found or a new master node is elected.This process starts with a list of seed addresses from one or more seed hosts providers, together with the addresses of any master-eligible nodes that were in the last-known cluster.https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-hosts-providers.html
Out of the box, without any network configuration, Elasticsearch will bind to the available loopback addresses and scan local ports 9300 to 9305 to connect with other nodes running on the same server. This behavior provides an auto-clustering experience without having to do any configuration.When you want to form a cluster with nodes on other hosts, use the static discovery.seed_hosts setting. This setting provides a list of other nodes in the cluster that are master-eligible and likely to be live and contactable to seed the discovery process. This setting accepts a YAML sequence or array of the addresses of all the master-eligible nodes in the cluster. Each address can be either an IP address or a hostname that resolves to one or more IP addresses via DNS.https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#discovery-settings
By default, when Elasticsearch first starts up it will try and discover other nodes running on the same host. If no elected master can be discovered within a few seconds then Elasticsearch will form a cluster that includes any other nodes that were discovered. It is useful to be able to form this cluster without any extra configuration in development mode, but this is unsuitable for production because it’s possible to form multiple clusters and lose data as a result.https://www.elastic.co/guide/en/elasticsearch/reference/8.15/_discovery_configuration_check.html
To join a cluster, an Elasticsearch node must be reachable via transport communication. To join a cluster via a non-loopback address, a node must bind transport to a non-loopback address and not be using single-node discovery. Thus, we consider an Elasticsearch node to be in development mode if it can not form a cluster with another machine via a non-loopback address, and is otherwise in production mode if it can join a cluster via non-loopback addresses.https://www.elastic.co/guide/en/elasticsearch/reference/8.15/bootstrap-checks.html
发现是集群形成模块寻找其他节点来形成集群的过程。当您启动 Elasticsearch 节点或节点认为主节点发生故障时，此过程将运行，并持续到找到主节点或选举出新的主节点为止。此过程从一个或多个种子主机提供的种子地址列表开始，以及最后已知集群中任何符合主节点资格的节点的地址。
开箱即用，无需任何网络配置，Elasticsearch 将绑定到可用的环回地址并扫描本地端口 9300 至 9305 以与同一服务器上运行的其他节点连接。此行为提供自动集群体验，无需进行任何配置。当您想要与其他主机上的节点形成集群时，请使用静态 discovery.seed_hosts 设置。此设置提供集群中其他节点的列表，这些节点符合主节点资格，并且可能处于活动状态并可联系以启动发现过程。此设置接受集群中所有符合主节点资格的节点的 YAML 序列或地址数组。每个地址可以是 IP 地址，也可以是通过 DNS 解析为一个或多个 IP 地址的主机名。
默认情况下，当 Elasticsearch 首次启动时，它会尝试发现同一主机上运行的其他节点。如果在几秒钟内无法发现选出的主节点，那么 Elasticsearch 将形成一个集群，其中包含已发现的任何其他节点。在开发模式下无需任何额外配置即可形成此集群很有用，但这不适用于生产，因为可能会形成多个集群并因此丢失数据。
要加入集群，必须可通过传输通信访问 Elasticsearch 节点。要通过非环回地址加入集群，节点必须将传输绑定到非环回地址并且不使用单节点发现。因此，如果 Elasticsearch 节点无法通过非环回地址与另一台机器形成集群，我们认为该节点处于开发模式；如果它可以通过非环回地址加入集群，则认为它处于生产模式。

script.painless.regex.enabled:
是否启用正则表达式并且不限制正则表达式的复杂性
8.15中默认值是limited，默认使用正则表达式，但限制了正则表达式的复杂性

Master-eligible node:
设置节点具有主节点资格并是个能执行CRUD的数据节点
A node that has the master role, which makes it eligible to be elected as the master node, which controls the cluster.The master node is responsible for lightweight cluster-wide actions such as creating or deleting an index, tracking which nodes are part of the cluster, and deciding which shards to allocate to which nodes. It is important for cluster health to have a stable master node.
具有master角色的节点，这使得它有资格被选举为master节点，它控制着集群。master节点负责集群范围内的轻量级操作，例如创建或删除索引、跟踪哪些节点是集群的一部分，并决定将哪些分片分配给哪些节点。拥有稳定的主节点对于集群健康非常重要。

Data node:
A node that has one of several data roles. Data nodes hold data and perform data related operations such as CRUD, search, and aggregations. A node with a generic data role can fill any of the specialized data node roles.
Data nodes hold the shards that contain the documents you have indexed. Data nodes handle data related operations like CRUD, search, and aggregations. These operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more data nodes if they are overloaded.
The main benefit of having dedicated data nodes is the separation of the master and data roles.
In a multi-tier deployment architecture, you use specialized data roles to assign data nodes to specific tiers: data_content,data_hot, data_warm, data_cold, or data_frozen. A node can belong to multiple tiers.
If you want to include a node in all tiers, or if your cluster does not use multiple tiers, then you can use the generic data role.
If you assign a node to a specific tier using a specialized data role, then you shouldn’t also assign it the generic data role. The generic data role takes precedence over specialized data roles.
Generic data nodes are included in all content tiers.
具有多个数据角色之一的节点。数据节点保存数据并执行数据相关操作，例如 CRUD、搜索和聚合。具有通用数据角色的节点可以担任任何专用数据节点角色。
数据节点保存包含您已索引的文档的分片。数据节点处理数据相关操作，例如 CRUD、搜索和聚合。这些操作是 I/O、内存和 CPU 密集型操作。监视这些资源并在过载时添加更多数据节点非常重要。
拥有专用数据节点的主要好处是主角色和数据角色的分离。
在多层部署体系结构中，您可以使用专门的数据角色将数据节点分配到特定层：data_content、data_hot、data_warm、data_cold 或 data_frozen。一个节点可以属于多个层。
如果您想要在所有层中包含一个节点，或者您的集群不使用多个层，那么您可以使用通用数据角色。
如果您使用专用数据角色将节点分配到特定层，则不应同时为其分配通用数据角色。通用数据角色优先于专用数据角色。
通用数据节点包含在所有内容层中。

Content data node：
Content data nodes are part of the content tier. Data stored in the content tier is generally a collection of items such as a product catalog or article archive. Unlike time series data, the value of thecontent remains relatively constant over time, so it doesn’t make sense to move it to a tier with different performance characteristics as it ages. Content data typically has long data retention requirements, and you want to be able to retrieve items quickly regardless of how old they are.Content tier nodes are usually optimized for query performance—​they prioritize processing power over IO throughput so they can process complex searches and aggregations and return results quickly. While they are also responsible for indexing, content data is generally not ingested at as high a rate as time series data such as logs and metrics. From a resiliency perspective the indices in this tier should be configured to use one or more replicas.The content tier is required. System indices and other indices that aren’t part of a data stream are automatically allocated to the content tier.
内容数据节点是内容层的一部分。存储在内容层中的数据通常是项目的集合，例如产品目录或文章档案。与时间序列数据不同，内容的价值随着时间的推移保持相对恒定，因此随着时间的推移将其移动到具有不同性能特征的层是没有意义的。内容数据通常具有长期数据保留要求，并且您希望能够快速检索项目，无论它们有多旧。内容层节点通常针对查询性能进行优化 - 它们优先考虑处理能力而不是 IO 吞吐量，以便可以处理复杂的搜索和聚合并快速返回结果。虽然它们还负责索引，但内容数据的摄取速度通常不如日志和指标等时间序列数据高。从弹性角度来看，这一层中的索引应配置为使用一个或多个副本。内容层是必需的。系统索引和不属于数据流一部分的其他索引会自动分配到内容层。

Hot data node：
Hot data nodes are part of the hot tier. The hot tier is the Elasticsearch entry point for time series data and holds your most-recent, most-frequently-searched time series data. Nodes in the hot tier need to be fast for both reads and writes, which requires more hardware resources and faster storage (SSDs). For resiliency, indices in the hot tier should be configured to use one or more replicas.The hot tier is required. New indices that are part of a data stream are automatically allocated to the hot tier.
热数据节点是热层的一部分。热层是 Elasticsearch 时间序列数据的入口点，保存最新、最常搜索的时间序列数据。热层中的节点需要快速读取和写入，这需要更多的硬件资源和更快的存储 (SSD)。为了实现弹性，热层中的索引应配置为使用一个或多个副本。热层是必需的。作为数据流一部分的新索引会自动分配到热层。

Warm data node：
Warm data nodes are part of the warm tier. Time series data can move to the warm tier once it is being queried less frequently than the recently-indexed data in the hot tier. The warm tier typically holds data from recent weeks. Updates are still allowed, but likely infrequent. Nodes in the warm tier generally don’t need to be as fast as those in the hot tier. For resiliency, indices in the warm tier should be configured to use one or more replicas.
温数据节点是温层的一部分。一旦查询时间序列数据的频率低于热层中最近索引的数据的频率，时间序列数据就可以移动到热层。温暖层通常保存最近几周的数据。仍然允许更新，但可能不频繁。热层中的节点通常不需要像热层中的节点那么快。为了实现弹性，热层中的索引应配置为使用一个或多个副本。

Cold data node：
Cold data nodes are part of the cold tier. When you no longer need to search time series data regularly, it can move from the warm tier to the cold tier. While still searchable, this tier is typically optimized for lower storage costs rather than search speed.For better storage savings, you can keep fully mounted indices of searchable snapshots on the cold tier. Unlike regular indices, these fully mounted indices don’t require replicas for reliability. In the event of a failure, they can recover data from the underlying snapshot instead. This potentially halves the local storage needed for the data. A snapshot repository is required to use fully mounted indices in the cold tier. Fully mounted indices are read-only.Alternatively, you can use the cold tier to store regular indices with replicas instead of using searchable snapshots. This lets you store older data on less expensive hardware but doesn’t reduce required disk space compared to the warm tier.
冷数据节点是冷层的一部分。当您不再需要定期搜索时间序列数据时，可以从暖层移动到冷层。虽然仍然可搜索，但该层通常针对较低的存储成本而不是搜索速度进行优化。为了更好地节省存储空间，您可以在冷层上保留可搜索快照的完全安装索引。与常规索引不同，这些完全安装的索引不需要副本来保证可靠性。如果发生故障，他们可以从底层快照恢复数据。这可能会将数据所需的本地存储减少一半。需要快照存储库才能在冷层中使用完全安装的索引。完全安装的索引是只读的。或者，您可以使用冷层来存储带有副本的常规索引，而不是使用可搜索的快照。这使您可以将较旧的数据存储在较便宜的硬件上，但与热层相比不会减少所需的磁盘空间。

Frozen data node：
Frozen data nodes are part of the frozen tier. Once data is no longer being queried, or being queried rarely, it may move from the cold tier to the frozen tier where it stays for the rest of its life.The frozen tier requires a snapshot repository. The frozen tier uses partially mounted indices to store and load data from a snapshot repository. This reduces local storage and operating costs while still letting you search frozen data. Because Elasticsearch must sometimes fetch frozen data from the snapshot repository, searches on the frozen tier are typically slower than on the cold tier.
冻结数据节点是冻结层的一部分。一旦数据不再被查询或很少被查询，它可能会从冷层移动到冻结层，并在其余生中保留在那里。冻结层需要快照存储库。冻结层使用部分安装的索引来存储快照存储库和从快照存储库加载数据。这降低了本地存储和运营成本，同时仍然允许您搜索冻结数据。由于 Elasticsearch 有时必须从快照存储库获取冻结数据，因此冻结层上的搜索通常比冷层上的搜索慢。

Ingest node:
A node that has the ingest role. Ingest nodes are able to apply an ingest pipeline to a document in order to transform and enrich the document before indexing. With a heavy ingest load, it makes sense to use dedicated ingest nodes and to not include the ingest role from nodes that have the master or data roles.Ingest nodes can execute pre-processing pipelines, composed of one or more ingest processors. Depending on the type of operations performed by the ingest processors and the required resources, it may make sense to have dedicated ingest nodes, that will only perform this specific task.
摄取节点：具有摄取角色的节点。摄取节点能够将摄取管道应用于文档，以便在索引之前转换和丰富文档。对于较重的摄取负载，使用专用摄取节点并且不包括具有主角色或数据角色的节点的摄取角色是有意义的。摄取节点可以执行由一个或多个摄取处理器组成的预处理管道。根据摄取处理器执行的操作类型和所需资源，拥有仅执行此特定任务的专用摄取节点可能是有意义的。

Coordinating only node：
If you take away the ability to be able to handle master duties, to hold data, and pre-process documents, then you are left with a coordinating node that can only route requests, handle the search reduce phase, and distribute bulk indexing. Essentially, coordinating only nodes behave as smart load balancers.
Coordinating only nodes can benefit large clusters by offloading the coordinating node role from data and master-eligible nodes. They join the cluster and receive the full cluster state, like every other node, and they use the cluster state to route requests directly to the appropriate place(s).
Adding too many coordinating only nodes to a cluster can increase the burden on the entire cluster because the elected master node must await acknowledgement of cluster state updates from every node! The benefit of coordinating only nodes should not be overstated — data nodes can happily serve the same purpose.
仅协调节点：如果你去掉了处理主职责、保存数据和预处理文档的能力，那么你只剩下一个协调节点，它只能路由请求、处理搜索减少阶段和分发批量索引。本质上，仅协调节点的行为就像智能负载均衡器一样。
仅协调节点可以通过从数据和符合主资格的节点卸载协调节点角色来使大型集群受益。它们加入集群并接收完整的集群状态，就像每个其他节点一样，它们使用集群状态将请求直接路由到适当的位置。
向集群添加太多仅协调节点会增加整个集群的负担，因为选出的主节点必须等待每个节点对集群状态更新的确认！仅协调节点的好处不应被夸大——“数据节点可以愉快地服务于相同的目的。

Remote-eligible node：
A remote-eligible node acts as a cross-cluster client and connects to remote clusters. Once connected, you can search remote clusters using cross-cluster search.
远程合格节点充当跨集群客户端并连接到远程集群。连接后，您可以使用跨集群搜索来搜索远程集群。

Machine learning node：
Machine learning nodes run jobs and handle machine learning API requests.
机器学习节点运行作业并处理机器学习API请求。

Transform node：
Transform nodes run transforms and handle transform API requests.
转换节点运行转换并处理转换 API 请求。

transport.port:
内部节点之间沟通端口

CORS：Configure Cross-Origin Resource Sharing 配置跨源资源共享
http.cors.enabled: true
是否启用http CORS
http.cors.allow-origin: “*”
http CORS允许哪些来源

6、新节点wesdbtest4加入到这个现有的集群，并且这个新节点的角色只是data节点，也就是这个新节点不参与master节点的任何工作
https://www.elastic.co/guide/en/elasticsearch/reference/current/add-elasticsearch-nodes.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html

节点wesdbtest1生成token

root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node --url "https://wesdbtest1:9200"
eyJ2ZXIiOiI4LjE0LjAiLCJh....ZTJmZTkxYmU1Iiwia2V5IjoiZ2psSTJKTUI2Um91bEFmaTVpSHU6akZqR3JPTF9UWWlmb2tFS3FBY3ROZyJ9

新节点wesdbtest4的配置，不配置cluster.initial_master_nodes，且配置node.roles: [ data ]

root@wesdbtest4:~# cat /etc/elasticsearch/elasticsearch.yml |grep -v "#"
cluster.name: desdbcluster
node.name: wesdbtest4
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
http.host: 0.0.0.0
discovery.seed_hosts: ["wesdbtest1:9300","wesdbtest2:9300","wesdbtest3:9300"]
node.roles: [ data ]
transport.port: 9300
script.painless.regex.enabled: true

启动新节点wesdbtest4并导入节点wesdbtest1生成的token，发现无法加入集群

root@wesdbtest4:~# systemctl start elasticsearch.service
root@wesdbtest4:~# /usr/share/elasticsearch/bin/elasticsearch --enrollment-token eyJ2ZXIiOiI4LjE0....MzI6OTIwMCJdLCJmZ3IiOiI2NjBlOTM5NzE3NmI2ZjZlNGI2NjMwMmQwNWIxYzRjNGNmMzg0ZDZhNTIxNjY4ZWZkY2ZmZGM1ZTJmZTkxYmU1Iiwia2V5IjoiZ2psSTJKTUI2Um91bEFmaTVpSHU6akZqR3JPTF9UWWlmb2tFS3FBY3ROZyJ9
ERROR: Skipping security auto configuration because it appears that the node is not starting up for the first time. The node might already be part of a cluster and this auto setup utility is designed to configure Security for new clusters only., with exit code 80

删除新节点wesdbtest4目录/var/lib/elasticsearch并重建个空目录/var/lib/elasticsearch，重新导入wesdbtest1生成的token，发现还是无法加入集群

root@wesdbtest4:~# systemctl stop elasticsearch.service
root@wesdbtest4:~# mv /var/lib/elasticsearch /var/lib/elasticsearch_bak
root@wesdbtest4:~# mkdir /var/lib/elasticsearch
root@wesdbtest4:~# chown /var/lib/elasticsearch.elasticsearch elasticsearch
root@wesdbtest4:~# systemctl start elasticsearch.service
root@wesdbtest4:~# /usr/share/elasticsearch/bin/elasticsearch --enrollment-token eyJ2ZXIiO....IyLjEzNy4xMzI6OTIwMCJdLCJmZ3IiOiI2NjBlOTM5NzE3NmI2ZjZlNGI2NjMwMmQwNWIxYzRjNGNmMzg0ZDZhNTIxNjY4ZWZkY2ZmZGM1ZTJmZTkxYmU1Iiwia2V5IjoiZ2psSTJKTUI2Um91bEFmaTVpSHU6akZqR3JPTF9UWWlmb2tFS3FBY3ROZyJ9
ERROR: Skipping security auto configuration because it appears that the node is not starting up for the first time. The node might already be part of a cluster and this auto setup utility is designed to configure Security for new clusters only., with exit code 80

重新启动节点wesdbtest1-4,发现集群还是只有wesdbtest1-3，节点wesdbtest4没有正常加入进来,节点wesdbtest4报错信息如下

[wesdbtest4] address [172.22.138.72:9300], node [unknown] discovery result: [][172.22.138.72:9300] connect_exception: Failed execution: javax.net.ssl.SSLHandshakeException: (certificate_unknown) PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: (certificate_unknown) PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: Path does not chain with any of the trust anchors; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/discovery-troubleshooting.html
[2024-12-18T06:11:26,675][WARN ][o.e.d.PeerFinder] [wesdbtest4] address [172.22.138.86:9300], node [unknown] discovery result: [][172.22.138.86:9300] connect_exception: Failed execution: javax.net.ssl.SSLHandshakeException: (certificate_unknown) PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: (certificate_unknown) PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors: Path does not chain with any of the trust anchors; for summary, see logs from org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/discovery-troubleshooting.html
[2024-12-18T06:11:26,676][WARN ][o.e.t.TcpTransport] [wesdbtest4] exception caught on transport layer [Netty4TcpChannel{localAddress=/172.22.138.110:51960, remoteAddress=wesdbtest1/172.22.138.72:9300, profile=default}], closing connection

把节点wesdbtest1目录/etc/elasticsearch/certs/下的文件拷贝到节点wesdbtest4目录/etc/elasticsearch/certs/，并重新配置wesdbtest4的keystore，密码和wesdbtest1-3的一样，再重启wesdbtest4的ES服务，自此节点wesdbtest4加入了wesdbtest1-3的集群

root@wesdbtest1:~# scp /etc/elasticsearch/certs/* root@wesdbtest4:/etc/elasticsearch/certs/
root@wesdbtest4:~# /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
root@wesdbtest4:~# /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
root@wesdbtest4:~# /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
root@wesdbtest4:~# /usr/share/elasticsearch/bin/elasticsearch-keystore add autoconfiguration.password_hash
root@wesdbtest4:~# systemctl restart elasticsearch.service

6、JVM参数的设置
https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-heap-size
https://www.elastic.co/guide/en/elasticsearch/reference/current/cat.html

查看节点的物理内存

root@wesdbtest4:~# free -m
               total        used        free      shared  buff/cache   available
Mem:            7941        5412         332           1        2505        2529
Swap:           4095           0        4095

查看节点的当前的JVM内存设置，参见heap.max项

root@wesdbtest4:~# curl -XGET -uelastic:password "https://wesdbtest4:9200/_cat/nodes?v&h=id,ip,port,heap.current,heap.percent,heap.max,ram.current,ram.percent,ram.max" -k
id   ip             port heap.current heap.percent heap.max ram.current ram.percent ram.max
WbxN 172.22.138.110 9300        1.8gb           48    3.8gb       7.4gb          96   7.7gb
BMss 172.22.138.90  9300        2.1gb           55    3.8gb       6.9gb          89   7.7gb
1bBl 172.22.138.86  9300        1.3gb           35    3.8gb       7.1gb          92   7.7gb
Vr99 172.22.138.72  9300        1.2gb           31    3.8gb       7.1gb          93   7.7gb

修改节点的JVM内存设置

root@wesdbtest4:~# vim /etc/elasticsearch/jvm.options
-Xms5g
-Xmx5g

修改JVM后需要重启ES服务才能生效

root@wesdbtest4:~# curl -XGET -uelastic:password "https://wesdbtest4:9200/_cat/nodes?v&h=name,ip,port,heap.current,heap.percent,heap.max,ram.current,ram.percent,ram.max,uptime,master,node.role" -k
name           ip             port heap.current heap.percent heap.max ram.current ram.percent ram.max uptime master node.role
wesdbtest2 172.22.138.86  9300        2.3gb           61    3.8gb       6.8gb          89   7.7gb     8d -      dm
wesdbtest1 172.22.138.72  9300      811.4mb           20    3.8gb       6.9gb          90   7.7gb     8d -      dm
wesdbtest4 172.22.138.110 9300      126.2mb            2      5gb       7.3gb          95   7.7gb   7.8d -      d
wesdbtest3 172.22.138.90  9300        1.6gb           43    3.8gb       6.7gb          87   7.7gb  13.1d *      dm

7、查看集群或节点的当前各个参数的配置信息
https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-info.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-get-settings.html

集群

curl -XGET  -uelastic:password "https://wesdbtest1:9200/_cluster/settings?include_defaults=true&pretty" -k
或
curl -XGET  -uelastic:password https://wesdbtest1:9200/_cluster/settings?include_defaults=true -k

节点elasticsearch.yml配置

curl -XGET  -uelastic:password "https://wesdbtest4:9200/_nodes/settings?pretty" -k
或
curl -XGET  -uelastic:password https://wesdbtest4:9200/_nodes/settings -k

节点jvm.options配置

curl -XGET  -uelastic:password "https://wesdbtest4:9200/_nodes/jvm?pretty" -k
或
curl -XGET  -uelastic:password https://wesdbtest4:9200/_nodes/jvm -k

8、配置指定的web网站证书
https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html

默认的web网站证书情况下，只要配置了https，使用http会报错如下

root@wesdbtest1:~# curl -XGET -uelastic:password "http://wesdbtest1:9200/_cat/health?v"
curl: (52) Empty reply from server
root@wesdbtest1:~# curl -XGET -uelastic:password  http://wesdbtest1.pan.com:9200/_cat/health?v
curl: (52) Empty reply from server
root@wesdbtest1:~# curl -XGET http://127.0.0.1:9200/_cat/health?v
curl: (52) Empty reply from server
root@wesdbtest1:~# curl -XGET http://172.22.138.72:9200/_cat/health?v
curl: (52) Empty reply from server

解决方法1:http加上s，再加上-k参数，-k参数就是–insecure不安全的意思，参见man curl

root@wesdbtest1:~# curl -XGET -uelastic:password "https://wesdbtest1:9200/_cat/health?v"
curl: (60) SSL: no alternative certificate subject name matches target host name 'wesdbtest1'
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
root@wesdbtest1:~# curl -XGET -uelastic:password "https://wesdbtest1:9200/_cat/health?v" -k
epoch      timestamp cluster              status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1735634742 08:45:42  wesdbtestcluster green           4         4     96  46    0    0        0             0                  -                100.0%

默认的web网站证书情况下，可以重设默认账号elastic的密码

root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic
This tool will reset the password of the [elastic] user.
You will be prompted to enter the password.
Please confirm that you would like to continue [y/N]y
Enter password for [elastic]:
Re-enter password for [elastic]:
Password for the [elastic] user successfully reset.

现在需要将默认的web网站证书配置为公司网站pan.com的证书

检查公司网站pan.com的证书的有效期

root@wesdbtest1:~# openssl x509 -in /etc/elasticsearch/certs/star.pan.com.pem -noout -dates
notBefore=Nov 10 00:00:00 2024 GMT
notAfter=Dec 11 23:59:59 2025 GMT

重新在每个ES节点的/etc/elasticsearch/elasticsearch.yml的配置文件中配置xpack.security.http.ssl和xpack.security.transport.ssl

原来的配置

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

–xpack.security.http.ssl.keystore.path:The path for the keystore file that contains a private key and certificate. 这个配置就是keystore file的路径，keystore文件是个加密过的文件，keystore文件的内容包括服务器网站本身的私钥和安全证书，安全证书里面有证书机构制作的服务器网站本身的公钥

新的配置，配置成如下，

xpack.security.http.ssl:
  enabled: true
  key: /etc/elasticsearch/certs/star.pan.com.key
  certificate: /etc/elasticsearch/certs/star.pan.com.pem
  certificate_authorities: ["/etc/elasticsearch/certs/Sectigo_RSA_Domain_Validation_Secure_Server_CA.pem"]
xpack.security.transport.ssl:
  enabled: true
  key: /etc/elasticsearch/certs/star.pan.com.key
  certificate: /etc/elasticsearch/certs/star.pan.com.pem
  certificate_authorities: ["/etc/elasticsearch/certs/Sectigo_RSA_Domain_Validation_Secure_Server_CA.pem"]

– xpack.security.http.ssl.key:Path to a PEM encoded file containing the private key. 服务器网站本身的私钥
– xpack.security.http.ssl.certificate:the path for the PEM encoded certificate (or certificate chain) that is associated with the key. 安全证书的路径
– xpack.security.http.ssl.certificate_authorities:List of paths to PEM encoded certificate files that should be trusted. 应信任的PEM编码证书文件的路径，PEM是证书的一种封装格式

重启ES集群4个节点,重启后，ES集群正常，重置集群的账号密码(集群状态健康的情况下，在一个节点设置账号密码成功后就会自动同步到集群的其他节点)，以便输入账号密码实现正常使用，但是前提还需要在三个节点上的/etc/hosts文件中加入如下证书涉及的dns配置，因为elasticsearch-reset-password命令没有-k参数不接受不安全的的方式，本机IP172.22.138.72或本机机器名wesdbtest1后面没有加上.pan.com，一律报错，所以这个时候必须强制使用安全证书涉及的信息即wesdbtest1后面必须加上.pan.com才能执行elasticsearch-reset-password命令

root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic --url "https://172.22.138.72:9200"
10:21:55.680 [main] WARN  org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [172.22.138.72]; the server provided a certificate with subject name [CN=*.pan.com], fingerprint [fd4014b1df2bbefa83f5538c252bc27417e48686], keyUsage [digitalSignature, keyEncipherment] and extendedKeyUsage [serverAuth, clientAuth]; the certificate is valid between [2024-01-11T00:00:00Z] and [2024-11-30T23:59:59Z] (current time is [2024-10-22T10:21:55.676564335Z], certificate dates are valid); the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate has subject alternative names [DNS:*.pan.com,DNS:pan.com]; the certificate is issued by [CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB] but the server did not provide a copy of the issuing certificate in the certificate chain; the issuing certificate with fingerprint [33e4e80807204c2b6182a3a14b591acd25b5f0db] is trusted in this ssl context ([xpack.security.http.ssl (with trust configuration: PEM-trust{/etc/elasticsearch/certs/Sectigo_RSA_Domain_Validation_Secure_Server_CA.pem})])
java.security.cert.CertificateException: No subject alternative names matching IP address 172.22.138.72 found
...
ERROR: Failed to determine the health of the cluster. , with exit code 69

root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic --url "https://wesdbtest1:9200"
10:30:36.059 [main] WARN  org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [wesdbtest1]; the server provided a certificate with subject name [CN=*.pan.com], fingerprint [fd4014b1df2bbefa83f5538c252bc27417e48686], keyUsage [digitalSignature, keyEncipherment] and extendedKeyUsage [serverAuth, clientAuth]; the certificate is valid between [2024-01-11T00:00:00Z] and [2024-11-30T23:59:59Z] (current time is [2024-10-22T10:30:36.055871303Z], certificate dates are valid); the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate has subject alternative names [DNS:*.pan.com,DNS:pan.com]; the certificate is issued by [CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB] but the server did not provide a copy of the issuing certificate in the certificate chain; the issuing certificate with fingerprint [33e4e80807204c2b6182a3a14b591acd25b5f0db] is trusted in this ssl context ([xpack.security.http.ssl (with trust configuration: PEM-trust{/etc/elasticsearch/certs/Sectigo_RSA_Domain_Validation_Secure_Server_CA.pem})])
java.security.cert.CertificateException: No subject alternative DNS name matching wesdbtest1 found.
...
ERROR: Failed to determine the health of the cluster. , with exit code 69


root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic --url "https://wesdbtest1.pan.com:9200"
This tool will reset the password of the [elastic] user.
You will be prompted to enter the password.
Please confirm that you would like to continue [y/N]y
Enter password for [elastic]:
Re-enter password for [elastic]:
Password for the [elastic] user successfully reset.

–虽然只是在节点wesdbtest1上重置了elastic的密码，但是作用是在整个集群也就是其他节点也自动重置好了

root@wesdbtest1:~# curl -XGET -uelastic -p https://wesdbtest1:9200/_cat/health?v -k
Enter host password for user 'elastic':
epoch      timestamp cluster              status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1729494685 07:11:25  wesdbtestcluster green           3         3      2   1    0    0        0             0                  -                100.0%

–curl -XGET可以wesdbtest1:9200，因为curl命令有-k参数，不需要强制使用安全证书涉及的信息即wesdbtest1后面不需要加上.pan.com即可

配置了公司网站pan.com的证书后，去除节点中/etc/hosts中pan.com试试，发现没有-k参数不接受不安全的的命令，必须强制使用安全证书涉及的信息即wesdbtest1后面必须加上.pan.com才能执行

root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic --url "https://wesdbtest1.pan.com:9200"
ERROR: Failed to determine the health of the cluster. , with exit code 69

root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic --url "https://wesdbtest1:9200"
08:21:23.980 [main] WARN  org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [DDBUD001]; the server provided a certificate with subject name [CN=*.pan.com], fingerprint [657afa0edfa925533cb9a372e596edc03e6b00cc], keyUsage [digitalSignature, keyEncipherment] and extendedKeyUsage [serverAuth, clientAuth]; the certificate is valid between [2024-11-10T00:00:00Z] and [2025-12-11T23:59:59Z] (current time is [2025-01-02T08:21:23.976711473Z], certificate dates are valid); the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate has subject alternative names [DNS:*.pan.com,DNS:pan.com]; the certificate is issued by [CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB]; the certificate is signed by (subject [CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB] fingerprint [33e4e80807204c2b6182a3a14b591acd25b5f0db] {trusted issuer}) signed by (subject [CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US] fingerprint [d89e3bd43d5d909b47a18977aa9d5ce36cee184c] {trusted issuer}) signed by (subject [CN=AAA Certificate Services,O=Comodo CA Limited,L=Salford,ST=Greater Manchester,C=GB] fingerprint [d1eb23a46d17d68fd92564c2f1f1601764d8e349] {trusted issuer}) which is self-issued; the [CN=AAA Certificate Services,O=Comodo CA Limited,L=Salford,ST=Greater Manchester,C=GB] certificate is trusted in this ssl context ([xpack.security.http.ssl (with trust configuration: Composite-Trust{JDK-trusted-certs,PEM-trust{/etc/elasticsearch/certs/star.pan.com.pem}})])
...
ERROR: Failed to determine the health of the cluster. , with exit code 69

ES集群的certificate到期后，如果ES服务不重启则ES集群可以继续使用，如果更换key和certificate文件，则必须ES集群的所有节点都更换并且重启ES服务，要不会出现集群或节点异常。自己亲自实验过的wesdbtest1-4节点集群的certificate到期，只把这个证书更换到节点wesdbtest1后重启，发现节点wesdbtest1掉出了ES集群，elastic账号居然连不上节点wesdbtest1，/usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic --url "https://wesdbtest1.pan.com:9200"重置elastic账号的密码时，居然报错ERROR: Failed to determine the health of the cluster. Unexpected http status [503], with exit code 65。把这个证书更换到其他3个节点wesdbtest2-4并重启，发现ES集群正常了，新的证书也在ES集群上生效了

配置了公司网站pan.com的证书后，如果想再加入新节点wesdbtest5到这个集群，使用elasticsearch-create-enrollment-token方式发现会有报错，因为使用的证书没有加密成keystore，现有集群的注册令牌无法生成，所以新节点wesdbtest5无法通导入现有集群的注册令牌方式加入现有集群

root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node --url "https://wesdbtest1:9200"
10:30:36.059 [main] WARN  org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [wesdbtest1]; the server provided a certificate with subject name [CN=*.pan.com], fingerprint [fd4014b1df2bbefa83f5538c252bc27417e48686], keyUsage [digitalSignature, keyEncipherment] and extendedKeyUsage [serverAuth, clientAuth]; the certificate is valid between [2024-01-11T00:00:00Z] and [2024-11-30T23:59:59Z] (current time is [2024-10-22T10:30:36.055871303Z], certificate dates are valid); the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate has subject alternative names [DNS:*.pan.com,DNS:pan.com]; the certificate is issued by [CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB] but the server did not provide a copy of the issuing certificate in the certificate chain; the issuing certificate with fingerprint [33e4e80807204c2b6182a3a14b591acd25b5f0db] is trusted in this ssl context ([xpack.security.http.ssl (with trust configuration: PEM-trust{/etc/elasticsearch/certs/Sectigo_RSA_Domain_Validation_Secure_Server_CA.pem})])
java.security.cert.CertificateException: No subject alternative DNS name matching wesdbtest1 found.
...
ERROR: Failed to determine the health of the cluster. , with exit code 69

root@wesdbtest1:~# /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node --url "https://wesdbtest1.pan.com:9200"
Unable to create enrollment token for scope [node]
ERROR: Unable to create an enrollment token. Elasticsearch node HTTP layer SSL configuration is not configured with a keystore, with exit code 73

配置新节点wesdbtest5参数如下，重启，可以正常加入集群，因为使用了指定的key和certificate文件而不再使用keystore文件，所以不再需要和节点wesdbtest4一样需要配置/usr/share/elasticsearch/bin/elasticsearch-keystore各个密码和集群现有节点的keystore密码一致，五个节点最终的配置项版本如下

root@wesdbtest5:/var/lib# cat /etc/elasticsearch/elasticsearch.yml |grep -v "#"
cluster.name: wesdbtestcluster
node.name: wesdbtest5
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  key: /etc/elasticsearch/certs/wildcard_pan_2024.key
  certificate: /etc/elasticsearch/certs/star.pan.com.pem
  certificate_authorities: ["/etc/elasticsearch/certs/Sectigo_RSA_Domain_Validation_Secure_Server_CA.pem"]

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  key: /etc/elasticsearch/certs/wildcard_pan_2024.key
  certificate: /etc/elasticsearch/certs/star.pan.com.pem
  certificate_authorities: ["/etc/elasticsearch/certs/Sectigo_RSA_Domain_Validation_Secure_Server_CA.pem"]

xpack.security.authc.token.enabled: true

discovery.seed_hosts:
   - wesdbtest1:9300
   - wesdbtest2:9300
   - wesdbtest3:9300

node.roles: [ data ]
transport.port: 9300
script.painless.regex.enabled: true

http.host: 0.0.0.0

–必须要设置discovery.seed_hosts否则日志会报错，且因为集群使用了证书的原因，而这个新加入的节点wesdbtest5因为之前作为单节点正常启动后，/var/lib/elasticsearch目录有自己作为单节点的数据，需要删除path.data配置项对应目录/var/lib/elasticsearch的所有文件，才能正常启动并正常加入集群
node validation exception
[1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch. For more information see [https://www.elastic.co/guide/en/elasticsearch/reference/8.15/bootstrap-checks.html]
bootstrap check failure [1] of [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured; for more information see [https://www.elastic.co/guide/en/elasticsearch/reference/8.15/_discovery_configuration_check.html]