一、建立Consul Cluster环境
利用Consul提供的服务实现服务的注册与发现，需要建立Consul Cluster。在Consul方案中，每个提供服务的节点上都要部署和运行Consul的agent，所有运行Consul agent节点的集合构成Consul Cluster。Consul agent有两种运行模式：Server和Client。这里的Server和Client只是Consul集群层面的区分，与搭建在Cluster之上 的应用服务无关。以Server模式运行的Consul agent节点用于维护Consul集群的状态，官方建议每个Consul Cluster至少有3个或以上的运行在Server mode的Agent，Client节点不限。
Consul架构说明
注：准备至少3台机器部署consul。
二.consul集群配置
两种配置方式：《1》通过进程管理器（supervisor）配置。前提是consul已经被supervisor管理，因此推荐新手使用命令行方式配置consul集群。
《2》通过consul命令行配置（此方式配置的consul集群将不被supervisor管理，在supervisor进程管理器上的状态会显示为FATAL）。
注：若是docker启动的，建议用命令行方式配置consul集群。
《1》通过进程管理器配置
1.修改启动脚本命令（以3台机器为例）
关闭防火墙
systemctl stop firewalld
vi /etc/supervisord.d/supervisor.ini
选取一台机器配置
[program:consul] priority = 1 user = yy command = /usr/bin/consul agent -server -data- dir = "/data/yy-monitor-server/data/consul-data" -bootstrap-expect=3 #集群中最少consul实例数为3 -client= "0.0.0.0" -bind= "本机ip" # 如 -bind="10.22.0.1002" -ui stdout_logfile = /data/yy-monitor-server/log/consul .log stderr_logfile = /data/yy-monitor-server/log/consul .log autorestart = true |
其他机器配置
[program:consul] priority = 1 user = yy command = /usr/bin/consul agent -server -data- dir = "/data/yy-monitor-server/data/consul-data" -client= "0.0.0.0" -bind= "本机ip" # 如-bind="10.22.0.1001" join =“其他consul ip” # 如join=“10.22.0.1002” -ui stdout_logfile = /data/yy-monitor-server/log/consul .log stderr_logfile = /data/yy-monitor-server/log/consul .log autorestart = true |
2.重启配置
#systemctl restart supervisord
# supervisorctl restart consul
3.查看日志
tail -300f consul.log
其中一台主机上的consul日志
==> Starting Consul agent... ==> Consul agent running! Version: 'v0.9.2' Node ID: '035f828b-8db9-8cb1-a3a8-8eabf5e821c4' Node name: 'monitor-test-02.yfb.sun.cn' Datacenter: 'dc1' Server: true (bootstrap: false ) Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 10.22.0.1001 (LAN: 8301, WAN: 8302) Gossip encrypt: false , RPC-TLS: false , TLS-Incoming: false ==> Log data will now stream in as it occurs: 2018 /08/27 13:43:20 [INFO] raft: Restored from snapshot 1370-114688-1535337744933 2018 /08/27 13:43:20 [INFO] raft: Initial configuration (index=117709): [{Suffrage:Voter ID:10.22.0.1001:8300 Address:10.22.0.1001:8300} {Suffrage:Voter ID:10.22.0.1002:8300 Address:10.22.0.1002:8300} {Suffrage:Voter ID:10.22.0.1000:8300 Address:10.22.0.1000:8300}] 2018 /08/27 13:43:20 [INFO] serf: EventMemberJoin: monitor- test -02.yfb.sun.cn.dc1 10.22.0.1001 2018 /08/27 13:43:20 [INFO] raft: Node at 10.22.0.1001:8300 [Follower] entering Follower state (Leader: "" ) 2018 /08/27 13:43:20 [INFO] serf: Attempting re- join to previously known node: n2.dc1: 10.22.0.1001:8302 2018 /08/27 13:43:20 [INFO] serf: EventMemberJoin: monitor- test -02.yfb.sun.cn 10.22.0.1001 2018 /08/27 13:43:20 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp) 2018 /08/27 13:43:20 [INFO] serf: Attempting re- join to previously known node: monitor- test -01.yfb.sun.cn: 10.22.0.1000:8301 2018 /08/27 13:43:20 [INFO] consul: Adding LAN server monitor- test -02.yfb.sun.cn (Addr: tcp /10 .22.0.1001:8300) (DC: dc1) 2018 /08/27 13:43:20 [INFO] consul: Handled member- join event for server "monitor-test-02.yfb.sun.cn.dc1" in area "wan" 2018 /08/27 13:43:20 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp) 2018 /08/27 13:43:20 [INFO] agent: Started HTTP server on [::]:8500 2018 /08/27 13:43:20 [INFO] serf: Re-joined to previously known node: n2.dc1: 10.22.0.1001:8302 2018 /08/27 13:43:20 [WARN] memberlist: Refuting a suspect message (from: monitor- test -02.yfb.sun.cn) 2018 /08/27 13:43:20 [INFO] serf: EventMemberJoin: monitor- test -03.yfb.sun.cn 10.22.0.1002 2018 /08/27 13:43:20 [INFO] serf: EventMemberJoin: monitor- test -01.yfb.sun.cn 10.22.0.1000 2018 /08/27 13:43:20 [INFO] serf: Re-joined to previously known node: monitor- test -01.yfb.sun.cn: 10.22.0.1000:8301 2018 /08/27 13:43:20 [INFO] consul: Adding LAN server monitor- test -03.yfb.sun.cn (Addr: tcp /10 .22.0.1002:8300) (DC: dc1) 2018 /08/27 13:43:20 [INFO] consul: Adding LAN server monitor- test -01.yfb.sun.cn (Addr: tcp /10 .22.0.1000:8300) (DC: dc1) 2018 /08/27 13:43:20 [INFO] serf: EventMemberJoin: monitor- test -01.yfb.sun.cn.dc1 10.22.0.1000 2018 /08/27 13:43:20 [WARN] memberlist: Refuting an alive message 2018 /08/27 13:43:20 [INFO] serf: EventMemberJoin: monitor- test -03.yfb.sun.cn.dc1 10.22.0.1002 2018 /08/27 13:43:20 [INFO] consul: Handled member- join event for server "monitor-test-01.yfb.sun.cn.dc1" in area "wan" 2018 /08/27 13:43:20 [INFO] consul: Handled member- join event for server "monitor-test-03.yfb.sun.cn.dc1" in area "wan" 2018 /08/27 13:43:20 [INFO] agent: Synced node info 2018 /08/27 13:43:54 [INFO] memberlist: Suspect monitor- test -01.yfb.sun.cn has failed, no acks received 2018 /08/27 13:43:54 [INFO] serf: EventMemberUpdate: monitor- test -01.yfb.sun.cn 2018 /08/27 13:43:55 [INFO] serf: EventMemberUpdate: monitor- test -01.yfb.sun.cn.dc1 2018 /08/27 13:43:56 [INFO] serf: EventMemberUpdate: monitor- test -01.yfb.sun.cn.dc1 2018 /08/27 13:43:57 [INFO] serf: EventMemberUpdate: monitor- test -01.yfb.sun.cn ==> Failed to check for updates: Get https: //checkpoint-api .hashicorp.com /v1/check/consul ?arch=amd64&os=linux&signature=143c3fc1-0a07-2f4e-79e3-528773c5260f&version=0.9.2: dial tcp 54.221.203.30:443: i /o timeout 2018 /08/27 13:44:01 [WARN] raft: Rejecting vote request from 10.22.0.1002:8300 since we have a leader: 10.22.0.1000:8300 2018 /08/27 13:44:01 [INFO] consul: New leader elected: monitor- test -03.yfb.sun.cn |
注：更新异常可忽略，内网云桌面不能连接github。
4 查看当前节点信息
使用命令 consul info
每次启动之后，都会有一台部署有consul的机器被选举为leader节点：
其他consul节点
state = Follower |
5.ui页面
访问任意节点localhost:8500 ,如下图
《2》通过consul命令行配置
1.部署前
关闭防火墙
#因consul集群间需通信，关闭部署monitor的虚拟机的防火墙
systemctl stop firewalld |
#monitor-server为容器名，按实际情况修改 docker exec - it monitor - server / bin / bash #停止consul supervisorctl stop consul [root@monitor - test - 01 ~] # supervisorctl alertmanager RUNNING pid 22016 , uptime 21 : 28 : 23 consul RUNNING pid 11109 , uptime 0 : 00 : 07 dashboard - upgrade EXITED Aug 14 05 : 51 PM yy - echo - webhook RUNNING pid 22025 , uptime 21 : 28 : 23 grafana RUNNING pid 22015 , uptime 21 : 28 : 23 nginx RUNNING pid 22017 , uptime 21 : 28 : 23 node_exporter RUNNING pid 22020 , uptime 21 : 28 : 23 prometheus RUNNING pid 22018 , uptime 21 : 28 : 23 supervisor> stop consul consul: stopped |
server：1.1 部署主节点
运行命令：consul agent -server -bootstrap-expect 以sever方式运行的主机个数 -client 0.0.0.0 -bind 主机地址 -data-dir=/home/consul/ -ui &
[root@monitor- test -01 ~] # consul agent -server -bootstrap-expect 3 -client 0.0.0.0 -bind 10.22.0.1000 -data-dir=/home/consul/ -ui & ==> WARNING: Expect Mode enabled, expecting 3 servers ==> Starting Consul agent... ==> Consul agent running! Version: 'v0.9.2' Node ID: '8fea634e-c3d6-32ef-f8d4-4ed1a2c0ee42' Node name: 'monitor-test-01.yfb.sun.cn' Datacenter: 'dc1' Server: true (bootstrap: false ) Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 10.22.0.1000 (LAN: 8301, WAN: 8302) Gossip encrypt: false , RPC-TLS: false , TLS-Incoming: false ==> Log data will now stream in as it occurs: 2018 /08/15 16:40:26 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:10.22.0.1000:8300 Address:10.22.0.1000:8300} {Suffrage:Voter ID:10.22.0.1001:8300 Address:10.22.0.1001:8300} {Suffrage:Voter ID:10.22.0.1002:8300 Address:10.22.0.1002:8300}] 2018 /08/15 16:40:26 [INFO] serf: EventMemberJoin: monitor- test -01.yfb.sun.cn.dc1 10.22.0.1000 2018 /08/15 16:40:26 [INFO] serf: EventMemberJoin: monitor- test -01.yfb.sun.cn 10.22.0.1000 2018 /08/15 16:40:26 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp) 2018 /08/15 16:40:26 [INFO] raft: Node at 10.22.0.1000:8300 [Follower] entering Follower state (Leader: "" ) 2018 /08/15 16:40:26 [INFO] serf: Attempting re- join to previously known node: monitor- test -02.yfb.sun.cn.dc1: 10.22.0.1001:8302 2018 /08/15 16:40:26 [INFO] serf: Attempting re- join to previously known node: monitor- test -02.yfb.sun.cn: 10.22.0.1001:8301 2018 /08/15 16:40:26 [INFO] consul: Adding LAN server monitor- test -01.yfb.sun.cn (Addr: tcp /10 .22.0.1000:8300) (DC: dc1) |
1.2 其他机器运行命令：
consul agent -server -retry-join 主节点ip地址 -bind 本机ip地址 -client 0.0.0.0 -data-dir=/home/consul/ -ui &
[root@monitor- test -02 ~] # consul agent -server -retry-join 10.22.0.1000 -bind 10.22.0.1001 -client 0.0.0.0 -data-dir=/home/consul/ -ui & ==> Starting Consul agent... ==> Consul agent running! Version: 'v0.9.2' Node ID: '9e35e3c2-9226-de85-2202-2d70e516f7b7' Node name: 'monitor-test-02.yfb.sun.cn' Datacenter: 'dc1' Server: true (bootstrap: false ) Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 10.22.0.1001 (LAN: 8301, WAN: 8302) Gossip encrypt: false , RPC-TLS: false , TLS-Incoming: false ==> Log data will now stream in as it occurs: 2018 /08/15 16:32:51 [INFO] raft: Initial configuration (index=0): [] 2018 /08/15 16:32:51 [INFO] serf: EventMemberJoin: monitor- test -02.yfb.sun.cn.dc1 10.22.0.1001 2018 /08/15 16:32:51 [INFO] serf: EventMemberJoin: monitor- test -02.yfb.sun.cn 10.22.0.1001 2018 /08/15 16:32:51 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp) 2018 /08/15 16:32:51 [INFO] raft: Node at 10.22.0.1001:8300 [Follower] entering Follower state (Leader: "" ) 2018 /08/15 16:32:51 [WARN] serf: Failed to re- join any previously known node 2018 /08/15 16:32:51 [WARN] serf: Failed to re- join any previously known node 2018 /08/15 16:32:51 [INFO] consul: Adding LAN server monitor- test -02.yfb.sun.cn (Addr: tcp /10 .22.0.1001:8300) (DC: dc1) 2018 /08/15 16:32:51 [INFO] consul: Handled member- join event for server "monitor-test-02.yfb.sun.cn.dc1" in area "wan" 2018 /08/15 16:32:51 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp) 2018 /08/15 16:32:51 [INFO] agent: Started HTTP server on 127.0.0.1:8500 2018 /08/15 16:32:51 [INFO] agent: Retry join is supported for : aws azure gce softlayer 2018 /08/15 16:32:51 [INFO] agent: Joining cluster... 2018 /08/15 16:32:51 [INFO] agent: (LAN) joining: [10.22.0.1000] 2018 /08/15 16:32:51 [INFO] serf: EventMemberJoin: monitor- test -01.yfb. su |
1.6 验证集群可用性
1、 consul leave 命令可优雅退出consul集群。
2、consul members 可查看集群状况。
3、可分别访问consul ui界面进行验证。页面访问主机地址即可。
4、如果集群加入不成功，可以用 consul join 命令在非主节点机器上触发Cluster bootstrap过程：consul join 主节点ip地址。
三、代理配置
1.各主机做consul代理配置
# cat nginx.conf upstream consul.cn{ server 其他机器ip:8500 backup; server 主机ip:8500; server 其他机器ip:8500 backup; } server { # Consul UI location /consul/ { proxy_pass http: //consul .cn /ui/ ; } # Consul API location /v1/ { proxy_pass http: //consul .cn /v1/ ; add_header X-Remote-IP $remote_addr; add_header X-Server-Time $date_gmt; } } |
2.nginx重启
supervisorctl restart nginx |
3.验证配置
3.1 可以通过关闭其中一台机器的consul服务，查看日志或ui页面。
supervisorctl stop consul |
集群中任一consul宕机不会影响consul的使用，数据不丢失。
注：如果其中两台的consul宕机，在集群中需要保证至少两台机器在线，否则无法自选出leader，因此consul不可用。
附录FAQ：
-
命令行参数
-advertise：通知展现地址用来改变我们给集群中的其他节点展现的地址，一般情况下-bind地址就是展现地址
-bootstrap：用来控制一个server是否在bootstrap模式，在一个datacenter中只能有一个server处于bootstrap模式，当一个server处于bootstrap模式时，可以自己选举为raft leader。
-bootstrap-expect：在一个datacenter中期望提供的server节点数目，当该值提供的时候，consul一直等到达到指定sever数目的时候才会引导整个集群，该标记不能和bootstrap公用。
-bind：该地址用来在集群内部的通讯，集群内的所有节点到地址都必须是可达的，默认是0.0.0.0。
-client：consul绑定在哪个client地址上，这个地址提供HTTP、DNS、RPC等服务，默认是127.0.0.1。
-config-file：明确的指定要加载哪个配置文件
-config-dir：配置文件目录，里面所有以.json结尾的文件都会被加载
-data-dir：提供一个目录用来存放agent的状态，所有的agent都需要该目录，该目录必须是稳定的，系统重启后都继续存在。
-dc：该标记控制agent的datacenter的名称，默认是dc1。
-encrypt：指定secret key，使consul在通讯时进行加密，key可以通过consul keygen生成，同一个集群中的节点必须使用相同的key。
-join：加入一个已经启动的agent的ip地址，可以多次指定多个agent的地址。如果consul不能加入任何指定的地址中，则agent会启动失败。默认agent启动时不会加入任何节点。
-retry-join：和join类似，但是允许你在第一次失败后进行尝试。
-retry-interval：两次join之间的时间间隔，默认是30s。
-retry-max：尝试重复join的次数，默认是0，也就是无限次尝试。
-log-level：consul agent启动后显示的日志信息级别。默认是info，可选：trace、debug、info、warn、err。
-node：节点在集群中的名称，在一个集群中必须是唯一的，默认是该节点的主机名。
-protocol：consul使用的协议版本。
-rejoin：使consul忽略先前的离开，在再次启动后仍旧尝试加入集群中。
-server：定义agent运行在server模式，每个集群至少有一个server，建议每个集群的server不要超过5个。
-syslog：开启系统日志功能，只在linux/osx上生效。
-ui-dir:提供存放web ui资源的路径，该目录必须是可读的。新版支持直接使用-ui。
-pid-file:提供一个路径来存放pid文件，可以使用该文件进行SIGINT/SIGHUP(关闭/更新)agent。
-enable-script-checks标志设置为true，以便启用可执行外部脚本的运行状况检查。 对于生产用途，需要将ACL配置为此以控制注册任意脚本的能力。
更多参数说明可参考：
[root@monitor- test -01 ~] # consul agent -server -bootstrap-expect 3 -client 0.0.0.0 -bind 10.22.0.1000 -data-dir=/home/consul/ -ui ==> WARNING: Expect Mode enabled, expecting 3 servers ==> Starting Consul agent... ==> Consul agent running! Version: 'v0.9.2' Node ID: '8fea634e-c3d6-32ef-f8d4-4ed1a2c0ee42' Node name: 'monitor-test-01.yfb.sun.cn' Datacenter: 'dc1' Server: true (bootstrap: false ) Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 10.22.0.1000 (LAN: 8301, WAN: 8302) Gossip encrypt: false , RPC-TLS: false , TLS-Incoming: false ==> Log data will now stream in as it occurs: 2018 /08/15 16:40:26 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:10.22.0.1000:8300 Address:10.22.0.1000:8300} {Suffrage:Voter ID:10.22.0.1001:8300 Address:10.22.0.1001:8300} {Suffrage:Voter ID:10.22.0.1002:8300 Address:10.22.0.1002:8300}] 2018 /08/15 16:40:26 [INFO] serf: EventMemberJoin: monitor- test -01.yfb.sun.cn.dc1 10.22.0.1000 2018 /08/15 16:40:26 [INFO] serf: EventMemberJoin: monitor- test -01.yfb.sun.cn 10.22.0.1000 2018 /08/15 16:40:26 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp) 2018 /08/15 16:40:26 [INFO] raft: Node at 10.22.0.1000:8300 [Follower] entering Follower state (Leader: "" ) 2018 /08/15 16:40:26 [INFO] serf: Attempting re- join to previously known node: monitor- test -02.yfb.sun.cn.dc1: 10.22.0.1001:8302 2018 /08/15 16:40:26 [INFO] serf: Attempting re- join to previously known node: monitor- test -02.yfb.sun.cn: 10.22.0.1001:8301 2018 /08/15 16:40:26 [INFO] consul: Adding LAN server monitor- test -01.yfb.sun.cn (Addr: tcp /10 .22.0.1000:8300) (DC: dc1) |