一、MHA原理
1、簡(jiǎn)介:
MHA(Master High Availability)目前在MySQL高可用方面是一個(gè)相對(duì)成熟的解決方案,它由日本DeNA公司youshimaton(現(xiàn)就職于Facebook公司)開發(fā),是一套優(yōu)秀的作為MySQL高可用性環(huán)境下故障切換和主從提升的高可用軟件。在MySQL故障切換過程中,MHA能做到在0~30秒之內(nèi)自動(dòng)完成數(shù)據(jù)庫的故障切換操作,并且在進(jìn)行故障切換的過程中,MHA能在最大程度上保證數(shù)據(jù)的一致性,以達(dá)到真正意義上的高可用。
該軟件由兩部分組成:MHA Manager(管理節(jié)點(diǎn))和MHA Node(數(shù)據(jù)節(jié)點(diǎn))。MHA Manager可以單獨(dú)部署在一臺(tái)獨(dú)立的機(jī)器上管理多個(gè)master-slave集群,也可以部署在一臺(tái)slave節(jié)點(diǎn)上。MHA Node運(yùn)行在每臺(tái)MySQL服務(wù)器上,MHA Manager會(huì)定時(shí)探測(cè)集群中的master節(jié)點(diǎn),當(dāng)master出現(xiàn)故障時(shí),它可以自動(dòng)將最新數(shù)據(jù)的slave提升為新的master,然后將所有其他的slave重新指向新的master。整個(gè)故障轉(zhuǎn)移過程對(duì)應(yīng)用程序完全透明。
在MHA自動(dòng)故障切換過程中,MHA試圖從宕機(jī)的主服務(wù)器上保存二進(jìn)制日志,最大程度的保證數(shù)據(jù)的不丟失,但這并不總是可行的。例如,如果主服務(wù)器硬件故障或無法通過ssh訪問,MHA沒法保存二進(jìn)制日志,只進(jìn)行故障轉(zhuǎn)移而丟失了最新的數(shù)據(jù)。使用MySQL 5.5的半同步復(fù)制,可以大大降低數(shù)據(jù)丟失的風(fēng)險(xiǎn)。MHA可以與半同步復(fù)制結(jié)合起來。如果只有一個(gè)slave已經(jīng)收到了最新的二進(jìn)制日志,MHA可以將最新的二進(jìn)制日志應(yīng)用于其他所有的slave服務(wù)器上,因此可以保證所有節(jié)點(diǎn)的數(shù)據(jù)一致性。
目前MHA主要支持一主多從的架構(gòu),要搭建MHA,要求一個(gè)復(fù)制集群中必須最少有三臺(tái)數(shù)據(jù)庫服務(wù)器,一主二從,即一臺(tái)充當(dāng)master,一臺(tái)充當(dāng)備用master,另外一臺(tái)充當(dāng)從庫,因?yàn)橹辽傩枰_(tái)服務(wù)器,出于機(jī)器成本的考慮,淘寶也在該基礎(chǔ)上進(jìn)行了改造,目前淘寶TMHA已經(jīng)支持一主一從。
我們自己使用其實(shí)也可以使用1主1從,但是master主機(jī)宕機(jī)后無法切換,以及無法補(bǔ)全binlog。master的mysqld進(jìn)程crash后,還是可以切換成功,以及補(bǔ)全binlog的。
(1)從宕機(jī)崩潰的master保存二進(jìn)制日志事件(binlog events);
(2)識(shí)別含有最新更新的slave;
(3)應(yīng)用差異的中繼日志(relay log)到其他的slave;
(4)應(yīng)用從master保存的二進(jìn)制日志事件(binlog events);
(5)提升一個(gè)slave為新的master;
(6)使其他的slave連接新的master進(jìn)行復(fù)制;
2、MHA組成
Manager工具包主要包括以下幾個(gè)工具:
masterha_check_ssh 檢查MHA的SSH配置狀況
masterha_check_repl 檢查MySQL復(fù)制狀況
masterha_manger 啟動(dòng)MHA
masterha_check_status 檢測(cè)當(dāng)前MHA運(yùn)行狀態(tài)
masterha_master_monitor 檢測(cè)master是否宕機(jī)
masterha_master_switch 控制故障轉(zhuǎn)移(自動(dòng)或者手動(dòng))
masterha_conf_host 添加或刪除配置的server信息
Node工具包(這些工具通常由MHA Manager的腳本觸發(fā),無需人為操作)主要包括以下幾個(gè)工具:
save_binary_logs 保存和復(fù)制master的二進(jìn)制日志
Apply_diff_relay_logs 識(shí)別差異的中繼日志事件并將其差異的事件應(yīng)用于其他的slave
filter_mysqlbinlog 去除不必要的ROLLBACK事件(MHA已不再使用這個(gè)工具)
purge_relay_logs 清除中繼日志(不會(huì)阻塞SQL線程)
由于mha的軟件包需要FQ,找到了一個(gè)大神百度云盤提供的下載鏈接:https://pan.baidu.com/s/1pJ0VkSz#list/path=%2F
二、環(huán)境準(zhǔn)備
主機(jī)ip描述系統(tǒng)linux-node1192.168.56.11master以及MHA管理節(jié)點(diǎn)centos 7.4linux-node2192.168.56.12slave節(jié)點(diǎn)centos 7.4linux-node3192.168.56.13slave節(jié)點(diǎn)centos 7.4
三、MHA部署實(shí)戰(zhàn)
1、安裝依賴
[root@linux-node1 ~]# yum install -y perl-DBD-MySQL
[root@linux-node1 ~]# yum install -y perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager
[root@linux-node2 ~]# yum install -y perl-DBD-MySQL
[root@linux-node3 ~]# yum install -y perl-DBD-MySQL
#如果無法安裝,需要安裝epel源:yum install -y epel-release
2、安裝軟件
[root@linux-node1 ~]# rpm -ivh mha4mysql-node-0.56-0.el6.noarch.rpm
準(zhǔn)備中... ################################# [100%]
正在升級(jí)/安裝...
1:mha4mysql-node-0.56-0.el6 ################################# [100%]
[root@linux-node2 ~]# rpm -ivh mha4mysql-node-0.56-0.el6.noarch.rpm
準(zhǔn)備中... ################################# [100%]
正在升級(jí)/安裝...
1:mha4mysql-node-0.56-0.el6 ################################# [100%]
[root@linux-node3 ~]# rpm -ivh mha4mysql-node-0.56-0.el6.noarch.rpm
Preparing... ################################# [100%]
Updating / installing...
1:mha4mysql-node-0.56-0.el6 ################################# [100%]
3、修改/etc/my.cnf
修改服務(wù)節(jié)點(diǎn)my.cnf,這里做臨時(shí)配置,最終生效要配置到my.cnf
MySQL [(none)]> set global relay_log_purge=0;
Query OK, 0 rows affected (0.04 sec)
MySQL [(none)]> grant all privileges on *.* to mha@'192.168.56.%' identified by '123456';
Query OK, 0 rows affected, 1 warning (0.04 sec)
MySQL [(none)]> flush privileges;
Query OK, 0 rows affected (0.03 sec)
配置如下:
[client]
port = 3306
socket = /data/mysql/mysql.sock
[mysql]
no-auto-rehash
[mysqld]
user = mysql
port = 3306
socket = /data/mysql/mysql.sock
datadir = /data/mysql/data
log-bin = /data/mysql/mysql-bin
server-id = 6
#skip-grant-tables
relay_log_purge=0
4、管理節(jié)點(diǎn)配置MHA
[root@linux-node1 ~]# mkdir /etc/mha
[root@linux-node1 ~]# mkdir /var/log/mha/app1 -p
[root@linux-node1 ~]# vim /etc/mha/app1.cnf
[server default]
manager_log=/var/log/mha/app1/manager.log #設(shè)置manager的日志
manager_workdir=/var/log/mha/app1.log #設(shè)置manager的工作目錄
master_binlog_dir=/data/mysql/data #設(shè)置master 保存binlog的位置,以便MHA可以找到master的日志
user=mha #設(shè)置監(jiān)控用戶mha
password=123456 #設(shè)置mysql中root用戶的密碼,這個(gè)密碼是前文中創(chuàng)建監(jiān)控用戶的那個(gè)密碼
ping_interval=2 #設(shè)置監(jiān)控主庫,發(fā)送ping包的時(shí)間間隔,默認(rèn)是3秒,嘗試三次沒有回應(yīng)的時(shí)候自動(dòng)進(jìn)行railover
repl_password=123456 #設(shè)置復(fù)制用戶的密碼
repl_user=rep #設(shè)置復(fù)制環(huán)境中的復(fù)制用戶名
ssh_user=root #設(shè)置ssh的登錄用戶名
[server1]
hostname=192.168.56.11
port=3306
[server2]
candidate_master=1 #設(shè)置為候選master,如果設(shè)置該參數(shù)以后,發(fā)生主從切換以后將會(huì)將此從庫提升為主庫,即使這個(gè)主庫不是集群中事件最新的slave
check_repl_delay=0 #默認(rèn)情況下如果一個(gè)slave落后master 100M的relay logs的話,MHA將不會(huì)選擇該slave作為一個(gè)新的master,因?yàn)閷?duì)于這個(gè)slave的恢復(fù)需要花費(fèi)很長(zhǎng)時(shí)間,通過設(shè)置check_repl_delay=0,MHA觸發(fā)切換在選擇一個(gè)新的master的時(shí)候?qū)?huì)忽略復(fù)制延時(shí),這個(gè)參數(shù)對(duì)于設(shè)置了candidate_master=1的主機(jī)非常有用,因?yàn)檫@個(gè)候選主在切換的過程中一定是新的master
hostname=192.168.56.12
port=3306
[server3]
hostname=192.168.56.13
port=3306
5、配置SSH登錄
[root@linux-node1 ~]# ssh-keygen -t rsa
ssh-copy-id -i .ssh/id_rsa.pub root@192.168.56.11
ssh-copy-id -i .ssh/id_rsa.pub root@192.168.56.12
ssh-copy-id -i .ssh/id_rsa.pub root@192.168.56.13
[root@linux-node1 ~]# ssh 192.168.56.12
Last login: Tue Jan 9 17:03:24 2018 from 192.168.56.1
[root@linux-node2 ~]# logout
Connection to 192.168.56.12 closed.
[root@linux-node1 ~]# ssh 192.168.56.13
Last login: Tue Jan 9 21:25:59 2018 from 192.168.56.1
[root@linux-node3 ~]# logout
Connection to 192.168.56.13 closed.
[root@linux-node1 ~]# ssh 192.168.56.11
Last failed login: Wed Jan 10 17:08:07 CST 2018 from linux-node2 on ssh:notty
There were 3 failed login attempts since the last successful login.
Last login: Sat Jan 6 08:52:06 2018 from 192.168.56.1
[root@linux-node1 ~]# logout
Connection to 192.168.56.11 closed.
6、檢查SSH登錄
[root@linux-node1 ~]# masterha_check_ssh --conf=/etc/mha/app1.cnf
Wed Jan 10 17:11:00 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed Jan 10 17:11:00 2018 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Wed Jan 10 17:11:00 2018 - [info] Reading server configuration from /etc/mha/app1.cnf..
Wed Jan 10 17:11:00 2018 - [info] Starting SSH connection tests..
Wed Jan 10 17:11:03 2018 - [debug]
Wed Jan 10 17:11:00 2018 - [debug] Connecting via SSH from root@192.168.56.11(192.168.56.11:22) to root@192.168.56.12(192.168.56.12:22)..
Wed Jan 10 17:11:01 2018 - [debug] ok.
Wed Jan 10 17:11:01 2018 - [debug] Connecting via SSH from root@192.168.56.11(192.168.56.11:22) to root@192.168.56.13(192.168.56.13:22)..
Wed Jan 10 17:11:02 2018 - [debug] ok.
Wed Jan 10 17:11:03 2018 - [debug]
Wed Jan 10 17:11:01 2018 - [debug] Connecting via SSH from root@192.168.56.12(192.168.56.12:22) to root@192.168.56.11(192.168.56.11:22)..
Wed Jan 10 17:11:02 2018 - [debug] ok.
Wed Jan 10 17:11:02 2018 - [debug] Connecting via SSH from root@192.168.56.12(192.168.56.12:22) to root@192.168.56.13(192.168.56.13:22)..
Wed Jan 10 17:11:02 2018 - [debug] ok.
Wed Jan 10 17:11:03 2018 - [debug]
Wed Jan 10 17:11:02 2018 - [debug] Connecting via SSH from root@192.168.56.13(192.168.56.13:22) to root@192.168.56.11(192.168.56.11:22)..
Wed Jan 10 17:11:02 2018 - [debug] ok.
Wed Jan 10 17:11:02 2018 - [debug] Connecting via SSH from root@192.168.56.13(192.168.56.13:22) to root@192.168.56.12(192.168.56.12:22)..
Wed Jan 10 17:11:03 2018 - [debug] ok.
Wed Jan 10 17:11:03 2018 - [info] All SSH connection tests passed successfully.
7、檢查mysql replication是否配置成功
[root@linux-node1 ~]# ln -s /usr/local/mysql/bin/mysql /usr/bin/mysql
[root@linux-node1 ~]# ln -s /usr/local/mysql/bin/mysqlbinlog /usr/bin/mysqlbinlog
#必須要做軟連接,或者添加到PATH環(huán)境變量,否則會(huì)報(bào)錯(cuò)
ot@linux-node1 ~]# masterha_check_repl --conf=/etc/mha/app1.cnf
MySQL Replication Health is OK.
8、啟動(dòng)監(jiān)控
[root@linux-node1 ~]# nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master < /dev/null > /var/log/mha/app1/manager.log 2>&1 &
[1] 20640
[root@linux-node1 ~]# masterha_check_status --conf=/etc/mha/app1.cnf
app1 monitoring program is now on initialization phase(10:INITIALIZING_MONITOR). Wait for a while and try checking again.
9、測(cè)試
(1)停止主庫
[root@linux-node1 ~]# /etc/init.d/mysqld stop
Shutting down MySQL............ SUCCESS!
(2)登錄從庫查看,node2變成了主庫,node3的主庫ip變成了192.168.56.12
[root@linux-node2 ~]# mysql -uroot -p123456
Welcome to the MariaDB monitor. Commands end with ; or g.
Your MySQL connection id is 24
Server version: 5.7.18-log MySQL Community Server (GPL)
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.
MySQL [(none)]> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000005 | 154 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
[root@linux-node3 ~]# mysql -uroot -p123456
mysql> show slave statusG
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.56.12
Master_User: rep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000005
Read_Master_Log_Pos: 154
Relay_Log_File: linux-node3-relay-bin.000002
Relay_Log_Pos: 320
Relay_Master_Log_File: mysql-bin.000005
Slave_IO_Running: Yes
Slave_SQL_Running: Yes