mysql innodb cluster 미리보기 (remote cluster test)

MySQL InnoDB Cluster 미리 보기

MySQL Power Group (http://cafe.naver.com/mysqlp)2017.01.14유 승민 (smyoo0316@gmail.com)

MySQL InnoDB Cluster

New HA solution (available on labs)

Read-out ScalabilityCombination of three Technologies

MySQL Group Replication

MySQL Router

MySQL Shell

MySQL Group Replication The plugin that brings multi-master update everywhere (atomic broadcasting)

MySQL Router Lightweight middle ware between application and MySQL Server Read-write connection redirect (first-available) Read-only connection distribution (round-robin) Connection fail-over (redirect the failed connection to available MySQL Server)

MySQL Shell New command interface for MySQL Server. Support three type languages : javascript, python, sql Support admin command for InnoDB Cluster Must install python 2.7

MySQL (3306)PRIMARY : R/W

[MySQL Group Replica-tion]Application

MySQL (3306)SECONDARY :

R/O

MySQL (3306)SECONDARY :

R/O

[MySQL Shell]shell> mysqlshmysqlsh-js> \c root@localhost:3306mysqlsh-js> c = dba.createCluster(“Test”)mysqlsh-js> c.addInstance(‘root@server2:3306)mysqlsh-js> c.addInstance(‘root@server3:3306)mysqlsh-js> c.describe()mysqlsh-js> c.status()

[MySQL Router]R/W : 6446R/O : 6447

READ-WRITEREAD-ONLY

MySQL (3306)PRIMARY : R/W

[MySQL Group Replica-tion]Application

MySQL (3306)PRIMARY : R/W

MySQL (3306)SECONDARY :

R/O

[MySQL Shell]mysql-js> c.status();{ "clusterName": "TestCluster", "defaultReplicaSet": { "status": "Cluster is NOT tolerant to any failures.", "topology": { “server2:3306": { "address": “server2:3306", "status": "ONLINE", "role": "HA", "mode": "R/W", "leaves": { “server3:3306": { "address": “server3:3306", "status": "ONLINE", "role": "HA", "mode": "R/O", "leaves": {} }, “server1:3306": { "address": “server1:3306", "status": "OFFLINE", "role": "HA", "mode": "R/O", "leaves": {} } } } } }}

[MySQL Router]R/W : 6446R/O : 6447

READ-WRITEREAD-ONLY

PLUGINUPDATE EVERYWHEREAUTO DISTRIBUTED RECOVERYCLOUD-FRIENDLY5.7.17 GA Released (2016.12.12)

MySQL MySQL MySQL MySQL MySQL

Group Replication Cluster

< 이미지출처 : http://mysqlhighavailability.com/mysql-group-replication-hello-world/>

Atomic Broadcasting.First writer wins

rules.

SLAVESLAVE

MASTER

Replica-tion

SERVER3SERVER2

SERVER1

Group Replication

< 이미지출처 : http://mysqlhighavailability.com/order-from-chaos-member-coordination-in-group-replica-tion/>

기존 replication framework 에서 사용하는 기능Binary log caching infrastructureGTID frameworkRow-based replication

완전 새로운 기능 (Xcom)MessagingGroup membershipAuto distributed recovery mechanismAtomic broadcastingTransaction coordination- Check whether the transaction should commit or not- Propagate the changes- Conflict resolution follows the first writer wins rule

MYSQL> SHOW CREATE TABLE T1\G;*************************** 1. row *************************** Table: T1Create Table: CREATE TABLE `t1` ( `col1` int(11) NOT NULL, `col2` varchar(100) DEFAULT NULL, PRIMARY KEY (`col1`)) ENGINE=MyISAM DEFAULT CHARSET=utf81 row in set (0.00 sec)

ERROR:No query specified

MYSQL> INSERT INTO T1 VALUES (100, 'AAAA');ERROR 3098 (HY000): The table does not comply with the requirements by an external plugin.

Only Available on InnoDB storage engine (That’s why they named InnoDB Cluster)

Single PRIMARY modeBeginning with 5.7.15

group_replication_single_primary_mode=TRUE|FALSEWhich enables/disables the single primary mode

group_replication_enforce_update_everywhere_checks=FALSE|TRUEWhich enables/disables strict consistency checks for multi-master update everywhere

PRIMARY 외 다른 인스턴스들은 모두 SUPER_READ_ONLY 상태가 됨 .

(5.7.15 lab version)

Please specify an administrative MASTER key for the cluster 'test':Creating InnoDB cluster 'test' on 'root@localhost:3310'...Traceback (most recent call last): File "/usr/lib64/python2.6/runpy.py", line 122, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code exec code in run_globals File "/home/mysql/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-shell-1.0.5-labs-linux-glibc2.12-x86-64bit/bin/mysqlprovision/__-main__.py", line 32, in <module> File "/home/mysql/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-shell-1.0.5-labs-linux-glibc2.12-x86-64bit/bin/mysqlprovision/mysql_gadgets/common/options.py", line 32, in <module> File "/home/mysql/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-shell-1.0.5-labs-linux-glibc2.12-x86-64bit/bin/mysqlprovision/mysql_gadgets/common/connection_parser.py", line 254 if hostportsock[0] in {'"', "'"}: ^SyntaxError: invalid syntaxArgumentError: Dba.createCluster: Dba.createCluster: Error while executing mysqlprovision (return 1) at (shell):1:14in cluster = dba.createCluster('test'); ^mysql-js> cluster = dba.createCluster('test',);SyntaxError: Unexpected token ) at (shell):1:35in cluster = dba.createCluster('test',);

Python 2.7 로 업그레이드 먼저 해야 함 .JS 문법을 써도 python 버전 에러가 남 .내부 구현체는 모두 python 인 듯 . 파이썬 2.7 업그레이드 방법은 “여기”

ERROR: Error executing the 'start-replicaset' command: The operation could not continue due to the following requirements not being met:Some active options on server 'localhost@3310' are incompatible with Group Replication.Please restart the server 'localhost@3310' with the updated options file and try again.Option name Required Value Current Value Result------------------------------- --------------- --------------- -----binlog_checksum NONE CRC32 FAILbinlog_format ROW MIXED FAILenforce_gtid_consistency ON OFF FAILgtid_mode ON OFF FAILlog_slave_updates ON 0 FAILmaster_info_repository TABLE FILE FAILrelay_log_info_repository TABLE FILE FAILtransaction_write_set_extraction XXHASH64 OFF FAILArgumentError: Dba.createCluster: Dba.createCluster: The operation could not continue due to the following requirements not being met: at (shell):1:14in cluster = dba.createCluster('test'); ^

필수 Parameter 가 맞지 않으면 유효성 검사에서 에러남 . 아래와 같이 my.cnf 에 추가하고 DB 재시작 ######################################## # InnoDB Cluster requirements ######################################## binlog_checksum = NONE binlog_format = ROW enforce_gtid_consistency = ON gtid_mode = ON log_slave_updates = ON master_info_repository = TABLE relay_log_info_repository = TABLE transaction_write_set_extraction = XXHASH64

shell> mysqlsh // mysql shell 실행mysql-js> \c root@192.168.237.81:3310 // 클러스터를 생성할 DB 접속mysql-js> c = dba.createCluster('TestCluster'); // Group Replication whitelist 관련 “ ERROR 발생” . 하단 에러 정보 참고 ( 리모트에서만 발생 )mysql-js> \sql // SQL 모드로 변경mysql-sql> show global variables like '%whitelist%';mysql-sql> set global group_replication_ip_whitelist = ‘192.168.237.81,192.168.237.82,192.168.237.83'; // whitelist 추가mysql-sql> \js // Java Script 모드로 변경mysql-js> c = dba.createCluster('TestCluster'); // 다시 클러스터 생성 . 이때 입력하는 “클러스터패스워드”를 잘 기억해야 함 .mysql-js> c.describe(); // 클러스터 멤버 정보 확인mysql-js> c.status(); // 클러스터 인스턴스 상태 확인< ERROR LOG >2016-11-16T06:36:36.582375Z 0 [Warning] Plugin group_replication reported: '[GCS] Connection attempt from IP address 192.168.237.81 refused. Address is not in the IP whitelist.'2016-11-16T06:36:36.582528Z 0 [ERROR] Plugin group_replication reported: '[GCS] Error connecting to the local group communication engine in-stance.'2016-11-16T06:36:36.582557Z 0 [Note] Plugin group_replication reported: 'state 4127 action xa_exit'2016-11-16T06:36:36.582682Z 0 [Note] Plugin group_replication reported: 'Exiting xcom thread'2016-11-16T06:36:37.593237Z 0 [ERROR] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 13310'

MySQL> install plugin group_replication soname 'group_replication.so';MySQL> set global group_replication_ip_whitelist = '192.168.237.81,192.168.237.82,192.168.237.83'; // whitelist 미리 추가MySQL> set global group_replication_allow_local_disjoint_gtids_join = TRUE; // 하단 설명 참조 .MySQL> exit$mysqlshmysql-js> \c c root@192.168.237.81:3310 // PRIMARY 인스턴스 접속mysql-js> dba.validateInstance('root@192.168.237.83:3310'); // 필수 parameter 점검mysql-js> c.addInstance('root@192.168.237.83:3310'); // 인스턴스 추가mysql-js> c.status(); // RECOERVING 상태로 대기함 . root 유저 생성한 sql 문 충돌 때문 ( 리모트에서만 발생 )mysql-js> \sql // SQL 모드로 변경stop group_replication;drop user 'root'@'192.168.237.%'; // error log 에서 충돌 원인 확인하여 조치start group_replication;mysql-sql> \js // Java Script 모드로 변경c.status(); // ONLINE 상태로 보이면 정상group_replication_allow_local_disjoint_gtids_join인스턴스 추가 할 때 기존 클러스터 멤버 보다 최신 데이터가 감지 될 시 복구나 런타임 에러를 방지하기 위해 REJECT 함 .무시해도 된다고 확신 할 경우 TRUE 로 하고 강제로 추가 .

mysql-js> c.status();{ "clusterName": "test", "defaultReplicaSet": { "status": "Cluster tolerant to up to ONE failure.", "topology": { "192.168.237.81:3310": { "address": "192.168.237.81:3310", "status": "ONLINE", "role": "HA", "mode": "R/W", "leaves": { "192.168.237.83:3310": { "address": "192.168.237.83:3310", "status": "ONLINE", "role": "HA", "mode": "R/O", "leaves": {} }, "192.168.237.82:3310": { "address": "192.168.237.82:3310", "status": "ONLINE", "role": "HA", "mode": "R/O", "leaves": {} } } } } }}

두 번째와 동일한 방식으로 추가하고 status() 에서 모두 ON-LINE 상태이면 정상 .

# mysqlsh 세션 종류 후 클러스터 인스턴스 컨트롤 하는 법mysql-sh> c = dba.getCluster(); // 클러스터패스워드 입력mysql-sh> c.help();

show grants for 'mysql_innodb_cluster_admin'@'192.168.237.81';+-----------------------------------------------------------------------------------------------------------------+| Grants for mysql_innodb_cluster_admin@192.168.237.81 |+-----------------------------------------------------------------------------------------------------------------+| GRANT USAGE ON *.* TO 'mysql_innodb_cluster_admin'@'192.168.237.81' || GRANT ALL PRIVILEGES ON `mysql_innodb_cluster_metadata`.* TO 'mysql_innodb_cluster_admin'@'192.168.237.81' || GRANT SELECT ON `performance_schema`.`replication_group_members` TO 'mysql_innodb_cluster_admin'@'192.168.237.81' |+-----------------------------------------------------------------------------------------------------------------+3 rows in set (0.00 sec)

// MySQL Router 에서 접속할 유저 생성

GRANT USAGE ON *.* TO 'mysql_innodb_cluster_admin'@'192.168.237.105'; GRANT ALL PRIVILEGES ON `mysql_innodb_cluster_metadata`.* TO 'mysql_innodb_cluster_admin'@'192.168.237.105';GRANT SELECT ON `performance_schema`.`replication_group_members` TO 'mysql_innodb_cluster_admin'@'192.168.237.105';set password for 'mysql_innodb_cluster_admin'@'192.168.237.105' = '1'; // 클러스터패스워드와 동일해야 함 .

// Failover 테스트를 위한 일반 DB 유저 생성

create user 'smyoo'@'192.168.237.105' identified by 'smyoo';

grant select, insert, update, delete on *.* to 'smyoo'@'192.168.237.105';

$ ./mysqlrouter --bootstrap 192.168.237.81:3310 // DB 로부터 클러스터 메타 정보를 읽어서 mysqlrouter.conf 파일을 생성Please enter the administrative MASTER key for the MySQL InnoDB cluster: // 클러스터 패스워드 입력MySQL Router has now been configured for the InnoDB cluster 'test'.

The following connection information can be used to connect to the cluster.

Classic MySQL protocol connections to cluster 'test':- Read/Write Connections: localhost:6446- Read/Only Connections: localhost:6447$$ nohup ./mysqlrouter & // 백그라운드로 기동[1] 43986$ nohup: ignoring input and appending output to `nohup.out'

$$ ps -ef | grep mysqlrouterroot 43986 8680 0 15:04 pts/0 00:00:00 ./mysqlrouterroot 44050 8680 0 15:04 pts/0 00:00:00 grep mysqlrouter$

// 자동으로 생성된 mysqlrouter.conf 파일 내용[DEFAULT]plugin_folder=/root/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-router-2.1.0-labs-linux-glibc2.12-x86-64bit/lib/mysqlrouter# logging_folder=/root/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-router-2.1.0-labs-linux-glibc2.12-x86-64bit/bin/../

[logger]level = INFO

[metadata_cache]bootstrap_server_addresses=mysql://192.168.237:3310,mysql://192.168.237.82:3310,mysql://192.168.237.83:3310user=mysql_innodb_cluster_readerpassword=,wY,JbZ9Saxf%}(Smetadata_cluster=testttl=300metadata_replicaset=default

[routing:default_rw]bind_port=6446destinations=metadata-cache:///default?role=PRIMARYmode=read-write

[routing:default_ro]bind_port=6447destinations=metadata-cache:///default?role=SECONDARYmode=read-only

$ mysqlsh --uri smyoo@localhost:6446 // MySQL Router 로 PRIMARY 인스턴스로 접속

mysql-js> \sql // SQL 모드로 변경Switching to SQL mode... Commands end with ;mysql-sql>mysql-sql> select @@hostname; // PRIMARY 인스턴스 접속 확인+------------+| @@hostname |+------------+| mytest1 |+------------+1 row in set (0.00 sec)mysql-sql>mysql-sql> select @@hostname;ERROR: 2013 (HY000): Lost connection to MySQL server during query // PRIMARY 인스턴스 DOWN 감지 ( 약 1 초 정도 딜레이 )The global session got disconnected. // 접속 실패한 세션은 그냥 사라짐 . 자동으로 재 접속 하지 않는다 .Attempting to reconnect to 'smyoo@localhost:6446'...The global session was successfully reconnected.mysql-sql>mysql-sql> select @@hostname; // Failover 된 것 확인+------------+| @@hostname |+------------+| mytest2 |+------------+1 row in set (0.00 sec)

MYSQL> SELECT * FROM PERFORMANCE_SCHEMA.REPLICATION_GROUP_MEMBERS ORDER BY MEMBER_HOST;+---------------------------+--------------------------------------+-------------+-------------+--------------+| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |+---------------------------+--------------------------------------+-------------+-------------+--------------+| group_replication_applier | dae27541-b2e5-11e6-a22c-ac162d76e1c4 | mytest1 | 3310 | ONLINE || group_replication_applier | deeb5d42-b2e5-11e6-9ef5-002655e4a549 | mytest2 | 3310 | ONLINE || group_replication_applier | e2449c17-b2e5-11e6-b72a-002655e4a51d | mytest3 | 3310 | ONLINE |+---------------------------+--------------------------------------+-------------+-------------+--------------+3 rows in set (0.00 sec)

MYSQL> SELECT * FROM PERFORMANCE_SCHEMA.REPLICATION_GROUP_MEMBER_STATS\G;*************************** 1. row *************************** CHANNEL_NAME: group_replication_applier VIEW_ID: 14800615308848506:7 MEMBER_ID: e2449c17-b2e5-11e6-b72a-002655e4a51d COUNT_TRANSACTIONS_IN_QUEUE: 0 큐에 누적된 트랜잭션 수 . ( 동기화 지연 정도 ) 0 을 유지하는 것이 바람직 . COUNT_TRANSACTIONS_CHECKED: 1217439 COUNT_CONFLICTS_DETECTED: 0COUNT_TRANSACTIONS_ROWS_VALIDATING: 0TRANSACTIONS_COMMITTED_ALL_MEMBERS: dcd7ae4b-b2e6-11e6-b4a5-002655e4a51d:1-1217444,e2449c17-b2e5-11e6-b72a-002655e4a51d:1-22 LAST_CONFLICT_FREE_TRANSACTION: dcd7ae4b-b2e6-11e6-b4a5-002655e4a51d:12174441 row in set (0.00 sec)

ERROR:No query specified

(with Grafana & In-fluxDB)

1 번 노드가 빠진 상태에서 다시 추가하기 전 큰 테이블(1 천만건 ) 미리 생성해 두고 sysbench 트래픽이 유입되는 상태에서 노드 추가 .

노드 추가 시 발생하는 자동 분산 복구 시 “아직은” 제대로 동작 안함 .DML 트래픽을 처리 못하고 트랜잭션 큐 지연 발생 .RECOVERING 상태에서 더 이상 진행이 안됨 .

요약완전히 다른 동기화 메커니즘구축 용이 , 편리한 운영 , 유연한 확장 ( 을 기대 합니다 )

현재상황 기본적인 기능은 무난하게 동작함 . (Single PRIMARY 에서의 동기화 처리 속도 , Failover)

Remote 구성이 “아직은” 매끄럽지 않음 .

mysqlsh 은 조금 익숙해지면 편리함 .

온라인 자동 복구는 “아직은” 제대로 동작 안 함 .

제약사항InnoDB 만 된다 .

ROW 포멧GTID 필수