[MCOL-1009] MCS Replication setup failure - mysqld failed to restart - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 1.1.1
Fix Version/s: 1.1.2
Component/s: MariaDB Server
Labels:
None
Environment:
Amazon EC2 AMI 1um / 2pm with replication enabled

Sprint:
2017-22

Description

Testing latest build of 1.1.1. Doing a postConfigure Install of a 1um / 2pm system will Replication enabled.

The postConfigure install failed with the Replication failure:

MariaDB ColumnStore Database Platform Starting, please wait .......... DONE

System Catalog Successfully Created

Run MariaDB ColumnStore Replication Setup..
ERROR: Error return in running the MariaDB ColumnStore Master replication, check /tmp/master-rep*.logs on um1

MariaDB ColumnStore Install Failed

Checking UM module logs and /tmp script log files pointed to a problem where mysqld failed to restart this this process, which caused the Replication to fail.

[mariadb-user@ip-172-30-0-135 columnstore]$ mcsmysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.soc' (2)
[mariadb-user@ip-172-30-0-135 columnstore]$ ps -ef | grep mysql
mariadb+ 4210 1 0 19:20 ? 00:00:00 /bin/sh /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld_safe --datadir=/home/mariadb-user/mariadb/columnstore/mysql/db --pid-file=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.pid --ledir=/home/mariadb-user/mariadb/columnstore/mysql//bin
mariadb+ 4395 4210 0 19:20 ? 00:00:00 /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld --basedir=/home/mariadb-user/mariadb/columnstore/mysql/ --datadir=/home/mariadb-user/mariadb/columnstore/mysql/db --plugin-dir=/home/mariadb-user/mariadb/columnstore/mysql/lib/plugin --log-error=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.err --pid-file=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.pid --socket=/home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock --port=3306
mariadb+ 7701 7618 0 19:29 pts/0 00:00:00 grep --color=auto mysql
[mariadb-user@ip-172-30-0-135 columnstore]$ ll /home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock
ls: cannot access /home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock: No such file or directory

tmp]$ cat actionMysqlCalpont.log
Shutting down MySQL.............................. ERROR!
ERROR! Failed to stop running server, so refusing to try to start.

Attachments

Activity

Ascending order - Click to sort in descending order

David Hill (Inactive) created issue - 2017-11-03 19:55

David Hill (Inactive) added a comment - 2017-11-03 19:58

(gdb) info threads
Id Target Id Frame
24 Thread 0x7f9236acd700 (LWP 4397) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
23 Thread 0x7f91ff5c2700 (LWP 4398) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
22 Thread 0x7f91fedc1700 (LWP 4399) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
21 Thread 0x7f91fe5c0700 (LWP 4400) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
20 Thread 0x7f91fddbf700 (LWP 4401) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
19 Thread 0x7f91fd5be700 (LWP 4402) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
18 Thread 0x7f91fcdbd700 (LWP 4403) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
17 Thread 0x7f91fc5bc700 (LWP 4404) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
16 Thread 0x7f91fbdbb700 (LWP 4405) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
15 Thread 0x7f91fb5ba700 (LWP 4406) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
14 Thread 0x7f91fadb9700 (LWP 4407) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
13 Thread 0x7f91fa5b8700 (LWP 4408) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
12 Thread 0x7f91f9390700 (LWP 4410) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
11 Thread 0x7f91f8b8f700 (LWP 4411) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
10 Thread 0x7f91f3fff700 (LWP 4412) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
9 Thread 0x7f91f9b91700 (LWP 4413) "mysqld" 0x00007f923b347bdd in nanosleep () from /lib64/libpthread.so.0
8 Thread 0x7f91f1ffb700 (LWP 4417) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
7 Thread 0x7f91d7fff700 (LWP 4420) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
6 Thread 0x7f91d77fe700 (LWP 4421) "mysqld" 0x00007f923b347bdd in nanosleep () from /lib64/libpthread.so.0
5 Thread 0x7f91d6ffd700 (LWP 4422) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
4 Thread 0x7f91d5ffb700 (LWP 4424) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
3 Thread 0x7f91ffa55700 (LWP 4426) "mysqld" 0x00007f923b348101 in sigwait () from /lib64/libpthread.so.0
2 Thread 0x7f91ff8d2700 (LWP 7329) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 Thread 0x7f923b7338c0 (LWP 4395) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f923bbbcacb in inline_mysql_cond_wait (src_file=0x7f923c2d6368 "/home/builder/mariadb-columnstore-server/sql/mysqld.cc", src_line=6089,
mutex=0x7f923cb37f40 <LOCK_thread_count>, that=0x7f923cb37840 <COND_thread_count>) at /home/builder/mariadb-columnstore-server/include/mysql/psi/mysql_thread.h:1149
#2 mysqld_main (argc=43, argv=0x7f923dbfa518) at /home/builder/mariadb-columnstore-server/sql/mysqld.cc:6089
#3 0x00007f923900ab35 in __libc_start_main () from /lib64/libc.so.6
#4 0x00007f923bbb13e9 in _start ()
(gdb)

David Hill (Inactive) added a comment - 2017-11-03 19:58 (gdb) info threads Id Target Id Frame 24 Thread 0x7f9236acd700 (LWP 4397) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 23 Thread 0x7f91ff5c2700 (LWP 4398) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 22 Thread 0x7f91fedc1700 (LWP 4399) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 21 Thread 0x7f91fe5c0700 (LWP 4400) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 20 Thread 0x7f91fddbf700 (LWP 4401) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 19 Thread 0x7f91fd5be700 (LWP 4402) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 18 Thread 0x7f91fcdbd700 (LWP 4403) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 17 Thread 0x7f91fc5bc700 (LWP 4404) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 16 Thread 0x7f91fbdbb700 (LWP 4405) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 15 Thread 0x7f91fb5ba700 (LWP 4406) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 14 Thread 0x7f91fadb9700 (LWP 4407) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 13 Thread 0x7f91fa5b8700 (LWP 4408) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 12 Thread 0x7f91f9390700 (LWP 4410) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 11 Thread 0x7f91f8b8f700 (LWP 4411) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 10 Thread 0x7f91f3fff700 (LWP 4412) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 9 Thread 0x7f91f9b91700 (LWP 4413) "mysqld" 0x00007f923b347bdd in nanosleep () from /lib64/libpthread.so.0 8 Thread 0x7f91f1ffb700 (LWP 4417) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 7 Thread 0x7f91d7fff700 (LWP 4420) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 6 Thread 0x7f91d77fe700 (LWP 4421) "mysqld" 0x00007f923b347bdd in nanosleep () from /lib64/libpthread.so.0 5 Thread 0x7f91d6ffd700 (LWP 4422) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 4 Thread 0x7f91d5ffb700 (LWP 4424) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 3 Thread 0x7f91ffa55700 (LWP 4426) "mysqld" 0x00007f923b348101 in sigwait () from /lib64/libpthread.so.0 2 Thread 0x7f91ff8d2700 (LWP 7329) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 1 Thread 0x7f923b7338c0 (LWP 4395) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) bt #0 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f923bbbcacb in inline_mysql_cond_wait (src_file=0x7f923c2d6368 "/home/builder/mariadb-columnstore-server/sql/mysqld.cc", src_line=6089, mutex=0x7f923cb37f40 <LOCK_thread_count>, that=0x7f923cb37840 <COND_thread_count>) at /home/builder/mariadb-columnstore-server/include/mysql/psi/mysql_thread.h:1149 #2 mysqld_main (argc=43, argv=0x7f923dbfa518) at /home/builder/mariadb-columnstore-server/sql/mysqld.cc:6089 #3 0x00007f923900ab35 in __libc_start_main () from /lib64/libc.so.6 #4 0x00007f923bbb13e9 in _start () (gdb)

David Hill (Inactive) made changes - 2017-11-03 20:37

Field	Original Value	New Value
Assignee		David Hill [ hill ]

David Hill (Inactive) added a comment - 2017-11-03 20:39

Issue is a known issue where mysqld isn't cleanly shutting down if you try after a startup to quickly. The restart is being done around the Replication setup where it updates the my.cnf file.
Change the code to where the changes to my.cnf are just made on the mysqld command line also and then DON'T do the mysqld restart.

Look at other times when mysqld restart is done and see if needed..

David Hill (Inactive) added a comment - 2017-11-03 20:39 Issue is a known issue where mysqld isn't cleanly shutting down if you try after a startup to quickly. The restart is being done around the Replication setup where it updates the my.cnf file. Change the code to where the changes to my.cnf are just made on the mysqld command line also and then DON'T do the mysqld restart. Look at other times when mysqld restart is done and see if needed..

David Hill (Inactive) made changes - 2017-11-03 20:40

Fix Version/s

1.2 [ 22554 ]

David Hill (Inactive) added a comment - 2017-11-03 20:49

NOTE: When in this case, mysqld will continue to fail to stop. so stopsystem or tryint to stop it directory will fail. Work around will be the need to FORCE kill it

kill -9 4210 4395

ps -ef | grep mysql
mariadb+ 4210 1 0 19:20 ? 00:00:00 /bin/sh /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld_safe --datadir=/home/mariadb-user/mariadb/columnstore/mysql/db --pid-file=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.pid --ledir=/home/mariadb-user/mariadb/columnstore/mysql//bin
mariadb+ 4395 4210 0 19:20 ? 00:00:03 /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld --basedir=/home/mariadb-user/mariadb/columnstore/mysql/ --datadir=/home/mariadb-user/mariadb/columnstore/mysql/db --plugin-dir=/home/mariadb-user/mariadb/columnstore/mysql/lib/plugin --log-error=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.err --pid-file=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.pid --socket=/home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock --port=3306
mariadb+ 8190 7618 0 20:44 pts/0 00:00:00 grep --color=auto mysql
[mariadb-user@ip-172-30-0-135 mysql]$ ./mysql-Columnstore stop
Shutting down MySQL.............................. ERROR!

David Hill (Inactive) added a comment - 2017-11-03 20:49 NOTE: When in this case, mysqld will continue to fail to stop. so stopsystem or tryint to stop it directory will fail. Work around will be the need to FORCE kill it kill -9 4210 4395 ps -ef | grep mysql mariadb+ 4210 1 0 19:20 ? 00:00:00 /bin/sh /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld_safe --datadir=/home/mariadb-user/mariadb/columnstore/mysql/db --pid-file=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.pid --ledir=/home/mariadb-user/mariadb/columnstore/mysql//bin mariadb+ 4395 4210 0 19:20 ? 00:00:03 /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld --basedir=/home/mariadb-user/mariadb/columnstore/mysql/ --datadir=/home/mariadb-user/mariadb/columnstore/mysql/db --plugin-dir=/home/mariadb-user/mariadb/columnstore/mysql/lib/plugin --log-error=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.err --pid-file=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.pid --socket=/home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock --port=3306 mariadb+ 8190 7618 0 20:44 pts/0 00:00:00 grep --color=auto mysql [mariadb-user@ip-172-30-0-135 mysql] $ ./mysql-Columnstore stop Shutting down MySQL.............................. ERROR!

David Thompson (Inactive) made changes - 2017-11-04 01:19

Fix Version/s		1.1.2 [ 22631 ]
Fix Version/s	1.2 [ 22554 ]
Sprint		2017-22 [ 201 ]

David Hill (Inactive) made changes - 2017-11-06 14:32

Status

Open [ 1 ]

In Progress [ 3 ]

David Hill (Inactive) made changes - 2017-11-06 23:53

Assignee	David Hill [ hill ]	Ben Thompson [ ben.thompson ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Ben Thompson (Inactive) made changes - 2017-11-07 18:14

Status

In Review [ 10002 ]

In Testing [ 10301 ]

Ben Thompson (Inactive) added a comment - 2017-11-07 18:14

Reviewed and Merged

Ben Thompson (Inactive) added a comment - 2017-11-07 18:14 Reviewed and Merged

Ben Thompson (Inactive) made changes - 2017-11-07 18:14

Assignee

Ben Thompson [ ben.thompson ]

Daniel Lee [ dleeyh ]

David Hill (Inactive) added a comment - 2017-11-07 18:48

To test: perform a multi-node install when it setup replication on mult ums or combo pm2 or local local pms.

New code will set the server-id and local query setting via the mysql console line and update the my.cnf file.
And after the replication is setup, it WILL NOT restart mysqld now at the end like in 1.1.1 and prior builds.

David Hill (Inactive) added a comment - 2017-11-07 18:48 To test: perform a multi-node install when it setup replication on mult ums or combo pm2 or local local pms. New code will set the server-id and local query setting via the mysql console line and update the my.cnf file. And after the replication is setup, it WILL NOT restart mysqld now at the end like in 1.1.1 and prior builds.

Daniel Lee (Inactive) added a comment - 2017-11-09 17:36

Build verified: 1.1.2 GitHub source
/root/columnstore/mariadb-columnstore-server
commit ed21e674cfd70db421957d0212ff7fc1835d06d5
Author: david hill <david.hill@mariadb.com>
Date: Mon Nov 6 08:41:53 2017 -0600
Update README.md
updated version
/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit e0febdcd1a88bc3fab55519bd679f93719687ccc
Merge: 1603ce9 f1dd92a
Author: david hill <david.hill@mariadb.com>
Date: Wed Nov 8 15:32:50 2017 -0600
Merge pull request #313 from mariadb-corporation/~~MCOL-989~~
~~MCOL-989~~

Daniel Lee (Inactive) added a comment - 2017-11-09 17:36 Build verified: 1.1.2 GitHub source /root/columnstore/mariadb-columnstore-server commit ed21e674cfd70db421957d0212ff7fc1835d06d5 Author: david hill <david.hill@mariadb.com> Date: Mon Nov 6 08:41:53 2017 -0600 Update README.md updated version /root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine commit e0febdcd1a88bc3fab55519bd679f93719687ccc Merge: 1603ce9 f1dd92a Author: david hill <david.hill@mariadb.com> Date: Wed Nov 8 15:32:50 2017 -0600 Merge pull request #313 from mariadb-corporation/ MCOL-989 MCOL-989

Daniel Lee (Inactive) made changes - 2017-11-09 17:36

Resolution		Fixed [ 1 ]
Status	In Testing [ 10301 ]	Closed [ 6 ]

People

Assignee:: Daniel Lee (Inactive)

Reporter:: David Hill (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 2017-11-03 19:55

Updated:: 2017-11-09 17:36

Resolved:: 2017-11-09 17:36

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.