[MCOL-1009] MCS Replication setup failure - mysqld failed to restart Created: 2017-11-03  Updated: 2017-11-09  Resolved: 2017-11-09

Status: Closed
Project: MariaDB ColumnStore
Component/s: MariaDB Server
Affects Version/s: 1.1.1
Fix Version/s: 1.1.2

Type: Bug Priority: Major
Reporter: David Hill (Inactive) Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Environment:

Amazon EC2 AMI 1um / 2pm with replication enabled

Sprint: 2017-22

 Description   

Testing latest build of 1.1.1. Doing a postConfigure Install of a 1um / 2pm system will Replication enabled.

The postConfigure install failed with the Replication failure:

MariaDB ColumnStore Database Platform Starting, please wait .......... DONE

System Catalog Successfully Created

Run MariaDB ColumnStore Replication Setup..
ERROR: Error return in running the MariaDB ColumnStore Master replication, check /tmp/master-rep*.logs on um1

MariaDB ColumnStore Install Failed

Checking UM module logs and /tmp script log files pointed to a problem where mysqld failed to restart this this process, which caused the Replication to fail.

[mariadb-user@ip-172-30-0-135 columnstore]$ mcsmysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.soc' (2)
[mariadb-user@ip-172-30-0-135 columnstore]$ ps -ef | grep mysql
mariadb+ 4210 1 0 19:20 ? 00:00:00 /bin/sh /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld_safe --datadir=/home/mariadb-user/mariadb/columnstore/mysql/db --pid-file=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.pid --ledir=/home/mariadb-user/mariadb/columnstore/mysql//bin
mariadb+ 4395 4210 0 19:20 ? 00:00:00 /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld --basedir=/home/mariadb-user/mariadb/columnstore/mysql/ --datadir=/home/mariadb-user/mariadb/columnstore/mysql/db --plugin-dir=/home/mariadb-user/mariadb/columnstore/mysql/lib/plugin --log-error=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.err --pid-file=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.pid --socket=/home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock --port=3306
mariadb+ 7701 7618 0 19:29 pts/0 00:00:00 grep --color=auto mysql
[mariadb-user@ip-172-30-0-135 columnstore]$ ll /home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock
ls: cannot access /home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock: No such file or directory

tmp]$ cat actionMysqlCalpont.log
Shutting down MySQL.............................. ERROR!
ERROR! Failed to stop running server, so refusing to try to start.

 Comments   
Comment by David Hill (Inactive) [ 2017-11-03 ]

(gdb) info threads
Id Target Id Frame
24 Thread 0x7f9236acd700 (LWP 4397) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
23 Thread 0x7f91ff5c2700 (LWP 4398) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
22 Thread 0x7f91fedc1700 (LWP 4399) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
21 Thread 0x7f91fe5c0700 (LWP 4400) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
20 Thread 0x7f91fddbf700 (LWP 4401) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
19 Thread 0x7f91fd5be700 (LWP 4402) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
18 Thread 0x7f91fcdbd700 (LWP 4403) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
17 Thread 0x7f91fc5bc700 (LWP 4404) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
16 Thread 0x7f91fbdbb700 (LWP 4405) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
15 Thread 0x7f91fb5ba700 (LWP 4406) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
14 Thread 0x7f91fadb9700 (LWP 4407) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
13 Thread 0x7f91fa5b8700 (LWP 4408) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
12 Thread 0x7f91f9390700 (LWP 4410) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
11 Thread 0x7f91f8b8f700 (LWP 4411) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
10 Thread 0x7f91f3fff700 (LWP 4412) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
9 Thread 0x7f91f9b91700 (LWP 4413) "mysqld" 0x00007f923b347bdd in nanosleep () from /lib64/libpthread.so.0
8 Thread 0x7f91f1ffb700 (LWP 4417) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
7 Thread 0x7f91d7fff700 (LWP 4420) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
6 Thread 0x7f91d77fe700 (LWP 4421) "mysqld" 0x00007f923b347bdd in nanosleep () from /lib64/libpthread.so.0
5 Thread 0x7f91d6ffd700 (LWP 4422) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
4 Thread 0x7f91d5ffb700 (LWP 4424) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
3 Thread 0x7f91ffa55700 (LWP 4426) "mysqld" 0x00007f923b348101 in sigwait () from /lib64/libpthread.so.0
2 Thread 0x7f91ff8d2700 (LWP 7329) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

  • 1 Thread 0x7f923b7338c0 (LWP 4395) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    (gdb) bt
    #0 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1 0x00007f923bbbcacb in inline_mysql_cond_wait (src_file=0x7f923c2d6368 "/home/builder/mariadb-columnstore-server/sql/mysqld.cc", src_line=6089,
    mutex=0x7f923cb37f40 <LOCK_thread_count>, that=0x7f923cb37840 <COND_thread_count>) at /home/builder/mariadb-columnstore-server/include/mysql/psi/mysql_thread.h:1149
    #2 mysqld_main (argc=43, argv=0x7f923dbfa518) at /home/builder/mariadb-columnstore-server/sql/mysqld.cc:6089
    #3 0x00007f923900ab35 in __libc_start_main () from /lib64/libc.so.6
    #4 0x00007f923bbb13e9 in _start ()
    (gdb)
Comment by David Hill (Inactive) [ 2017-11-03 ]

Issue is a known issue where mysqld isn't cleanly shutting down if you try after a startup to quickly. The restart is being done around the Replication setup where it updates the my.cnf file.
Change the code to where the changes to my.cnf are just made on the mysqld command line also and then DON'T do the mysqld restart.

Look at other times when mysqld restart is done and see if needed..

Comment by David Hill (Inactive) [ 2017-11-03 ]

NOTE: When in this case, mysqld will continue to fail to stop. so stopsystem or tryint to stop it directory will fail. Work around will be the need to FORCE kill it

  1. kill -9 4210 4395

ps -ef | grep mysql
mariadb+ 4210 1 0 19:20 ? 00:00:00 /bin/sh /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld_safe --datadir=/home/mariadb-user/mariadb/columnstore/mysql/db --pid-file=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.pid --ledir=/home/mariadb-user/mariadb/columnstore/mysql//bin
mariadb+ 4395 4210 0 19:20 ? 00:00:03 /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld --basedir=/home/mariadb-user/mariadb/columnstore/mysql/ --datadir=/home/mariadb-user/mariadb/columnstore/mysql/db --plugin-dir=/home/mariadb-user/mariadb/columnstore/mysql/lib/plugin --log-error=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.err --pid-file=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.pid --socket=/home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock --port=3306
mariadb+ 8190 7618 0 20:44 pts/0 00:00:00 grep --color=auto mysql
[mariadb-user@ip-172-30-0-135 mysql]$ ./mysql-Columnstore stop
Shutting down MySQL.............................. ERROR!

Comment by Ben Thompson (Inactive) [ 2017-11-07 ]

Reviewed and Merged

Comment by David Hill (Inactive) [ 2017-11-07 ]

To test: perform a multi-node install when it setup replication on mult ums or combo pm2 or local local pms.

New code will set the server-id and local query setting via the mysql console line and update the my.cnf file.
And after the replication is setup, it WILL NOT restart mysqld now at the end like in 1.1.1 and prior builds.

Comment by Daniel Lee (Inactive) [ 2017-11-09 ]

Build verified: 1.1.2 GitHub source
/root/columnstore/mariadb-columnstore-server
commit ed21e674cfd70db421957d0212ff7fc1835d06d5
Author: david hill <david.hill@mariadb.com>
Date: Mon Nov 6 08:41:53 2017 -0600
Update README.md
updated version
/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit e0febdcd1a88bc3fab55519bd679f93719687ccc
Merge: 1603ce9 f1dd92a
Author: david hill <david.hill@mariadb.com>
Date: Wed Nov 8 15:32:50 2017 -0600
Merge pull request #313 from mariadb-corporation/MCOL-989
MCOL-989

Generated at Thu Feb 08 02:25:29 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.