[MDEV-11823] MariaDB-Galera-server-5.5.54-1.el7.centos.x86_64 fails to start Created: 2017-01-17  Updated: 2017-01-31  Resolved: 2017-01-31

Status: Closed
Project: MariaDB Server
Component/s: Galera, Platform RedHat
Affects Version/s: 5.5.54-galera
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: BM Shukla Assignee: Nirbhay Choubey (Inactive)
Resolution: Not a Bug Votes: 0
Labels: None
Environment:

MariaDB-Galera-server RPM Upgrade issue in Cent OS 7.3


Sprint: 10.2.4-1, 10.2.4-2

 Description   

I have upgraded my MariaDB-Galera-server to MariaDB-Galera-server-5.5.54-1.el7.centos.x86_64 on 10/01/2017. OS is CentOS 7.3. My cluster was working fine before this upgrade but now it is failing. It is not logging any error in file /var/lib/mysql/dbc3.err . No error reporting in /var/log/messages even. Whenever I try to bootstrap my primary db system it creates a zero byte file in /var/lib/mysql/ as

-rw-------. 1 mysql root          0 Jan 17 09:51 wsrep_recovery.OGgRRE

yum downgrade command also denies to downgrade even. I am stuck now.
What to do? Please suggest.



 Comments   
Comment by Nirbhay Choubey (Inactive) [ 2017-01-19 ]

bms I tried to do an upgrade from 5.5.53 to 5.5.54 and it worked ok.
Could you share some specific commands that you are trying out?
Like, commands you are trying to start, stop or bootstrap the server?

Running transaction
  Updating   : MariaDB-Galera-server-5.5.54-1.el7.centos.x86_64                                                                                                                                                                    1/2 
  Cleanup    : MariaDB-Galera-server-5.5.53-1.el7.centos.x86_64                                                                                                                                                                    2/2 
Shutting down MySQL...... SUCCESS! 
Starting MySQL.170119 16:10:04 mysqld_safe Logging to '/var/lib/mysql/centos70-x86-64.err'.
170119 16:10:04 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
... SUCCESS! 
  Verifying  : MariaDB-Galera-server-5.5.54-1.el7.centos.x86_64                                                                                                                                                                    1/2 
  Verifying  : MariaDB-Galera-server-5.5.53-1.el7.centos.x86_64                                                                                                                                                                    2/2 
 
Updated:
  MariaDB-Galera-server.x86_64 0:5.5.54-1.el7.centos  

Comment by BM Shukla [ 2017-01-20 ]

Nirbhaya,
For starting first node I am using following command
/etc/init.d/mysql bootstrap
Sample output is shown below.
====
[root@dbc3 ~]# /etc/init.d/mysql bootstrap
Bootstrapping the cluster.. Starting MySQL. ERROR!
[root@dbc3 ~]#
========

Comment by Nirbhay Choubey (Inactive) [ 2017-01-20 ]

bms Could you try and share the output:

sudo sh -x /etc/init.d/mysql bootstrap

Comment by BM Shukla [ 2017-01-20 ]

[root@dbc3 ~]# sudo sh -x /etc/init.d/mysql bootstrap
+ SYSTEMD_NO_WRAP=1
+ _SYSTEMCTL_SKIP_REDIRECT=true
+ basedir=
+ datadir=
+ service_startup_timeout=900
+ startup_sleep=1
+ lockdir=/var/lock/subsys
+ lock_file_path=/var/lock/subsys/mysql
+ mysqld_pid_file_path=
+ test -z ''
+ basedir=/usr
+ bindir=/usr/bin
+ test -z ''
+ datadir=/var/lib/mysql
+ sbindir=/usr/sbin
+ libexecdir=/usr/sbin
+ datadir_set=
+ lsb_functions=/lib/lsb/init-functions
+ test -f /lib/lsb/init-functions
+ PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/bin
+ export PATH
+ mode=bootstrap
+ '[' 1 -ge 1 ']'
+ shift
+ other_args=
+ case `echo "testing\c"`,`echo -n testing` in
++ echo 'testing\c'
++ echo -n testing
+ echo_n=-n
+ echo_c=
+ test -x /usr/bin/my_print_defaults
+ print_defaults=/usr/bin/my_print_defaults
+ extra_args=
+ test -r /usr/my.cnf
+ test -r /var/lib/mysql/my.cnf
++ /usr/bin/my_print_defaults --mysqld mysql.server
+ parse_server_arguments --skip-name-resolve --query_cache_size=0 --binlog_format=ROW --default_storage_engine=innodb --innodb_autoinc_lock_mode=2 --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --wsrep_cluster_address=gcomm://172.27.16.37,172.27.16.38 --wsrep_cluster_name=csedbcluster1 --wsrep_node_address=172.27.16.53 --wsrep_node_name=dbc3 --wsrep_sst_method=rsync --wsrep_sst_auth=root:password
+ for arg in '"$@"'
+ case "$arg" in
+ for arg in '"$@"'
+ case "$arg" in
+ for arg in '"$@"'
+ case "$arg" in
+ for arg in '"$@"'
+ case "$arg" in
+ for arg in '"$@"'
+ case "$arg" in
+ for arg in '"$@"'
+ case "$arg" in
+ for arg in '"$@"'
+ case "$arg" in
+ for arg in '"$@"'
+ case "$arg" in
+ for arg in '"$@"'
+ case "$arg" in
+ for arg in '"$@"'
+ case "$arg" in
+ for arg in '"$@"'
+ case "$arg" in
+ for arg in '"$@"'
+ case "$arg" in
+ test -z ''
++ hostname
+ mysqld_pid_file_path=/var/lib/mysql/dbc3.pid
+ '[' -f /etc/default/mysql ']'
+ '[' -f /etc/sysconfig/mysql ']'
+ '[' -f /etc/conf.d/mysql ']'
+ case "$mode" in
+ echo -n 'Bootstrapping the cluster.. '
Bootstrapping the cluster.. + /etc/init.d/mysql start --wsrep-new-cluster
Starting MySQL. ERROR!
+ exit 1
[root@dbc3 ~]#

Comment by Nirbhay Choubey (Inactive) [ 2017-01-21 ]

bms Do you have error log enabled? Can you share the last few lines from it that might relate to the bootstrap failure? Or lines from syslog (in case error log is not enabled).
You could also run :
sudo /etc/init.d/mysql bootstrap --log-error=/tmp/bootstrap.err
and share /tmp/bootstrap.err file.

Comment by BM Shukla [ 2017-01-24 ]

I have run the command as advised. No file named /tmp/bootstrap.err created. Please see the output.

[root@dbc3 ~]# sudo /etc/init.d/mysql bootstrap --log-error=/tmp/bootstrap.err
Bootstrapping the cluster.. Starting MySQL. ERROR!
[root@dbc3 ~]# ls -l /tmp/bootstrap.err
ls: cannot access /tmp/bootstrap.err: No such file or directory
[root@dbc3 ~]#

Comment by Nirbhay Choubey (Inactive) [ 2017-01-25 ]

elenst Any idea why error log isn't getting created in this case?
I found and fixed an issue after 5.5.54-galera release :
https://github.com/MariaDB/server/commit/ec9a48112b3caac39fb133d524f499172942d43a
Could this issue be related to the above fix?

Comment by Elena Stepanova [ 2017-01-26 ]

nirbhay_c,

SELinux would be my first guess, Enforcing in particular. If it's set to Enforcing, try to change to Permissive and repeat.

Comment by Nirbhay Choubey (Inactive) [ 2017-01-26 ]

bms Could you check if SELinux is causing the problem?

Comment by Daniel Black [ 2017-01-27 ]

There are some selinux notes here: https://mariadb.com/kb/en/mariadb/what-to-do-if-mariadb-doesnt-start/

unrelated - https://mariadb.com/kb/en/mariadb/what-is-mariadb-55/ - 5.5 support ends in 2 and a bit months.

Comment by BM Shukla [ 2017-01-27 ]

Dear Nirbhay, Daniel Black,

Yes, Problem was due to SELinux. See the execution output below.
======================
[root@dbc3 ~]# getenforce
Enforcing
[root@dbc3 ~]# setenforce 0
[root@dbc3 ~]# /etc/init.d/mysql bootstrap
Bootstrapping the cluster.. Starting MySQL.170127 09:26:45 mysqld_safe Logging to '/var/lib/mysql/dbc3.err'.
170127 09:26:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
... SUCCESS!
[root@dbc3 ~]# getenforce
Permissive
[root@dbc3 ~]#
====================

What next? I want SELinux in enforcing mode.

Comment by Daniel Black [ 2017-01-27 ]

See the previously mentioned URL. https://github.com/MariaDB/server/tree/5.5-galera/policy/selinux and https://blogs.oracle.com/jsmyth/entry/selinux_and_mysql provide guides on getting selinux back into enforcing mode.

https://mariadb.com/kb/en/mariadb/what-to-do-if-mariadb-doesnt-start/ can be edited by you if you see ways it can be improved.

Generated at Thu Feb 08 07:52:56 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.