Nirbhay Choubey (Inactive) added a comment - 2015-12-29 16:36 I tried some permutations on centos70-x86-64 and couldn't reproduce this. Could you try running the failing command under sh -x? I need to see what going on with the pid file. [root@centos70-x86-64 ~]# service mysql start Starting mysql (via systemctl): [ OK ] [root@centos70-x86-64 ~]# mysqladmin -uroot ping mysqld is alive [root@centos70-x86-64 ~]# service mysql status SUCCESS! MySQL running (11341) [root@centos70-x86-64 ~]# systemctl status mysql mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled) Drop-In: /etc/systemd/system/mariadb.service.d └─migrated-from-my.cnf-settings.conf Active: active (running) since Tue 2015-12-29 09:28:34 EST; 28s ago Main PID: 11341 (mysqld) Status: "Taking your SQL requests now..." CGroup: /system.slice/mariadb.service └─11341 /usr/sbin/mysqld   Dec 29 09:28:34 centos70-x86-64 mysqld[11341]: 2015-12-29 9:28:34 140552782350080 [Note] WSREP: wsrep_notify_c...tion. Dec 29 09:28:34 centos70-x86-64 mysqld[11341]: 2015-12-29 9:28:34 140552782350080 [Note] WSREP: REPL Protocols...3, 2) Dec 29 09:28:34 centos70-x86-64 mysqld[11341]: 2015-12-29 9:28:34 140552544712448 [Note] WSREP: Service thread...shed. Dec 29 09:28:34 centos70-x86-64 mysqld[11341]: 2015-12-29 9:28:34 140552782350080 [Note] WSREP: Assign initial...on: 3 Dec 29 09:28:34 centos70-x86-64 mysqld[11341]: 2015-12-29 9:28:34 140552544712448 [Note] WSREP: Service thread...shed. Dec 29 09:28:34 centos70-x86-64 mysqld[11341]: 2015-12-29 9:28:34 140552782350080 [Note] WSREP: Synchronized w...tions Dec 29 09:28:34 centos70-x86-64 mysqld[11341]: 2015-12-29 9:28:34 140552782350080 [Note] WSREP: wsrep_notify_c...tion. Dec 29 09:28:34 centos70-x86-64 mysqld[11341]: 2015-12-29 9:28:34 140552782657664 [Note] /usr/sbin/mysqld: rea...ions. Dec 29 09:28:34 centos70-x86-64 mysqld[11341]: Version: '10.1.10-MariaDB' socket: '/var/lib/mysql/mysql.sock' ...erver Dec 29 09:28:34 centos70-x86-64 systemd[1]: Started MariaDB database server. Hint: Some lines were ellipsized, use -l to show in full.     [root@centos70-x86-64 ~]# systemctl stop mysql [root@centos70-x86-64 ~]# service mysql status ERROR! MySQL is not running [root@centos70-x86-64 ~]# systemctl status mysql mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled) Drop-In: /etc/systemd/system/mariadb.service.d └─migrated-from-my.cnf-settings.conf Active: inactive (dead) since Tue 2015-12-29 09:29:54 EST; 12s ago Process: 11341 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=exited, status=0/SUCCESS) Main PID: 11341 (code=exited, status=0/SUCCESS) Status: "MariaDB server is down"   Dec 29 09:29:52 centos70-x86-64 mysqld[11341]: 2015-12-29 9:29:52 140552781612800 [Note] WSREP: wsdb trx map u...age 0 Dec 29 09:29:52 centos70-x86-64 mysqld[11341]: 2015-12-29 9:29:52 140552781612800 [Note] WSREP: MemPool(LocalT...ol: 0 Dec 29 09:29:52 centos70-x86-64 mysqld[11341]: 2015-12-29 9:29:52 140552781612800 [Note] WSREP: MemPool(SlaveT...ol: 0 Dec 29 09:29:52 centos70-x86-64 mysqld[11341]: 2015-12-29 9:29:52 140552781612800 [Note] WSREP: Shifting CLOSE...O: 0) Dec 29 09:29:52 centos70-x86-64 mysqld[11341]: 2015-12-29 9:29:52 140552781612800 [Note] WSREP: Flushing memor...sk... Dec 29 09:29:52 centos70-x86-64 mysqld[11341]: 2015-12-29 9:29:52 140551793899264 [Note] InnoDB: FTS optimize ...ting. Dec 29 09:29:52 centos70-x86-64 mysqld[11341]: 2015-12-29 9:29:52 140552781612800 [Note] InnoDB: Starting shutdown... Dec 29 09:29:54 centos70-x86-64 mysqld[11341]: 2015-12-29 9:29:54 140552781612800 [Note] InnoDB: Shutdown comp...17009 Dec 29 09:29:54 centos70-x86-64 mysqld[11341]: 2015-12-29 9:29:54 140552781612800 [Note] /usr/sbin/mysqld: Shu...plete Dec 29 09:29:54 centos70-x86-64 systemd[1]: Stopped MariaDB database server. Hint: Some lines were ellipsized, use -l to show in full.     [root@centos70-x86-64 ~]# systemctl status mysql mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled) Drop-In: /etc/systemd/system/mariadb.service.d └─migrated-from-my.cnf-settings.conf Active: active (running) since Tue 2015-12-29 09:30:38 EST; 5s ago Main PID: 11424 (mysqld) Status: "Taking your SQL requests now..." CGroup: /system.slice/mariadb.service └─11424 /usr/sbin/mysqld   Dec 29 09:30:38 centos70-x86-64 mysqld[11424]: 2015-12-29 9:30:38 140254956128000 [Note] WSREP: wsrep_notify_c...tion. Dec 29 09:30:38 centos70-x86-64 mysqld[11424]: 2015-12-29 9:30:38 140254956435584 [Note] /usr/sbin/mysqld: rea...ions. Dec 29 09:30:38 centos70-x86-64 mysqld[11424]: Version: '10.1.10-MariaDB' socket: '/var/lib/mysql/mysql.sock' ...erver Dec 29 09:30:38 centos70-x86-64 mysqld[11424]: 2015-12-29 9:30:38 140254956128000 [Note] WSREP: REPL Protocols...3, 2) Dec 29 09:30:38 centos70-x86-64 mysqld[11424]: 2015-12-29 9:30:38 140254719768320 [Note] WSREP: Service thread...shed. Dec 29 09:30:38 centos70-x86-64 mysqld[11424]: 2015-12-29 9:30:38 140254956128000 [Note] WSREP: Assign initial...on: 3 Dec 29 09:30:38 centos70-x86-64 mysqld[11424]: 2015-12-29 9:30:38 140254719768320 [Note] WSREP: Service thread...shed. Dec 29 09:30:38 centos70-x86-64 mysqld[11424]: 2015-12-29 9:30:38 140254956128000 [Note] WSREP: Synchronized w...tions Dec 29 09:30:38 centos70-x86-64 mysqld[11424]: 2015-12-29 9:30:38 140254956128000 [Note] WSREP: wsrep_notify_c...tion. Dec 29 09:30:38 centos70-x86-64 systemd[1]: Started MariaDB database server. Hint: Some lines were ellipsized, use -l to show in full. [root@centos70-x86-64 ~]# service mysql status SUCCESS! MySQL running (11424)

Ahmad El-Bardan added a comment - 2015-12-29 16:47 [root@server ~]# sh -x service mysql status + . /etc/init.d/functions ++ TEXTDOMAIN=initscripts ++ umask 022 ++ PATH=/sbin:/usr/sbin:/bin:/usr/bin ++ export PATH ++ '[' 12595 -ne 1 -a -z '' ']' ++ /bin/mountpoint -q /cgroup/systemd ++ /bin/mountpoint -q /sys/fs/cgroup/systemd ++ case "$0" in ++ '[' -z '' ']' ++ COLUMNS=80 ++ '[' -z '' ']' ++ '[' -c /dev/stderr -a -r /dev/stderr ']' +++ /sbin/consoletype ++ CONSOLETYPE=pty ++ '[' -z '' ']' ++ '[' -z '' ']' ++ '[' -f /etc/sysconfig/i18n -o -f /etc/locale.conf ']' ++ . /etc/profile.d/lang.sh ++ unset LANGSH_SOURCED ++ '[' -z '' ']' ++ '[' -f /etc/sysconfig/init ']' ++ . /etc/sysconfig/init +++ BOOTUP=color +++ RES_COL=60 +++ MOVE_TO_COL='echo -en \033[60G' +++ SETCOLOR_SUCCESS='echo -en \033[0;32m' +++ SETCOLOR_FAILURE='echo -en \033[0;31m' +++ SETCOLOR_WARNING='echo -en \033[0;33m' +++ SETCOLOR_NORMAL='echo -en \033[0;39m' ++ '[' pty = serial ']' ++ __sed_discard_ignored_files='/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d' ++ '[' '' = 1 ']' ++ basename service + VERSION='service ver. 1.1' ++ basename service + USAGE='Usage: service < option > | --status-all | [ service_name [ command | --full-restart ] ]' + SERVICEDIR=/etc/init.d + ACTIONDIR=/usr/libexec/initscripts/legacy-actions + SERVICE= + ACTION= + OPTIONS= + '[' 2 -eq 0 ']' + cd / + '[' 2 -gt 0 ']' + case "${1}" in + '[' -z '' -a 2 -eq 1 -a mysql = --status-all ']' + '[' 2 -eq 2 -a status = --full-restart ']' + '[' -z '' ']' + SERVICE=mysql + shift + '[' 1 -gt 0 ']' + case "${1}" in + '[' -z mysql -a 1 -eq 1 -a status = --status-all ']' + '[' 1 -eq 2 -a '' = --full-restart ']' + '[' -z mysql ']' + '[' -z '' ']' + ACTION=status + shift + '[' 0 -gt 0 ']' + '[' -f /etc/init.d/mysql ']' + '[' status = start ']' + env -i PATH=/sbin:/usr/sbin:/bin:/usr/bin TERM=xterm SYSTEMCTL_IGNORE_DEPENDENCIES= SYSTEMCTL_SKIP_REDIRECT= /etc/init.d/mysql status ERROR! MySQL is running but PID file could not be found [root@server ~]#

Nirbhay Choubey (Inactive) added a comment - 2015-12-29 16:59 Thanks. Could you verify if pid file is really missing? Place "set -x" at the beginning of /etc/init.d/mysql and run service mysql status.

Ahmad El-Bardan added a comment - 2015-12-29 17:08 - edited There is /var/lib/mysql/server.pid so the name differs, it should be server.mydomain.com.pid [root@server init.d]# service mysql status + basedir= + datadir= + service_startup_timeout=900 + lockdir=/var/lock/subsys + lock_file_path=/var/lock/subsys/mysql + mysqld_pid_file_path= + test -z '' + basedir=/usr + bindir=/usr/bin + test -z '' + datadir=/var/lib/mysql + sbindir=/usr/sbin + libexecdir=/usr/sbin + datadir_set= + lsb_functions=/lib/lsb/init-functions + test -f /lib/lsb/init-functions + init_functions=/etc/init.d/functions + test -f /etc/init.d/functions + . /etc/init.d/functions ++ TEXTDOMAIN=initscripts ++ umask 022 ++ PATH=/sbin:/usr/sbin:/bin:/usr/bin ++ export PATH ++ '[' 17709 -ne 1 -a -z '' ']' ++ /bin/mountpoint -q /cgroup/systemd ++ /bin/mountpoint -q /sys/fs/cgroup/systemd ++ case "$0" in ++ _use_systemctl=1 ++ '[' -z '' ']' ++ COLUMNS=80 ++ '[' -z '' ']' ++ '[' -c /dev/stderr -a -r /dev/stderr ']' +++ /sbin/consoletype ++ CONSOLETYPE=pty ++ '[' -z '' ']' ++ '[' -z '' ']' ++ '[' -f /etc/sysconfig/i18n -o -f /etc/locale.conf ']' ++ . /etc/profile.d/lang.sh ++ unset LANGSH_SOURCED ++ '[' -z '' ']' ++ '[' -f /etc/sysconfig/init ']' ++ . /etc/sysconfig/init +++ BOOTUP=color +++ RES_COL=60 +++ MOVE_TO_COL='echo -en \033[60G' +++ SETCOLOR_SUCCESS='echo -en \033[0;32m' +++ SETCOLOR_FAILURE='echo -en \033[0;31m' +++ SETCOLOR_WARNING='echo -en \033[0;33m' +++ SETCOLOR_NORMAL='echo -en \033[0;39m' ++ '[' pty = serial ']' ++ __sed_discard_ignored_files='/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d' ++ '[' 1 = 1 ']' ++ '[' xstatus = xstart -o xstatus = xstop -o xstatus = xrestart -o xstatus = xreload -o xstatus = xtry-restart -o xstatus = xforce-reload -o xstatus = xcondrestart ']' + PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/bin + export PATH + mode=status + '[' 1 -ge 1 ']' + shift + case `echo "testing\c"`,`echo -n testing` in ++ echo 'testing\c' ++ echo -n testing + echo_n=-n + echo_c= + test -x ./bin/my_print_defaults + print_defaults=./bin/my_print_defaults + extra_args= + test -r /usr/my.cnf + test -r /var/lib/mysql/my.cnf ++ ./bin/my_print_defaults --mysqld mysql.server + parse_server_arguments --local-infile=0 --ignore_db_dirs=lost+found --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --tmpdir=/home/mysqltmp --innodb=ON --back_log=256 --max_connections=300 --key_buffer_size=32M --myisam_sort_buffer_size=64M --myisam_max_sort_file_size=2048M --join_buffer_size=128K --read_buffer_size=128K --sort_buffer_size=256K --table_definition_cache=8192 --table_open_cache=4096 --thread_cache_size=128 --wait_timeout=1800 --connect_timeout=10 --tmp_table_size=128M --max_heap_table_size=128M --max_allowed_packet=64M --max_seeks_for_key=1000 --group_concat_max_len=1024 --max_length_for_sort_data=1024 --net_buffer_length=16384 --max_connect_errors=100000 --concurrent_insert=2 --read_rnd_buffer_size=256K --bulk_insert_buffer_size=8M --query_cache_limit=512K --query_cache_size=16M --query_cache_type=1 --query_cache_min_res_unit=2K --query_prealloc_size=262144 --query_alloc_block_size=65536 --transaction_alloc_block_size=8192 --transaction_prealloc_size=4096 --default-storage-engine=InnoDB --log_warnings=1 --slow_query_log=0 --long_query_time=1 --slow_query_log_file=/var/lib/mysql/slowq.log --log-error=/var/log/mysqld.log --innodb_large_prefix=1 --innodb_purge_threads=1 --innodb_file_format=Barracuda --innodb_file_per_table=1 --innodb_open_files=1000 --innodb_data_file_path=ibdata1:10M:autoextend --innodb_buffer_pool_size=192M --innodb_log_files_in_group=2 --innodb_log_file_size=128M --innodb_log_buffer_size=8M --innodb_flush_log_at_trx_commit=2 --innodb_lock_wait_timeout=50 --innodb_flush_method=O_DIRECT --innodb_support_xa=1 --innodb_io_capacity=200 --innodb_read_io_threads=2 --innodb_write_io_threads=2 --userstat=0 --key_cache_segments=1 --aria_group_commit=none --aria_group_commit_interval=0 --aria_log_file_size=32M --aria_log_purge_type=immediate --aria_pagecache_buffer_size=32M --aria_sort_buffer_size=32M --innodb_buffer_pool_dump_at_shutdown=1 --innodb_buffer_pool_load_at_startup=1 --innodb_buffer_pool_populate=0 --performance_schema=OFF --innodb_stats_on_metadata=OFF --innodb_sort_buffer_size=2M --innodb_online_alter_log_max_size=128M --query_cache_strip_comments=0 --log_slow_filter=admin,filesort,filesort_on_disk,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk --innodb_defragment=1 --innodb_defragment_n_pages=7 --innodb_defragment_stats_accuracy=0 --innodb_defragment_fill_factor_n_recs=20 --innodb_defragment_fill_factor=0.9 --innodb_defragment_frequency=40 --wsrep_on=ON --binlog_format=ROW --default-storage-engine=innodb --innodb_autoinc_lock_mode=2 --innodb_locks_unsafe_for_binlog=1 --query_cache_size=0 --query_cache_type=0 --bind-address=0.0.0.0 --datadir=/var/lib/mysql --innodb_log_file_size=100M --innodb_file_per_table --innodb_flush_log_at_trx_commit=2 --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --wsrep_cluster_address=gcomm:// --wsrep_cluster_name=galera_cluster --wsrep_node_address=*************** --wsrep_node_name=db1 --wsrep_sst_method=rsync --wsrep_sst_auth=user:pass + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in ++ sed -e 's/^[^=]*=//' ++ echo --datadir=/var/lib/mysql + datadir=/var/lib/mysql + datadir_set=1 + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in ++ echo --datadir=/var/lib/mysql ++ sed -e 's/^[^=]*=//' + datadir=/var/lib/mysql + datadir_set=1 + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + for arg in '"$@"' + case "$arg" in + parse_server_arguments + test -z '' ++ hostname + mysqld_pid_file_path=/var/lib/mysql/server.mydomain.com.pid + '[' -f /etc/default/mysql ']' + '[' -f /etc/sysconfig/mysql ']' + '[' -f /etc/conf.d/mysql ']' + case "$mode" in + test -s /var/lib/mysql/server.mydomain.com.pid ++ pidof /usr/sbin/mysqld + mysqld_pid=19324 ++ echo 19324 ++ wc -w + pid_count=1 + test 1 -gt 1 + test -z 19324 + log_failure_msg 'MySQL is running but PID file could not be found' + echo ' ERROR! MySQL is running but PID file could not be found' ERROR! MySQL is running but PID file could not be found + exit 4

Nirbhay Choubey (Inactive) added a comment - 2016-02-19 00:20 - edited Ahmad It appears that the hostname on your system returns FQDN (server.mydomain.com) by default, while gethostname(), used by mysqld, returns the short form (server). I would try to find why hostname utility returns a name different from one returned by gethostname()? What output do you get for the following commands : hostname hostname --fqdn hostname --short uname -n cat /etc/hostname cat /etc/sysconfig/network cat /proc/sys/kernel/hostname strace hostname A potential workaround would be to restart server with --pid-file set to what init script expects (i.e. `hostname`.pid).