[MCOL-819] mysqld not shutdown by shutdownsystem commands, sometimes Created: 2017-07-19  Updated: 2023-10-26  Resolved: 2017-09-05

Status: Closed
Project: MariaDB ColumnStore
Component/s: ?
Affects Version/s: 1.0.6, 1.0.7
Fix Version/s: 1.1.0

Type: Bug Priority: Minor
Reporter: David Hill (Inactive) Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Environment:

nightly regression test system

Sprint: 2017-14, 2017-15, 2017-16, 2017-17, 2017-18

 Description   

This has happened on my nightly regression test system, I haven't seen it happen on a regular install mysql.

As shown below, there is a script that gets run to shutdown and run postConfigure on the nightly build. Shutdown reports suggess, but postConfigure reports error that port 3306 is in use and that is because mysqld is still running..

Because its the build machine, its probably happening because:

1. it didnt get shutdown cleanly at the start of the build
2. since the install directories are deleted, the pdi file is deleted
3. and when the shutdown is done, its not shutting down because the PDI file isn't there, but there should be code in mysql-Columnstore to do a force shutdown when the PDI file isn't there. So need to check that logic

[root@ip-172-30-0-119 nightly]# ./localSystemRestart.sh
shutdownsystem Mon Dec 19 14:51:41 2016

This command stops the processing of applications on all Modules within the MariaDB Columnstore System

Checking for active transactions

Stopping System...
Successful stop of System

Shutting Down System...
Successful shutdown of System

Mariab Columnstore uninstall completed
The next step is:

/usr/local/mariadb/columnstore/bin/postConfigure

This is the MariaDB Columnstore System Configuration and Installation tool.
It will Configure the MariaDB Columnstore System and will perform a Package
Installation of all of the Servers within the System that is being configured.

IMPORTANT: This tool should only be run on the Parent OAM Module
which is a Performance Module, preferred Module #1

With the no-Prompting Option being specified, you will be required to have the following:

1. Root user ssh keys setup between all nodes in the system or
use the password command line option.
2. A Configure File to use to retrieve configure data, default to Columnstore.xml.rpmsave
or use the '-c' option to point to a configuration file.

The Calpont Configuration Data is taken from /usr/local/mariadb/columnstore/etc/Columnstore.xml.rpmsave

Do you want to utilize the configuration data from the saved copy? [y,n] >
NOTE: my.cnf file was upgraded based on my.cnf.rpmsave

===== Setup System Server Type Configuration =====

There are 2 options when configuring the System Server Type: single and multi

'single' - Single-Server install is used when there will only be 1 server configured
on the system. It can also be used for production systems, if the plan is
to stay single-server.

'multi' - Multi-Server install is used when you want to configure multiple servers now or
in the future. With Multi-Server install, you can still configure just 1 server
now and add on addition servers/modules in the future.

Select the type of System Server install [1=single, 2=multi] (1) >

Performing the Single Server Install.
Enter System Name (columnstore-1) >

===== Storage Configuration = internal =====

Enter the list (Nx,Ny,Nz) or range (Nx-Nz) of DBRoot IDs assigned to module 'pm1' (1) >

The MariaDB Columnstore port of '3306' is already in-use
For No-prompt install, use the command line argument of 'port' to enter a different number

ps -ef | grep mysql
root 5215 1 0 Dec15 ? 00:00:00 /bin/sh /usr/local/mariadb/columnstore/mysql//bin/mysqld_safe --datadir=/usr/local/mariadb/columnstore/mysql/db --pid-file=/usr/local/mariadb/columnstore/mysql/db/ip-172-30-0-119.pid --ledir=/usr/local/mariadb/columnstore/mysql//bin
mysql 5398 5215 0 Dec15 ? 00:00:48 /usr/local/mariadb/columnstore/mysql//bin/mysqld --basedir=/usr/local/mariadb/columnstore/mysql/ --datadir=/usr/local/mariadb/columnstore/mysql/db --plugin-dir=/usr/local/mariadb/columnstore/mysql/lib/plugin --user=mysql --log-error=/usr/local/mariadb/columnstore/mysql/db/ip-172-30-0-119.err --pid-file=/usr/local/mariadb/columnstore/mysql/db/ip-172-30-0-119.pid --socket=/usr/local/mariadb/columnstore/mysql/lib/mysql/mysql.sock --port=3306
root 66142 62289 0 15:01 pts/0 00:00:00 grep --color=auto mysql

 Comments   
Comment by David Hill (Inactive) [ 2017-08-28 ]

failure from QA

Build tested: Github source 1.1.0
[root@localhost mariadb-columnstore-server]# git show
commit 8e07495da650d922c4d1f3f09d77382168132b11
Merge: 80e57a8 c27e1e5
Author: David.Hall <david.hall@mariadb.com>
Date: Wed Jul 12 13:07:42 2017 -0500
[root@localhost mariadb-columnstore-engine]# git show
commit 853eb1388ea5f880abc9b11e9498bd8a3407fa5a
Merge: d138692 9ad2b0c
Author: David.Hall <david.hall@mariadb.com>
Date: Tue Jul 18 09:21:39 2017 -0500
mysql processes remained after stopsystem command.
1. ColumnStore is up and running
2. Renamed mysql's pid file
3. shutdownsystem
root 11194 1 0 19:43 ? 00:00:00 /bin/sh /usr/local/mariadb/columnstore/mysql//bin/mysqld_safe --datadir=/usr/local/mariadb/columnstore/mysql/db --pid-file=/usr/local/mariadb/columnstore/mysql/db/localhost.localdomain.pid --ledir=/usr/local/mariadb/columnstore/mysql//bin
mysql 11382 11194 0 19:43 ? 00:00:05 /usr/local/mariadb/columnstore/mysql//bin/mysqld --basedir=/usr/local/mariadb/columnstore/mysql/ --datadir=/usr/local/mariadb/columnstore/mysql/db --plugin-dir=/usr/local/mariadb/columnstore/mysql/lib/plugin --user=mysql --log-error=/usr/local/mariadb/columnstore/mysql/db/localhost.localdomain.err --pid-file=/usr/local/mariadb/columnstore/mysql/db/localhost.localdomain.pid --socket=/usr/local/mariadb/columnstore/mysql/lib/mysql/mysql.sock --port=3306
root 14819 13361 0 19:56 pts/0 00:00:00 grep --color=auto mysql

Comment by David Hill (Inactive) [ 2017-08-31 ]

didnt reproduce when I tired it...

Enter 'mcsmysql' to access the MariaDB ColumnStore SQL console
Enter 'mcsadmin' to access the MariaDB ColumnStore Admin console

[root@ip-172-30-0-192 bin]# cd ../mysql/db/
[root@ip-172-30-0-192 db]# ll
total 122952
rw-rw--- 1 mysql mysql 16384 Aug 31 20:05 aria_log.00000001
rw-rw--- 1 mysql mysql 52 Aug 31 20:05 aria_log_control
drwx------ 2 mysql mysql 58 Aug 31 20:05 calpontsys
drwx------ 2 mysql mysql 19 Aug 31 20:05 columnstore_info
rw-rw--- 1 mysql mysql 2855 Aug 31 20:05 ib_buffer_pool
rw-rw--- 1 mysql mysql 12582912 Aug 31 20:05 ibdata1
rw-rw--- 1 mysql mysql 50331648 Aug 31 20:05 ib_logfile0
rw-rw--- 1 mysql mysql 50331648 Aug 31 20:05 ib_logfile1
rw-rw--- 1 mysql mysql 12582912 Aug 31 20:05 ibtmp1
drwx------ 2 mysql mysql 105 Aug 31 20:05 infinidb_querystats
drwx------ 2 mysql mysql 19 Aug 31 20:05 infinidb_vtable
rw-rw--- 1 mysql mysql 6640 Aug 31 20:05 ip-172-30-0-192.err
rw-rw--- 1 mysql mysql 5 Aug 31 20:05 ip-172-30-0-192.pid
rw-rw--- 1 mysql mysql 0 Aug 31 20:05 multi-master.info
drwx-x-x 2 mysql mysql 4096 Aug 31 20:05 mysql
rw-rw--- 1 mysql mysql 21381 Aug 31 20:05 mysql-bin.000001
rw-rw--- 1 mysql mysql 342 Aug 31 20:05 mysql-bin.000002
rw-rw--- 1 mysql mysql 114 Aug 31 20:05 mysql-bin.index
drwx------ 2 mysql mysql 19 Aug 31 20:05 performance_schema
drwxr-xr-x 2 mysql mysql 6 Aug 31 20:05 test
[root@ip-172-30-0-192 db]# mv ip-172-30-0-192.pid ip-172-30-0-192.pid.old
[root@ip-172-30-0-192 db]#
[root@ip-172-30-0-192 db]# ma shutd y
shutdownsystem Thu Aug 31 20:07:29 2017

This command stops the processing of applications on all Modules within the MariaDB ColumnStore System

Checking for active transactions

Stopping System...
Successful stop of System

Shutting Down System...
Successful shutdown of System

[root@ip-172-30-0-192 db]# ps -ef | grep mysql
root 5167 2899 0 20:09 pts/0 00:00:00 grep --color=auto mysql
[root@ip-172-30-0-192 db]#

Comment by David Hill (Inactive) [ 2017-08-31 ]

please retest on last 1.1.0 builds...

Comment by Daniel Lee (Inactive) [ 2017-09-05 ]

Build verified: 1.1.0 Github source

/root/columnstore/mariadb-columnstore-server
commit 9e855a6415e0edd6771c449a6591c21c3915bfec
Merge: 6ed33d1 c206e51
Author: David.Hall <david.hall@mariadb.com>
Date: Tue Sep 5 09:43:29 2017 -0500

/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit 90353b9b908e1c9ee241c4a156a2a377c53cc807
Author: david hill <david.hill@mariadb.com>
Date: Fri Sep 1 14:46:07 2017 -0500

Could no longer reproduce the issue, with shutdownsystem on both UM1 or PM1.

Generated at Thu Feb 08 02:24:04 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.