mysqld not shutdown by shutdownsystem commands, sometimes

This has happened on my nightly regression test system, I haven't seen it happen on a regular install mysql.

As shown below, there is a script that gets run to shutdown and run postConfigure on the nightly build. Shutdown reports suggess, but postConfigure reports error that port 3306 is in use and that is because mysqld is still running..

Because its the build machine, its probably happening because:

1. it didnt get shutdown cleanly at the start of the build
2. since the install directories are deleted, the pdi file is deleted
3. and when the shutdown is done, its not shutting down because the PDI file isn't there, but there should be code in mysql-Columnstore to do a force shutdown when the PDI file isn't there. So need to check that logic

[root@ip-172-30-0-119 nightly]# ./localSystemRestart.sh
shutdownsystem Mon Dec 19 14:51:41 2016

This command stops the processing of applications on all Modules within the MariaDB Columnstore System

Checking for active transactions

Stopping System...
Successful stop of System

Shutting Down System...
Successful shutdown of System

Mariab Columnstore uninstall completed
The next step is:

/usr/local/mariadb/columnstore/bin/postConfigure

This is the MariaDB Columnstore System Configuration and Installation tool.
It will Configure the MariaDB Columnstore System and will perform a Package
Installation of all of the Servers within the System that is being configured.

IMPORTANT: This tool should only be run on the Parent OAM Module
which is a Performance Module, preferred Module #1

With the no-Prompting Option being specified, you will be required to have the following:

1. Root user ssh keys setup between all nodes in the system or
use the password command line option.
2. A Configure File to use to retrieve configure data, default to Columnstore.xml.rpmsave
or use the '-c' option to point to a configuration file.

The Calpont Configuration Data is taken from /usr/local/mariadb/columnstore/etc/Columnstore.xml.rpmsave

Do you want to utilize the configuration data from the saved copy? [y,n] >
NOTE: my.cnf file was upgraded based on my.cnf.rpmsave

===== Setup System Server Type Configuration =====

There are 2 options when configuring the System Server Type: single and multi

'single' - Single-Server install is used when there will only be 1 server configured
on the system. It can also be used for production systems, if the plan is
to stay single-server.

'multi' - Multi-Server install is used when you want to configure multiple servers now or
in the future. With Multi-Server install, you can still configure just 1 server
now and add on addition servers/modules in the future.

Select the type of System Server install [1=single, 2=multi] (1) >

Performing the Single Server Install.
Enter System Name (columnstore-1) >

===== Storage Configuration = internal =====

Enter the list (Nx,Ny,Nz) or range (Nx-Nz) of DBRoot IDs assigned to module 'pm1' (1) >

The MariaDB Columnstore port of '3306' is already in-use
For No-prompt install, use the command line argument of 'port' to enter a different number

ps -ef | grep mysql
root 5215 1 0 Dec15 ? 00:00:00 /bin/sh /usr/local/mariadb/columnstore/mysql//bin/mysqld_safe --datadir=/usr/local/mariadb/columnstore/mysql/db --pid-file=/usr/local/mariadb/columnstore/mysql/db/ip-172-30-0-119.pid --ledir=/usr/local/mariadb/columnstore/mysql//bin
mysql 5398 5215 0 Dec15 ? 00:00:48 /usr/local/mariadb/columnstore/mysql//bin/mysqld --basedir=/usr/local/mariadb/columnstore/mysql/ --datadir=/usr/local/mariadb/columnstore/mysql/db --plugin-dir=/usr/local/mariadb/columnstore/mysql/lib/plugin --user=mysql --log-error=/usr/local/mariadb/columnstore/mysql/db/ip-172-30-0-119.err --pid-file=/usr/local/mariadb/columnstore/mysql/db/ip-172-30-0-119.pid --socket=/usr/local/mariadb/columnstore/mysql/lib/mysql/mysql.sock --port=3306
root 66142 62289 0 15:01 pts/0 00:00:00 grep --color=auto mysql