[MCOL-1106] multi-node install fails, mysqld didnt startup in time Created: 2017-12-12  Updated: 2018-02-07  Resolved: 2018-02-07

Status: Closed
Project: MariaDB ColumnStore
Component/s: None
Affects Version/s: None
Fix Version/s: 1.0.13, 1.1.3

Type: Bug Priority: Major
Reporter: David Hill (Inactive) Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Sprint: 2017-25, 2018-01, 2018-02, 2018-03

 Description   

multi-node install fails, mysqld didnt startup in time. Timeout is 30 seconds, needs to be increased to 60. Had same issue in 1.1.2..

also the binary installer script timed out after increase to 60 seconds, so that timeout will need to be changed...

pm1

debug1: Sending command: /home/mariadb-user/mariadb/columnstore/bin/module_installer.sh --module=um --port=3306 --installdir=/home/mariadb-user/mariadb/columnstore
Setup rc.local on Module
Run Upgrade on my.cnf on Module
Run Mysql Port update on my.cnf on Module
Run post-mysqld-install
Run post-mysql-install
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
ERROR: post-mysql-install failed: check /tmp/post-mysql-install.log
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 3016, received 3044 bytes, in 51.9 seconds
Bytes per second: sent 58.1, received 58.7
debug1: Exit status 1

um1

ll /tmp/post-mysql-install.log
rw-rw-r- 1 mariadb-user mariadb-user 285 Dec 12 16:16 /tmp/post-mysql-install.log

[mariadb-user@ip-172-30-0-111 ~]$ cat /tmp/post-mysql-install.log
Starting MySQL.............................. ERROR!
checking for engine columnstore...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock' (2 "No such file or directory")
columnstore doesn't exist

ll /home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock
srwxrwxrwx 1 mariadb-user mariadb-user 0 Dec 12 16:17 /home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock

mariadb+ 13119 1 0 16:15 ? 00:00:00 /bin/sh /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld_safe --datadir
mariadb+ 13425 13119 5 16:16 ? 00:00:07 /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld --basedir=/home/mariad
root 13454 1188 0 16:18 ? 00:00:00 sshd: mariadb-user [priv]
mariadb+ 13456 13454 0 16:18 ? 00:00:00 sshd: mariadb-user@pts/0
mariadb+ 13457 13456 0 16:18 pts/0 00:00:00 -bash
mariadb+ 13479 13457 0 16:19 pts/0 00:00:00 ps -ef
[mariadb-user@ip-172-30-0-111 ~]$ mcsmysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 10.1.29-MariaDB Columnstore 1.0.12-1

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

cat ip-172-30-0-111.us-west-2.compute.internal.err
171212 16:15:49 mysqld_safe Starting mysqld daemon with databases from /home/mariadb-user/mariadb/columnstore/mysql/db
2017-12-12 16:16:42 140486689269888 [Note] /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld (mysqld 10.1.29-MariaDB) starting as process 13425 ...
2017-12-12 16:16:48 140486689269888 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2017-12-12 16:16:48 140486689269888 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2017-12-12 16:16:48 140486689269888 [Note] InnoDB: The InnoDB memory heap is disabled
2017-12-12 16:16:48 140486689269888 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-12-12 16:16:48 140486689269888 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-12-12 16:16:48 140486689269888 [Note] InnoDB: Compressed tables use zlib 1.2.7
2017-12-12 16:16:48 140486689269888 [Note] InnoDB: Using Linux native AIO
2017-12-12 16:16:48 140486689269888 [Note] InnoDB: Using SSE crc32 instructions
2017-12-12 16:16:48 140486689269888 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2017-12-12 16:16:48 140486689269888 [Note] InnoDB: Completed initialization of buffer pool
2017-12-12 16:16:48 140486689269888 [Note] InnoDB: Highest supported file format is Barracuda.
2017-12-12 16:16:51 140486689269888 [Note] InnoDB: 128 rollback segment(s) are active.
2017-12-12 16:16:51 140486689269888 [Note] InnoDB: Waiting for purge to start
2017-12-12 16:16:51 140486689269888 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.2 started; log sequence number 1616717
2017-12-12 16:17:04 140485477783296 [Note] InnoDB: Dumping buffer pool(s) not yet started
2017-12-12 16:17:04 140486689269888 [Note] Plugin 'FEEDBACK' is disabled.
2017-12-12 16:17:04 140486689269888 [Note] Server socket created on IP: '::'.
2017-12-12 16:17:04 140486689269888 [Note] /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld: ready for connections.
Version: '10.1.29-MariaDB' socket: '/home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock' port: 3306 Columnstore 1.0.12-1



 Comments   
Comment by David Hill (Inactive) [ 2017-12-12 ]

updated files:

mysql-ColumnStore - increased timeout from 30 to 90 seconds. Added code to handle a force shutdown when graceful shutdown doesnt work. This logic backported from 1.1

binary_installer.sh and user_installer.sh - increased timeout for module_installer

Comment by Ben Thompson (Inactive) [ 2017-12-12 ]

Reviewed / Merged

Comment by David Hill (Inactive) [ 2017-12-12 ]

To test, need to do a separate um and pm system like a 1um/2pm system. Verify that postConfigure completes successfully. Error was the it was failing on doing remote install of um.

Comment by Roman [ 2018-01-05 ]

JFYI In my one node docker environment I have to increase timeout up to 320 seconds to deal with installation problems for 1um/2pm setup.

Comment by Daniel Lee (Inactive) [ 2018-02-07 ]

Build verified: 1.1.3-1 Github source

[root@localhost ~]# cat mariadb-columnstore-1.1.3-1-centos7.x86_64.bin.tar.gz.txt
/root/columnstore/mariadb-columnstore-server
commit e5499e513d88a3dfefbe9a356e20a1bceb1bde38
Merge: 99cdb0a 4840a43
Author: david hill <david.hill@mariadb.com>
Date: Wed Jan 31 16:53:52 2018 -0600

Merge pull request #92 from mariadb-corporation/MCOL-1152

MCOL-1152: Change the debian package names.

/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit ffcc4e94ccd07b6bbdfc47531c4c2f065f3fab47
Merge: 05431bf 3fb5dde
Author: benthompson15 <ben.thompson@mariadb.com>
Date: Mon Feb 5 22:37:05 2018 -0600

Merge pull request #398 from mariadb-corporation/statuscontrol-fix

add ProcStatusControl to postConfigure setup

Comment by Daniel Lee (Inactive) [ 2018-02-07 ]

Build verified: 1.0.13-1 Github

/root/columnstore/mariadb-columnstore-server
commit da2e7b8cfaf852e08187d3ebcfddcc3482bded86
Merge: e5b122c adc20a8
Author: David.Hall <david.hall@mariadb.com>
Date: Tue Feb 6 17:17:29 2018 -0600

Merge pull request #93 from mariadb-corporation/MCOL-1206

MCOL-1206 Merge MariaDB 10.1.31

/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit 06979c8d1b06d1d8640dba1f53144f9f53de9a58
Merge: cc2bbe8 def46ca
Author: benthompson15 <ben.thompson@mariadb.com>
Date: Mon Jan 22 14:17:38 2018 -0600

Merge pull request #378 from mariadb-corporation/RELEASE

update version

Generated at Thu Feb 08 02:26:15 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.