Run MariaDB ColumnStore Replication Setup..
ERROR: Error return in running the MariaDB ColumnStore Master replication, check /tmp/master-rep*.logs on um1
MariaDB ColumnStore Install Failed
Checking UM module logs and /tmp script log files pointed to a problem where mysqld failed to restart this this process, which caused the Replication to fail.
[mariadb-user@ip-172-30-0-135 columnstore]$ mcsmysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.soc' (2) [mariadb-user@ip-172-30-0-135 columnstore]$ ps -ef | grep mysql
mariadb+ 4210 1 0 19:20 ? 00:00:00 /bin/sh /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld_safe --datadir=/home/mariadb-user/mariadb/columnstore/mysql/db --pid-file=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.pid --ledir=/home/mariadb-user/mariadb/columnstore/mysql//bin
mariadb+ 4395 4210 0 19:20 ? 00:00:00 /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld --basedir=/home/mariadb-user/mariadb/columnstore/mysql/ --datadir=/home/mariadb-user/mariadb/columnstore/mysql/db --plugin-dir=/home/mariadb-user/mariadb/columnstore/mysql/lib/plugin --log-error=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.err --pid-file=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.pid --socket=/home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock --port=3306
mariadb+ 7701 7618 0 19:29 pts/0 00:00:00 grep --color=auto mysql [mariadb-user@ip-172-30-0-135 columnstore]$ ll /home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock
ls: cannot access /home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock: No such file or directory
tmp]$ cat actionMysqlCalpont.log
Shutting down MySQL.............................. ERROR!
ERROR! Failed to stop running server, so refusing to try to start.
Issue is a known issue where mysqld isn't cleanly shutting down if you try after a startup to quickly. The restart is being done around the Replication setup where it updates the my.cnf file.
Change the code to where the changes to my.cnf are just made on the mysqld command line also and then DON'T do the mysqld restart.
Look at other times when mysqld restart is done and see if needed..
David Hill (Inactive)
added a comment - Issue is a known issue where mysqld isn't cleanly shutting down if you try after a startup to quickly. The restart is being done around the Replication setup where it updates the my.cnf file.
Change the code to where the changes to my.cnf are just made on the mysqld command line also and then DON'T do the mysqld restart.
Look at other times when mysqld restart is done and see if needed..
NOTE: When in this case, mysqld will continue to fail to stop. so stopsystem or tryint to stop it directory will fail. Work around will be the need to FORCE kill it
David Hill (Inactive)
added a comment - NOTE: When in this case, mysqld will continue to fail to stop. so stopsystem or tryint to stop it directory will fail. Work around will be the need to FORCE kill it
kill -9 4210 4395
ps -ef | grep mysql
mariadb+ 4210 1 0 19:20 ? 00:00:00 /bin/sh /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld_safe --datadir=/home/mariadb-user/mariadb/columnstore/mysql/db --pid-file=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.pid --ledir=/home/mariadb-user/mariadb/columnstore/mysql//bin
mariadb+ 4395 4210 0 19:20 ? 00:00:03 /home/mariadb-user/mariadb/columnstore/mysql//bin/mysqld --basedir=/home/mariadb-user/mariadb/columnstore/mysql/ --datadir=/home/mariadb-user/mariadb/columnstore/mysql/db --plugin-dir=/home/mariadb-user/mariadb/columnstore/mysql/lib/plugin --log-error=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.err --pid-file=/home/mariadb-user/mariadb/columnstore/mysql/db/ip-172-30-0-135.pid --socket=/home/mariadb-user/mariadb/columnstore/mysql/lib/mysql/mysql.sock --port=3306
mariadb+ 8190 7618 0 20:44 pts/0 00:00:00 grep --color=auto mysql
[mariadb-user@ip-172-30-0-135 mysql] $ ./mysql-Columnstore stop
Shutting down MySQL.............................. ERROR!
To test: perform a multi-node install when it setup replication on mult ums or combo pm2 or local local pms.
New code will set the server-id and local query setting via the mysql console line and update the my.cnf file.
And after the replication is setup, it WILL NOT restart mysqld now at the end like in 1.1.1 and prior builds.
David Hill (Inactive)
added a comment - To test: perform a multi-node install when it setup replication on mult ums or combo pm2 or local local pms.
New code will set the server-id and local query setting via the mysql console line and update the my.cnf file.
And after the replication is setup, it WILL NOT restart mysqld now at the end like in 1.1.1 and prior builds.
Build verified: 1.1.2 GitHub source
/root/columnstore/mariadb-columnstore-server
commit ed21e674cfd70db421957d0212ff7fc1835d06d5
Author: david hill <david.hill@mariadb.com>
Date: Mon Nov 6 08:41:53 2017 -0600
Update README.md
updated version
/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit e0febdcd1a88bc3fab55519bd679f93719687ccc
Merge: 1603ce9 f1dd92a
Author: david hill <david.hill@mariadb.com>
Date: Wed Nov 8 15:32:50 2017 -0600
Merge pull request #313 from mariadb-corporation/MCOL-989 MCOL-989
Daniel Lee (Inactive)
added a comment - Build verified: 1.1.2 GitHub source
/root/columnstore/mariadb-columnstore-server
commit ed21e674cfd70db421957d0212ff7fc1835d06d5
Author: david hill <david.hill@mariadb.com>
Date: Mon Nov 6 08:41:53 2017 -0600
Update README.md
updated version
/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit e0febdcd1a88bc3fab55519bd679f93719687ccc
Merge: 1603ce9 f1dd92a
Author: david hill <david.hill@mariadb.com>
Date: Wed Nov 8 15:32:50 2017 -0600
Merge pull request #313 from mariadb-corporation/ MCOL-989
MCOL-989
People
Daniel Lee (Inactive)
David Hill (Inactive)
Votes:
0Vote for this issue
Watchers:
3Start watching this issue
Dates
Created:
Updated:
Resolved:
Git Integration
Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.
{"report":{"fcp":1028.0999999046326,"ttfb":220.30000019073486,"pageVisibility":"visible","entityId":64119,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"3de32934-d57e-4784-b569-97cedfdd2765","navigationType":0,"readyForUser":1092.7000002861023,"redirectCount":0,"resourceLoadedEnd":1257.5999999046326,"resourceLoadedStart":225.59999990463257,"resourceTiming":[{"duration":290.30000019073486,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":225.59999990463257,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":225.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":515.9000000953674,"responseStart":0,"secureConnectionStart":0},{"duration":290.30000019073486,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":225.90000009536743,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":225.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":516.2000002861023,"responseStart":0,"secureConnectionStart":0},{"duration":299.30000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":226.09999990463257,"connectEnd":226.09999990463257,"connectStart":226.09999990463257,"domainLookupEnd":226.09999990463257,"domainLookupStart":226.09999990463257,"fetchStart":226.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":226.09999990463257,"responseEnd":525.4000000953674,"responseStart":525.4000000953674,"secureConnectionStart":226.09999990463257},{"duration":329.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":226.30000019073486,"connectEnd":226.30000019073486,"connectStart":226.30000019073486,"domainLookupEnd":226.30000019073486,"domainLookupStart":226.30000019073486,"fetchStart":226.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":226.30000019073486,"responseEnd":555.4000000953674,"responseStart":555.3000001907349,"secureConnectionStart":226.30000019073486},{"duration":333.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":226.5,"connectEnd":226.5,"connectStart":226.5,"domainLookupEnd":226.5,"domainLookupStart":226.5,"fetchStart":226.5,"redirectEnd":0,"redirectStart":0,"requestStart":226.5,"responseEnd":559.5999999046326,"responseStart":559.5999999046326,"secureConnectionStart":226.5},{"duration":333.40000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":226.59999990463257,"connectEnd":226.59999990463257,"connectStart":226.59999990463257,"domainLookupEnd":226.59999990463257,"domainLookupStart":226.59999990463257,"fetchStart":226.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":226.59999990463257,"responseEnd":560,"responseStart":560,"secureConnectionStart":226.59999990463257},{"duration":334.19999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":226.90000009536743,"connectEnd":226.90000009536743,"connectStart":226.90000009536743,"domainLookupEnd":226.90000009536743,"domainLookupStart":226.90000009536743,"fetchStart":226.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":226.90000009536743,"responseEnd":561.0999999046326,"responseStart":561.0999999046326,"secureConnectionStart":226.90000009536743},{"duration":443.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":227.09999990463257,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":227.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":670.5999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":334.59999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":227.2000002861023,"connectEnd":227.2000002861023,"connectStart":227.2000002861023,"domainLookupEnd":227.2000002861023,"domainLookupStart":227.2000002861023,"fetchStart":227.2000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":227.2000002861023,"responseEnd":561.8000001907349,"responseStart":561.8000001907349,"secureConnectionStart":227.2000002861023},{"duration":443.2000002861023,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":227.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":227.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":670.7000002861023,"responseStart":0,"secureConnectionStart":0},{"duration":334.80000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":227.59999990463257,"connectEnd":227.59999990463257,"connectStart":227.59999990463257,"domainLookupEnd":227.59999990463257,"domainLookupStart":227.59999990463257,"fetchStart":227.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":227.59999990463257,"responseEnd":562.4000000953674,"responseStart":562.4000000953674,"secureConnectionStart":227.59999990463257},{"duration":1027.4000000953674,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":228.40000009536743,"connectEnd":228.40000009536743,"connectStart":228.40000009536743,"domainLookupEnd":228.40000009536743,"domainLookupStart":228.40000009536743,"fetchStart":228.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":228.40000009536743,"responseEnd":1255.8000001907349,"responseStart":1255.8000001907349,"secureConnectionStart":228.40000009536743},{"duration":1024.5999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":233,"connectEnd":233,"connectStart":233,"domainLookupEnd":233,"domainLookupStart":233,"fetchStart":233,"redirectEnd":0,"redirectStart":0,"requestStart":233,"responseEnd":1257.5999999046326,"responseStart":1257.5999999046326,"secureConnectionStart":233},{"duration":112.59999990463257,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":681.8000001907349,"connectEnd":681.8000001907349,"connectStart":681.8000001907349,"domainLookupEnd":681.8000001907349,"domainLookupStart":681.8000001907349,"fetchStart":681.8000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":681.8000001907349,"responseEnd":794.4000000953674,"responseStart":794.4000000953674,"secureConnectionStart":681.8000001907349}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":57,"responseStart":220,"responseEnd":229,"domLoading":224,"domInteractive":1280,"domContentLoadedEventStart":1280,"domContentLoadedEventEnd":1321,"domComplete":1931,"loadEventStart":1931,"loadEventEnd":1932,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1262.8000001907349},{"name":"bigPipe.sidebar-id.end","time":1263.5999999046326},{"name":"bigPipe.activity-panel-pipe-id.start","time":1263.8000001907349},{"name":"bigPipe.activity-panel-pipe-id.end","time":1265.7000002861023},{"name":"activityTabFullyLoaded","time":1338.8000001907349}],"measures":[],"correlationId":"f3257fb806c150","effectiveType":"4g","downlink":9,"rtt":0,"serverDuration":105,"dbReadsTimeInMs":11,"dbConnsTimeInMs":19,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Bug
Major
(gdb) info threads
Id Target Id Frame
24 Thread 0x7f9236acd700 (LWP 4397) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
23 Thread 0x7f91ff5c2700 (LWP 4398) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
22 Thread 0x7f91fedc1700 (LWP 4399) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
21 Thread 0x7f91fe5c0700 (LWP 4400) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
20 Thread 0x7f91fddbf700 (LWP 4401) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
19 Thread 0x7f91fd5be700 (LWP 4402) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
18 Thread 0x7f91fcdbd700 (LWP 4403) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
17 Thread 0x7f91fc5bc700 (LWP 4404) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
16 Thread 0x7f91fbdbb700 (LWP 4405) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
15 Thread 0x7f91fb5ba700 (LWP 4406) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
14 Thread 0x7f91fadb9700 (LWP 4407) "mysqld" 0x00007f923aafb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
13 Thread 0x7f91fa5b8700 (LWP 4408) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
12 Thread 0x7f91f9390700 (LWP 4410) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
11 Thread 0x7f91f8b8f700 (LWP 4411) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
10 Thread 0x7f91f3fff700 (LWP 4412) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
9 Thread 0x7f91f9b91700 (LWP 4413) "mysqld" 0x00007f923b347bdd in nanosleep () from /lib64/libpthread.so.0
8 Thread 0x7f91f1ffb700 (LWP 4417) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
7 Thread 0x7f91d7fff700 (LWP 4420) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
6 Thread 0x7f91d77fe700 (LWP 4421) "mysqld" 0x00007f923b347bdd in nanosleep () from /lib64/libpthread.so.0
5 Thread 0x7f91d6ffd700 (LWP 4422) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
4 Thread 0x7f91d5ffb700 (LWP 4424) "mysqld" 0x00007f923b344a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
3 Thread 0x7f91ffa55700 (LWP 4426) "mysqld" 0x00007f923b348101 in sigwait () from /lib64/libpthread.so.0
2 Thread 0x7f91ff8d2700 (LWP 7329) "mysqld" 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007f923b3446d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f923bbbcacb in inline_mysql_cond_wait (src_file=0x7f923c2d6368 "/home/builder/mariadb-columnstore-server/sql/mysqld.cc", src_line=6089,
mutex=0x7f923cb37f40 <LOCK_thread_count>, that=0x7f923cb37840 <COND_thread_count>) at /home/builder/mariadb-columnstore-server/include/mysql/psi/mysql_thread.h:1149
#2 mysqld_main (argc=43, argv=0x7f923dbfa518) at /home/builder/mariadb-columnstore-server/sql/mysqld.cc:6089
#3 0x00007f923900ab35 in __libc_start_main () from /lib64/libc.so.6
#4 0x00007f923bbb13e9 in _start ()
(gdb)