[MCOL-50] mysql crash right after a startsystem completed - At least one DBRoot required for that query is offline Created: 2016-05-10  Updated: 2023-10-26  Resolved: 2016-05-31

Status: Closed
Project: MariaDB ColumnStore
Component/s: ?
Affects Version/s: None
Fix Version/s: 1.0.0

Type: Bug Priority: Major
Reporter: David Hill (Inactive) Assignee: David Hall (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: None
Environment:

centos 6.4 regression test vm

Sprint: MCOL Sprint 2 - Priority 2

 Description   

Nightly regression test log showed a lot of query failures.

Log file showed the reason as :

crit.log-20160510:May 9 21:10:51 srvregtest joblist[17423]: 51.234265 |2147483653|0|0| C 05 CAL0000: IDB-2034: At least one DBRoot required for that query is offline.

Logs files showed that a startsystem just completed and mysql crashed right after the system went active.

debug.log-20160510:May 9 21:10:50 srvregtest ProcessManager[14143]: 50.030447 |0|0|0| D 17 CAL0000: Set System State = ACTIVE
debug.log-20160510:May 9 21:10:50 srvregtest ProcessMonitor[14051]: 50.030996 |0|0|0| D 18 CAL0000: statusControl: REQUEST RECEIVED: Set System State = ACTIVE
debug.log-20160510:May 9 21:10:50 srvregtest ProcessManager[14143]: 50.060952 |0|0|0| D 17 CAL0000: startSystemThread Exit
debug.log-20160510:May 9 21:10:50 srvregtest ProcessMonitor[14051]: 50.348887 |0|0|0| D 18 CAL0000: statusControl: REQUEST RECEIVED: Set Process pm1/mysqld State = MAN_OFFLINE
debug.log-20160510:May 9 21:10:50 srvregtest ProcessMonitor[14051]: 50.349033 |0|0|0| D 18 CAL0000: statusControl: Set Process pm1/mysqld State = MAN_OFFLINE PID = 0
debug.log-20160510:May 9 21:10:50 srvregtest ProcessMonitor[14051]: 50.350594 |0|0|0| D 18 CAL0000: statusControl: REQUEST RECEIVED: Set Module pm1 State = DEGRADED
debug.log-20160510:May 9 21:10:51 srvregtest ExeMgr[17423]: 51.073112 |2147483653|0|0| D 16 CAL0041: Start SQL statement: select objectid,columnname from syscolumn where schema='tpch1c' and tablename='datatypetestm' --columnRIDs/FE; ||

crit.log-20160510:May 9 21:10:51 srvregtest joblist[17423]: 51.234265 |2147483653|0|0| C 05 CAL0000: IDB-2034: At least one DBRoot required for that query is offline.
crit.log-20160510:May 9 21:10:51 srvregtest joblist[17423]: 51.234535 |2147483653|0|0| C 05 CAL0000: IDB-2034: At least one DBRoot required for that query is offline.

Server version: 10.1.10-MariaDB
key_buffer_size=536870912
read_buffer_size=4194304
max_used_connections=2
max_threads=153
thread_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1780770 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x7f15baf95798
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f15796c6d98 thread_stack 0x80000
/usr/local/Calpont/mysql//sbin/mysqld(my_print_stacktrace+0x26)[0x7f15b7b4c9c6]
/usr/local/Calpont/mysql//sbin/mysqld(handle_fatal_signal+0x465)[0x7f15b772cce5]
/lib64/libpthread.so.0(+0x3c0dc0f500)[0x7f15b6d82500]
/lib64/libc.so.6(strnlen+0x21)[0x7f15b5209451]
/usr/local/Calpont/mysql//sbin/mysqld(+0x9ced77)[0x7f15b7b80d77]
/usr/local/Calpont/mysql//sbin/mysqld(my_vsnprintf_ex+0x182)[0x7f15b7b81822]
/usr/local/Calpont/mysql//sbin/mysqld(_ZN3THD18raise_error_printfEjz+0xd4)[0x7f15b75b1c54]
/usr/local/Calpont/mysql/lib64/mysql/plugin/libcalmysql.so(_ZN11cal_impl_if8setErrorEP3THDjSs+0x53)[0x7f15b2eea1f3]
/usr/local/Calpont/mysql/lib64/mysql/plugin/libcalmysql.so(_Z24ha_calpont_impl_rnd_initP5TABLE+0x5b53)[0x7f15b2ebdbe3]
/usr/local/Calpont/mysql//sbin/mysqld(_ZN7handler22ha_rnd_init_with_errorEb+0x15)[0x7f15b7732265]
/usr/local/Calpont/mysql//sbin/mysqld(_Z16init_read_recordP11READ_RECORDP3THDP5TABLEP10SQL_SELECTibb+0x3e5)[0x7f15b7817425]
/usr/local/Calpont/mysql//sbin/mysqld(Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_ordery15enum_duplicatesbPySB+0xc6f)[0x7f15b766b56f]
/usr/local/Calpont/mysql//sbin/mysqld(_Z21mysql_execute_commandP3THD+0x44c5)[0x7f15b75e03b5]
/usr/local/Calpont/mysql//sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x23e)[0x7f15b75e313e]
/usr/local/Calpont/mysql//sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1c7e)[0x7f15b75e831e]
/usr/local/Calpont/mysql//sbin/mysqld(_Z10do_commandP3THD+0xed)[0x7f15b75e85bd]
/usr/local/Calpont/mysql//sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7f15b769a9da]
/usr/local/Calpont/mysql//sbin/mysqld(handle_one_connection+0x46)[0x7f15b769aab6]
/lib64/libpthread.so.0(+0x3c0dc07851)[0x7f15b6d7a851]
/lib64/libc.so.6(clone+0x6d)[0x7f15b527094d]

 Comments   
Comment by David Hill (Inactive) [ 2016-05-10 ]

from mysql log before crash

2016-05-09 21:10:50 139731242887136 [Note] /usr/local/Calpont/mysql//sbin/mysqld: ready for connections.
Version: '10.1.10-MariaDB' socket: '/usr/local/Calpont/mysql/lib/mysql/mysql.sock' port: 3306 InfiniDB 5.0.0
DERIVED TABLE DEBUG
select count(tpch1c.region.r_regionkey) AS `col1` from `tpch1c`.`region`
DERIVED TABLE DEBUG END
DERIVED TABLE DEBUG
select tpch1c.nation.n_nationkey AS `n_nationkey` from `tpch1c`.`nation`
DERIVED TABLE DEBUG END
DERIVED TABLE DEBUG
select tpch1c.lineitem.l_orderkey AS `l_orderkey` from `tpch1c`.`lineitem`
DERIVED TABLE DEBUG END
DERIVED TABLE DEBUG
select tpch1c.part.p_partkey AS `p_partkey`,tpch1c.part.p_name AS `p_name`,tpch1c.part.p_mfgr AS `p_mfgr`,tpch1c.part.p_brand AS `p_brand`,tpch1c.part.p_type AS `p_type`,tpch1c.part.p_size AS `p_size`,tpch1c.part.p_container AS `p_container`,tpch1c.part.p_retailprice AS `p_retailprice`,tpch1c.part.p_comment AS `p_comment` from `tpch1c`.`part`
DERIVED TABLE DEBUG END
DERIVED TABLE DEBUG
select tpch1c.nation.n_name AS `n_name`,tpch1c.nation.n_regionkey AS `n_regionkey` from `tpch1c`.`nation`
DERIVED TABLE DEBUG END
160509 21:10:54 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Comment by Dipti Joshi (Inactive) [ 2016-05-31 ]

hill If this is not happening - please close this

Comment by David Hill (Inactive) [ 2016-05-31 ]

problem not happening anymore, closing

Generated at Thu Feb 08 02:18:08 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.