[MCOL-1229] IS.columnstore_columns crashes when DDL is simultaneously executing Created: 2018-02-22  Updated: 2018-05-22  Resolved: 2018-05-22

Status: Closed
Project: MariaDB ColumnStore
Component/s: None
Affects Version/s: 1.1.1
Fix Version/s: 1.1.5

Type: Bug Priority: Major
Reporter: VAROQUI Stephane Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Attachments: File loop.sh     File q1.sql     File q2.sql    
Sprint: 2018-08, 2018-09, 2018-10, 2018-11

 Description   

8275 | root | localhost | vo | Query | 15 | Unlocking tables | DROP TABLE starpdv_vo_2013_05_08 | 0.000 |

8414 root localhost columnstore_info Query 126 Filling schema table CREATE TABLE columnstore_info.columnstore_columns engine=myisam as (select * from information_schema 0.000
8433 root localhost NULL Query 0 init show processlist 0.000
8437 root localhost voit Query 3 creating table CREATE TABLE STARVO_IT_2016_11_08(

I don't unfortunatly done show processlit but the crash just happen on that
MariaDB [(none)]> show full processlist;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...

Note that the schema table size requested by UDF was not the one beeing created or dropped .

180222 15:49:05 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.2.10-MariaDB-log
key_buffer_size=536870912
read_buffer_size=4194304
max_used_connections=5
max_threads=153
thread_count=12
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1780876 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fbc800009a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fbcf15aed18 thread_stack 0x80000
/usr/local/mariadb/columnstore/mysql//bin/mysqld(my_print_stacktrace+0x29)[0x559e0afdeae9]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(handle_fatal_signal+0x3bd)[0x559e0ab23fad]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7fbd2c3c50c0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7fbd2a472fcf]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fbd2a4743fa]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(ZN9gnu_cxx27_verbose_terminate_handlerEv+0x15d)[0x7fbd2ad8b0ad]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8f066)[0x7fbd2ad89066]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8f0b1)[0x7fbd2ad890b1]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8f2c9)[0x7fbd2ad892c9]
/usr/local/mariadb/columnstore/lib/libexecplan.so.1(_ZN8execplan20CalpontSystemCatalog10columnRIDsERKNS0_9TableNameEb+0x439d)[0x7fbd030c5b3d]
/usr/local/mariadb/columnstore/mysql/lib/plugin/is_columnstore_columns.so(+0xa09a)[0x7fbcfc5aa09a]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x26f)[0x559e0aa23abf]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_ZN4JOIN10exec_innerEv+0x805)[0x559e0aa0bf75]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_ZN4JOIN4execEv+0x29)[0x559e0aa0c3c9]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x85b)[0x559e0aa0cc8b]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x11c)[0x559e0aa0d00c]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_Z21mysql_execute_commandP3THD+0x714f)[0x559e0a9c19cf]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x14)[0x559e0ac23b64]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x93)[0x559e0ac29903]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x1fc)[0x559e0ac29f0c]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_ZN7sp_head7executeEP3THDb+0x767)[0x559e0ac266f7]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x37b)[0x559e0ac2793b]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(+0x49aa6e)[0x559e0a9b3a6e]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_Z21mysql_execute_commandP3THD+0x2cca)[0x559e0a9bd54a]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x23d)[0x559e0a9c1c5d]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_Z18idb_vtable_processP3THDyP9Statement+0x952)[0x559e0a9c2f62]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0xcb4)[0x559e0a9c6174]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_Z10do_commandP3THD+0xc8)[0x559e0a9c7b28]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x224)[0x559e0aa81984]
/usr/local/mariadb/columnstore/mysql//bin/mysqld(handle_one_connection+0x34)[0x559e0aa81a14]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7494)[0x7fbd2c3bb494]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fbd2a528aff]



 Comments   
Comment by Andrew Hutchings (Inactive) [ 2018-02-22 ]

Interesting. I think it will be the DROP TABLE whilst i_s.columnstore_columns is getting the table name for the table being dropped that is the trigger.

Comment by VAROQUI Stephane [ 2018-02-22 ]

call columnstore_info.table_usage('voat', 'starvo_at_2017_11'); was the command not touching the drop at all

Comment by VAROQUI Stephane [ 2018-02-22 ]

Some cpimport was ingesting at the same time but not on such db...

Comment by Andrew Hutchings (Inactive) [ 2018-02-22 ]

I think you misunderstand. table_usage() was being executed the same time as the DROP and was trying to get info on the table when it was no longer there, causing the crash.

Comment by VAROQUI Stephane [ 2018-02-22 ]

Ok thanks , table_usage was a request on a specific table not overlapping other tasks can't get why the dropped table was read in first time , but the internals may work on it's own way that exceed my understanding !

Comment by Andrew Hutchings (Inactive) [ 2018-02-22 ]

Yes, sorry. The I_S tables don't (yet) support pushdown so get info on all tables and then filter.

Comment by Ravi Prakash (Inactive) [ 2018-04-30 ]

Replaced the re-throw of exception by a return of error status (value 1).

Comment by Ravi Prakash (Inactive) [ 2018-04-30 ]

This issue is a concurrency issue involving two clients where one client is fetching from fetching from information_schema.columnstore_columns as in "select * from information_schema.columnstore_columns" and other client is dropping an existing table at the same time. The select query crashes the server when it trips over a non-existing table that it thinks, exists since it started executing the query. I reproduced the problem manually by running two queries as follows:
Client 1:
mcsmysql test <q1.sql
sh loop.sh q1.sql 100
Client 2:
sh loop.sh q2.sql 10

The files are attached.
loop.sh q1.sql q2.sql

Comment by Daniel Lee (Inactive) [ 2018-05-22 ]

Build verified: 1.1.5-1 source

[root@localhost ~]# cat mariadb-columnstore-1.1.5-1-centos7.x86_64.bin.tar.gz.txt
/root/columnstore/mariadb-columnstore-server
commit 0c983bff02172849a174dde46b62d76aa66485f8
Merge: 6b8a674 d5e6d89
Author: benthompson15 <ben.thompson@mariadb.com>
Date: Thu Apr 26 16:16:51 2018 -0500

Merge pull request #112 from mariadb-corporation/davidhilldallas-patch-3

update to 1.1.5

/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit 1ea5198e0e9ecc2a8d13e6b44bf6c632f8561199
Merge: 4533116 59858aa
Author: Andrew Hutchings <andrew@linuxjedi.co.uk>
Date: Fri May 18 12:37:47 2018 +0100

Merge pull request #475 from drrtuy/MCOL-1415

MCOL-1415

Added -f switch for mcsmysql in the loop.sh.

Reproduced the "MySQL server has gone away" issue in 1.1.4-1 and verified the fix.

Generated at Thu Feb 08 02:27:11 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.