[MDEV-16148] Core dump in mysqld when executing SELECT statement Created: 2018-05-11  Updated: 2018-07-30  Resolved: 2018-07-30

Status: Closed
Project: MariaDB Server
Component/s: Optimizer
Affects Version/s: 10.2.13, 10.3.6, 10.2, 10.3
Fix Version/s: 10.2.16, 10.3.8

Type: Bug Priority: Major
Reporter: Ben Johnson Assignee: Varun Gupta (Inactive)
Resolution: Fixed Votes: 0
Labels: crash
Environment:

Ubuntu 16.04 LTS


Attachments: File 16148.sql     File mysql-test.php    

 Description   

I apologize in advance for my lack of experience in reporting MariaDB issues; I'll do my best.

I'm not even sure where to start troubleshooting this, but I have a PHP application that queries MariaDB, and for whatever reason, this particular query is causing a core-dump in mysqld.

Curiously, the very same query, when executed directly against mysqld (i.e., without PHP involved) doesn't seem to cause the crash. I am hoping that somebody here may be able to articulate the reason for this observation, or at least point me towards the actual culprit. (If providing all of my PHP information would be prudent or helpful, just let me know.)

Similarly, if table dumps would be helpful, I am willing to provide them, but they are sensitive in nature and contain confidential information.

I was using MariaDB 10.2.13 when I noticed the issue, but am able to reproduce it on 10.3.6.

I built MariaDB 10.3.6 with debugging enabled and captured a core-dump, which is massive for whatever reason (816MB), so I can't upload it here (the limit looks to be 10MB). I have instead uploaded the core dump and corresponding mysqld binary to ftp.askmonty.org/private, as "MDEV-16148-core" and "MDEV-16148-mysqld", respectively.

The entry in error.log is as follows:

180511 14:36:20 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.3.6-MariaDB-1:10.3.6+maria~xenial-log
key_buffer_size=402653184
read_buffer_size=2097152
max_used_connections=3
max_threads=102
thread_count=10
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 813259 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f8a6c0009a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f8ac440ccf8 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x555bbc26097e]
mysys/stacktrace.c:270(my_print_stacktrace)[0x555bbbd0fd67]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f8b088c7390]
/usr/sbin/mysqld(_Z30setup_sj_materialization_part2P13st_join_table+0x2ab)[0x555bbbc4769b]
sql/opt_subselect.cc:3955(setup_sj_materialization_part2(st_join_table*))[0x555bbba51f73]
sql/sql_select.cc:11916(make_join_readinfo(JOIN*, unsigned long long, unsigned int))[0x555bbbb8a050]
sql/sql_select.cc:2341(JOIN::optimize_stage2())[0x555bbbb8c01f]
sql/sql_select.cc:1511(JOIN::optimize_inner())[0x555bbbb8c207]
sql/sql_select.cc:1433(JOIN::optimize())[0x555bbbb8e0d0]
sql/sql_select.cc:4138(mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x555bbbb8e208]
sql/sql_select.cc:382(handle_select(THD*, LEX*, select_result*, unsigned long))[0x555bbba516eb]
sql/sql_parse.cc:6541(execute_sqlcom_select(THD*, TABLE_LIST*) [clone .constprop.235])[0x555bbbb3d2bd]
sql/sql_parse.cc:3768(mysql_execute_command(THD*))[0x555bbbb5387d]
sql/sql_prepare.cc:4740(Prepared_statement::execute(String*, bool))[0x555bbbb53f34]
sql/sql_prepare.cc:4170(Prepared_statement::execute_loop(String*, bool, unsigned char*, unsigned char*))[0x555bbbb54a08]
sql/sql_prepare.cc:3170(mysql_stmt_execute_common(THD*, unsigned long, unsigned char*, unsigned char*, unsigned long, bool, bool) [clone .constprop.202])[0x555bbbb54b45]
sql/sql_prepare.cc:3069(mysqld_stmt_execute(THD*, char*, unsigned int))[0x555bbbb416ad]
sql/sql_parse.cc:1788(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x555bbbb428e7]
sql/sql_parse.cc:1393(do_command(THD*))[0x555bbbc1083a]
sql/sql_connect.cc:1402(do_handle_one_connection(CONNECT*))[0x555bbbc109ad]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f8b088bd6ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f8b07f6841d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f8a6c013770): select count(*) as aggregate from `onsite_assessment`.`mmic_focuses` where `onsite_consult_id` = 58 and exists (select * from `onsite_assessment`.`mmic_sub_focuses` where `onsite_assessment`.`mmic_focuses`.`id` = `onsite_assessment`.`mmic_sub_focuses`.`focus_id` and `id` != '12' and `onsite_assessment`.`mmic_sub_focuses`.`deleted_at` is null) and exists (select * from `onsite_assessment`.`mmic_sub_focuses` where `onsite_assessment`.`mmic_focuses`.`id` = `onsite_assessment`.`mmic_sub_focuses`.`focus_id` and `id` != '7' and `onsite_assessment`.`mmic_sub_focuses`.`deleted_at` is null) and exists (select * from `onsite_assessment`.`mmic_sub_focuses` where `onsite_assessment`.`mmic_focuses`.`id` = `onsite_assessment`.`mmic_sub_focuses`.`focus_id` and `id` != '9' and `onsite_assessment`.`mmic_sub_focuses`.`deleted_at` is null) and exists (select * from `onsite_assessment`.`mmic_sub_focuses` where `onsite_assessment`.`mmic_focuses`.`id` = `onsite_assessment`.`mmic_sub_focuses`.`focus_id` and `id` != '11' and `onsite_assessment`.`mmic_sub_focuses`.`deleted_at` is null) and exists (select * from `onsite_assessment`.`mmic_sub_focuses` where `onsite_assessment`.`mmic_focuses`.`id` = `onsite_assessment`.`mmic_sub_focuses`.`focus_id` and `id` != '4' and `onsite_assessment`.`mmic_sub_focuses`.`deleted_at` is null) and exists (select * from `onsite_assessment`.`mmic_sub_focuses` where `onsite_assessment`.`mmic_focuses`.`id` = `onsite_assessment`.`mmic_sub_focuses`.`focus_id` and `id` != '10' and `onsite_assessment`.`mmic_sub_focuses`.`deleted_at` is null) and exists (select * from `onsite_assessment`.`mmic_sub_focuses` where `onsite_assessment`.`mmic_focuses`.`id` = `onsite_assessment`.`mmic_sub_focuses`.`focus_id` and `id` != '1' and `onsite_assessment`.`mmic_sub_focuses`.`deleted_at` is null) and exists (select * from `onsite_assessment`.`mmic_sub_focuses` where `onsite_assessment`.`mmic_focuses`.`id` = `onsite_assessment`.`mmic_sub_focuses`.`focus_id` and `id` != '2' and `onsite_assessment`.`mmic_sub_focuses`.`deleted_at` is null) and exists (select * from `onsite_assessment`.`mmic_sub_focuses` where `onsite_assessment`.`mmic_focuses`.`id` = `onsite_assessment`.`mmic_sub_focuses`.`focus_id` and `id` != '3' and `onsite_assessment`.`mmic_sub_focuses`.`deleted_at` is null) and `onsite_assessment`.`mmic_focuses`.`deleted_at` is null
Connection ID (thread ID): 10
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file at /var/lib/mysql/

And my configuration:

mysqld --print-defaults
mysqld would have been started with the following arguments:
--stack-trace --core-file --disable-gdb --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --basedir=/usr --datadir=/var/lib/mysql --tmpdir=/tmp --lc_messages_dir=/usr/share/mysql --lc_messages=en_US --skip-external-locking --bind-address=0.0.0.0 --max_connections=100 --connect_timeout=5 --wait_timeout=600 --max_allowed_packet=16M --thread_cache_size=128 --sort_buffer_size=4M --bulk_insert_buffer_size=16M --tmp_table_size=32M --max_heap_table_size=32M --myisam_recover_options=BACKUP --key_buffer_size=128M --table_open_cache=400 --myisam_sort_buffer_size=512M --concurrent_insert=2 --read_buffer_size=2M --read_rnd_buffer_size=1M --query_cache_limit=128K --query_cache_size=64M --log_warnings=2 --slow_query_log_file=/var/log/mysql/mariadb-slow.log --long_query_time=10 --log_slow_verbosity=query_plan --log_bin=/var/log/mysql/mariadb-bin --log_bin_index=/var/log/mysql/mariadb-bin.index --expire_logs_days=10 --max_binlog_size=100M --default_storage_engine=InnoDB --innodb_buffer_pool_size=256M --innodb_log_buffer_size=8M --innodb_file_per_table=1 --innodb_open_files=400 --innodb_io_capacity=400 --innodb_flush_method=O_DIRECT --default_storage_engine=InnoDB --skip-character-set-client-handshake --collation_server=utf8_unicode_ci --character_set_server=utf8 --long_query_time=10 --log-queries-not-using-indexes=OFF --skip-external-locking --key_buffer_size=384M --max_allowed_packet=128M --table_open_cache=512 --sort_buffer_size=2M --read_buffer_size=2M --read_rnd_buffer_size=8M --myisam_sort_buffer_size=64M --thread_cache_size=8 --query_cache_size=32M --innodb_data_home_dir=/var/lib/mysql --innodb_data_file_path=ibdata1:10M:autoextend --innodb_log_group_home_dir=/var/lib/mysql --innodb_buffer_pool_size=384M --innodb_log_file_size=100M --innodb_log_buffer_size=8M --innodb_flush_log_at_trx_commit=1 --innodb_lock_wait_timeout=50 --general-log --general-log-file=/var/log/mysql/queries.log --log-output=file --log-error=/var/log/mysql/error.log

I am of course happy to provide any other information that would be useful.



 Comments   
Comment by Daniel Black [ 2018-05-12 ]

Can you SHOW CREATE TABLE xxx and SHOW INDEX FROM xxx for the tables involved. EXPLAIN query may also be useful.

Comment by Ben Johnson [ 2018-05-14 ]

Thanks for taking a look, Daniel!

Each of the items you requested follows.

CREATE TABLE statements:

MariaDB [(none)]> show create table onsite_assessment.mmic_focuses;
+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table        | Create Table                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mmic_focuses | CREATE TABLE `mmic_focuses` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `onsite_consult_id` int(10) unsigned NOT NULL,
  `created_at` timestamp NULL DEFAULT NULL,
  `updated_at` timestamp NULL DEFAULT NULL,
  `deleted_at` timestamp NULL DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `fk_ltc_focuses_rt_onsite_consults_consult_id` (`onsite_consult_id`)
) ENGINE=InnoDB AUTO_INCREMENT=31 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
 
MariaDB [(none)]> show create table onsite_assessment.mmic_sub_focuses;
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table            | Create Table                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mmic_sub_focuses | CREATE TABLE `mmic_sub_focuses` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `focus_id` int(10) unsigned NOT NULL,
  `created_at` timestamp NULL DEFAULT NULL,
  `updated_at` timestamp NULL DEFAULT NULL,
  `deleted_at` timestamp NULL DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `fk_ltc_subfocuses_ltc_focus_ltc_focus_id` (`focus_id`),
  CONSTRAINT `fk_ltc_subfocuses_ltc_focus_ltc_focus_id` FOREIGN KEY (`focus_id`) REFERENCES `mmic_focuses` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=96 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Indexes:

MariaDB [(none)]> show index from onsite_assessment.mmic_focuses;
+--------------+------------+----------------------------------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table        | Non_unique | Key_name                                     | Seq_in_index | Column_name       | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------------+------------+----------------------------------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| mmic_focuses |          0 | PRIMARY                                      |            1 | id                | A         |          30 |     NULL | NULL   |      | BTREE      |         |               |
| mmic_focuses |          1 | fk_ltc_focuses_rt_onsite_consults_consult_id |            1 | onsite_consult_id | A         |          10 |     NULL | NULL   |      | BTREE      |         |               |
+--------------+------------+----------------------------------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
2 rows in set (0.00 sec)
 
MariaDB [(none)]> show index from onsite_assessment.mmic_sub_focuses;
+------------------+------------+------------------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table            | Non_unique | Key_name                                 | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+------------------+------------+------------------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| mmic_sub_focuses |          0 | PRIMARY                                  |            1 | id          | A         |          95 |     NULL | NULL   |      | BTREE      |         |               |
| mmic_sub_focuses |          1 | fk_ltc_subfocuses_ltc_focus_ltc_focus_id |            1 | focus_id    | A         |          47 |     NULL | NULL   |      | BTREE      |         |               |
+------------------+------------+------------------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
2 rows in set (0.01 sec)

EXPLAIN for query noted in segfault message:

+------+--------------+------------------+--------+------------------------------------------------------+--------------+---------+------+------+-------------+
| id   | select_type  | table            | type   | possible_keys                                        | key          | key_len | ref  | rows | Extra       |
+------+--------------+------------------+--------+------------------------------------------------------+--------------+---------+------+------+-------------+
|    1 | PRIMARY      | mmic_focuses     | ALL    | PRIMARY,fk_ltc_focuses_rt_onsite_consults_consult_id | NULL         | NULL    | NULL |   30 | Using where |
|    1 | PRIMARY      | <subquery10>     | eq_ref | distinct_key                                         | distinct_key | 4       | func |    1 |             |
|    1 | PRIMARY      | <subquery9>      | eq_ref | distinct_key                                         | distinct_key | 4       | func |    1 |             |
|    1 | PRIMARY      | <subquery8>      | eq_ref | distinct_key                                         | distinct_key | 4       | func |    1 |             |
|    1 | PRIMARY      | <subquery7>      | eq_ref | distinct_key                                         | distinct_key | 4       | func |    1 |             |
|    1 | PRIMARY      | <subquery6>      | eq_ref | distinct_key                                         | distinct_key | 4       | func |    1 |             |
|    1 | PRIMARY      | <subquery5>      | eq_ref | distinct_key                                         | distinct_key | 4       | func |    1 |             |
|    1 | PRIMARY      | <subquery4>      | eq_ref | distinct_key                                         | distinct_key | 4       | func |    1 |             |
|    1 | PRIMARY      | <subquery3>      | eq_ref | distinct_key                                         | distinct_key | 4       | func |    1 |             |
|    1 | PRIMARY      | <subquery2>      | eq_ref | distinct_key                                         | distinct_key | 4       | func |    1 |             |
|   10 | MATERIALIZED | mmic_sub_focuses | range  | PRIMARY,fk_ltc_subfocuses_ltc_focus_ltc_focus_id     | PRIMARY      | 4       | NULL |   94 | Using where |
|    9 | MATERIALIZED | mmic_sub_focuses | range  | PRIMARY,fk_ltc_subfocuses_ltc_focus_ltc_focus_id     | PRIMARY      | 4       | NULL |   94 | Using where |
|    8 | MATERIALIZED | mmic_sub_focuses | range  | PRIMARY,fk_ltc_subfocuses_ltc_focus_ltc_focus_id     | PRIMARY      | 4       | NULL |   95 | Using where |
|    7 | MATERIALIZED | mmic_sub_focuses | range  | PRIMARY,fk_ltc_subfocuses_ltc_focus_ltc_focus_id     | PRIMARY      | 4       | NULL |   94 | Using where |
|    6 | MATERIALIZED | mmic_sub_focuses | range  | PRIMARY,fk_ltc_subfocuses_ltc_focus_ltc_focus_id     | PRIMARY      | 4       | NULL |   94 | Using where |
|    5 | MATERIALIZED | mmic_sub_focuses | range  | PRIMARY,fk_ltc_subfocuses_ltc_focus_ltc_focus_id     | PRIMARY      | 4       | NULL |   94 | Using where |
|    4 | MATERIALIZED | mmic_sub_focuses | range  | PRIMARY,fk_ltc_subfocuses_ltc_focus_ltc_focus_id     | PRIMARY      | 4       | NULL |   94 | Using where |
|    3 | MATERIALIZED | mmic_sub_focuses | range  | PRIMARY,fk_ltc_subfocuses_ltc_focus_ltc_focus_id     | PRIMARY      | 4       | NULL |   94 | Using where |
|    2 | MATERIALIZED | mmic_sub_focuses | range  | PRIMARY,fk_ltc_subfocuses_ltc_focus_ltc_focus_id     | PRIMARY      | 4       | NULL |   94 | Using where |
+------+--------------+------------------+--------+------------------------------------------------------+--------------+---------+------+------+-------------+
19 rows in set (0.31 sec)

Please let me know if further info would be helpful.

Comment by Alice Sherepa [ 2018-05-16 ]

cbj4074, I can not reproduce it so far, had the same query plan, but no crash. Is it reproducible every time with php script for you? please add your script (here or ftp.askmonty.org/private)

Comment by Ben Johnson [ 2018-05-16 ]

Thanks for all your help so far, Alice. I really appreciate it.

Frustratingly, no, the segfault does not occur every time the PHP script is executed. It seems random-ish in that the query needs to be executed several times before mysqld segfaults, but after it has crashed once, it seems to crash nearly every time the query is executed. Not terribly helpful, I know.

In my PHP script, the query is being executed through a framework, but I'll work to reproduce the issue in a bare-bones PHP script that I can send to you. After all, the framework is simply using the "mysqli" driver via PDO, so I expect the failure to occur without the framework layer in the mix. I'll get that over to you shortly.

I'm also going to test whether the segfault occurs across PHP versions. I'm using 7.1 currently, but I'll try it with 7.2 as well.

Finally, I will test whether it happens in MySQL, too.

More soon...

Comment by Ben Johnson [ 2018-05-16 ]

A few interesting findings:

  1. The crash does not occur in MySQL 5.7.22.
  2. The crash occurs regardless of PHP version (I tested 7.1 and 7.2).
  3. I am able to reproduce the crash in a simple PHP script that executes the query. Most interesting is the fact that the crash does not happen if I forego parameter binding and hard-code the values in the query string. This seems to indicate rather strongly that the problem is related to PDO parameter binding.

I have attached the simple PHP script that allows me to reproduce the crash to this ticket (it doesn't contain anything sensitive).

I have also dumped the two tables involved and uploaded them to the private FTP drop as "MDEV-16148-sample-tables.sql". (These tables don't contain PII or similar, but they are proprietary, and should not be posted publicly.)

Please let me know if I can do anything further on this end.

Thanks again!

Comment by Ben Johnson [ 2018-05-16 ]

Yes, the bindings are causing the crash.

I noticed earlier that the "id" values in the WHERE clauses are supplied as quoted strings within the raw query, but didn't think it to be a "problem", no less capable of causing a segfault.

If I change the binding types for those values from string to integer, the crash does not occur, e.g.:

$stmt->bind_param("isssssssss", $one, $two, $three, $four, $five, $six, $seven, $eight, $nine, $ten);

to:

$stmt->bind_param("iiiiiiiiii", $one, $two, $three, $four, $five, $six, $seven, $eight, $nine, $ten);

Comment by Ben Johnson [ 2018-05-18 ]

Given the fact that any unprivileged user can crash the mysqld daemon using this technique, this seems like a fairly urgent issue.

Couldn't this bug be exploited in a denial-of-service capacity? If I'm in a shared hosting environment, it seems I could configure a cron script to crash the daemon repeatedly.

Comment by Alice Sherepa [ 2018-05-25 ]

I reproduced the case on MariaDB 10.2,10.3
it looks like the problem is reproducible, while general_log is ON.

--source include/have_innodb.inc
 
CREATE TABLE t1 (
  id int  NOT NULL ,
  a1 int  ,
  d1 timestamp NULL DEFAULT NULL,
  PRIMARY KEY (id),
  KEY f1 (a1)
) ENGINE=Innodb ;
INSERT INTO `t1` VALUES (1,58,NULL),(2,58,NULL),(3,56,NULL),(4,58,NULL),(5,56,NULL),(6,58,NULL),(7,57,NULL),(8,55,NULL),(9,58,NULL),(10,56,NULL);
 
CREATE TABLE t2 (
  id int  NOT NULL ,
  b1 int ,
  d1 timestamp NULL DEFAULT NULL,
  PRIMARY KEY (id),
  KEY f2 (b1)
 ) ENGINE=Innodb ;
INSERT INTO `t2` VALUES (1,1,NULL),(2,2,NULL),(3,3,NULL),(4,4,NULL),(5,5,NULL),(6,6,NULL),(7,7,NULL),(8,8,NULL),(9,9,NULL),(10,10,NULL);
 
prepare stmt1 from "
SELECT count(*) FROM t1
WHERE a1 = '58'
  AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '8' AND t2.d1 IS NULL)
  AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '7' AND t2.d1 IS NULL)
  AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '6' AND t2.d1 IS NULL)
  AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '2' AND t2.d1 IS NULL)
  AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '1' AND t2.d1 IS NULL)
  AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '4' AND t2.d1 IS NULL)
  AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '5' AND t2.d1 IS NULL)
  AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '9' AND t2.d1 IS NULL)
  AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '3' AND t2.d1 IS NULL)";
 
execute stmt1;
execute stmt1;

2018-05-25 17:09:51 140361822512896 [ERROR] mysqld: Table 'general_log' is marked as crashed and should be repaired
180525 17:09:52 [ERROR] mysqld got signal 11 ;
 
Server version: 10.2.14-MariaDB-log
 
Thread pointer: 0x7fa82c0009a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fa888921ec8 thread_stack 0x49000
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(my_print_stacktrace+0x2e)[0xddebae]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(handle_fatal_signal+0x471)[0x7df981]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fa89fcb6390]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld[0x638983]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld[0x64960c]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_ZN4JOIN14optimize_innerEv+0x1540)[0x64bc10]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_ZN4JOIN8optimizeEv+0x56)[0x64e2d6]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xa2)[0x64e502]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x254)[0x650274]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld[0x5e7cf4]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z21mysql_execute_commandP3THD+0x1327)[0x5eb287]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4dd)[0x604c5d]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x9e)[0x60a81e]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z22mysql_sql_stmt_executeP3THD+0x156)[0x60b0d6]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z21mysql_execute_commandP3THD+0x2df9)[0x5ecd59]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x3a2)[0x5f4ee2]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x26ef)[0x5f7d5f]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z10do_commandP3THD+0x1c2)[0x5f8702]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x235)[0x6ddc35]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(handle_one_connection+0x3f)[0x6dddff]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fa89fcac6ba]
x86_64/clone.S:111(clone)[0x7fa89eb1141d]

180525 17:02:22 [ERROR] mysqld got signal 6 ;
 
Server version: 10.2.14-MariaDB-log
Thread pointer: 0x7fc1ec0009a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fc2580c0ec8 thread_stack 0x49000
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(my_print_stacktrace+0x2e)[0xddebae]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(handle_fatal_signal+0x471)[0x7df981]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fc25cce9390]
linux/raise.c:54(__GI_raise)[0x7fc25ba72428]
stdlib/abort.c:91(__GI_abort)[0x7fc25ba7402a]
posix/libc_fatal.c:175(__fsetlocking)[0x7fc25bab47ea]
malloc/malloc.c:4105(_int_free)[0x7fc25babd37a]
malloc/malloc.c:2969(__GI___libc_free)[0x7fc25bac153c]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(delete_dynamic+0x1f)[0xdc1ccf]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_ZN4JOIN7destroyEv+0x1b3)[0x627233]
sql/gcalc_tools.cc:502(Gcalc_result_receiver::add_point(double, double))[0x694edc]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x151)[0x64e5b1]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x254)[0x650274]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld[0x5e7cf4]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z21mysql_execute_commandP3THD+0x1327)[0x5eb287]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4dd)[0x604c5d]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x9e)[0x60a81e]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z22mysql_sql_stmt_executeP3THD+0x156)[0x60b0d6]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z21mysql_execute_commandP3THD+0x2df9)[0x5ecd59]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x3a2)[0x5f4ee2]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x26ef)[0x5f7d5f]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z10do_commandP3THD+0x1c2)[0x5f8702]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x235)[0x6ddc35]
/alice/m4-10.2.14/../_depot/m-tar/10.2.14/bin/mysqld(handle_one_connection+0x3f)[0x6dddff]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fc25ccdf6ba]
x86_64/clone.S:111(clone)[0x7fc25bb4441d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7fc1ec00f0c8): SELECT 1 FROM t1 WHERE a1 = '58'   AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '8' AND t2.d1 IS NULL)   AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '7' AND t2.d1 IS NULL)   AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '6' AND t2.d1 IS NULL)   AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '2' AND t2.d1 IS NULL)   AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '1' AND t2.d1 IS NULL)   AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '4' AND t2.d1 IS NULL)   AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '5' AND t2.d1 IS NULL)   AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '9' AND t2.d1 IS NULL)   AND EXISTS (SELECT * FROM t2 WHERE t1.id = t2.b1 AND id != '3' AND t2.d1 IS NULL)
Connection ID (thread ID): 10
Status: NOT_KILLED

on 10.2

Thread 1 (Thread 0x7f7bf4576700 (LWP 6364)):
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
#1  0x00005647ce075faa in my_write_core (sig=sig@entry=6) at /home/alice/git/10.2/mysys/stacktrace.c:477
#2  0x00005647cdb8be14 in handle_fatal_signal (sig=6) at /home/alice/git/10.2/sql/signal_handler.cc:305
#3  <signal handler called>
#4  0x00007f7c01926428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#5  0x00007f7c0192802a in __GI_abort () at abort.c:89
#6  0x00007f7c019687ea in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f7c01a81ed8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#7  0x00007f7c0197137a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>, str=0x7f7c01a81fe8 "double free or corruption (out)", action=3) at malloc.c:5006
#8  _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3867
#9  0x00007f7c0197553c in __GI___libc_free (mem=<optimized out>) at malloc.c:2968
#10 0x00005647ce072910 in my_free (ptr=<optimized out>) at /home/alice/git/10.2/mysys/my_malloc.c:217
#11 0x00005647ce06b80d in free_root (root=0x7f7bb0005a90, MyFlags=MyFlags@entry=1) at /home/alice/git/10.2/mysys/my_alloc.c:392
#12 0x00005647cda0d7d0 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7f7bb00009a8, packet=<optimized out>, packet@entry=0x7f7bb0006a09 "", packet_length=<optimized out>, packet_length@entry=13, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /home/alice/git/10.2/sql/sql_parse.cc:2414
#13 0x00005647cda0f8bf in do_command (thd=0x7f7bb00009a8) at /home/alice/git/10.2/sql/sql_parse.cc:1375
#14 0x00005647cdacd484 in do_handle_one_connection (connect=connect@entry=0x5647d0e214c8) at /home/alice/git/10.2/sql/sql_connect.cc:1335
#15 0x00005647cdacd5f4 in handle_one_connection (arg=arg@entry=0x5647d0e214c8) at /home/alice/git/10.2/sql/sql_connect.cc:1241
#16 0x00005647cdd16534 in pfs_spawn_thread (arg=0x5647d0dbeb58) at /home/alice/git/10.2/storage/perfschema/pfs.cc:1862
#17 0x00007f7c025636ba in start_thread (arg=0x7f7bf4576700) at pthread_create.c:333
#18 0x00007f7c019f841d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

once I got this stacktrace (maybe somehow related with MDEV-15509?)

 10.2 52df8040264cc6
Thread 1 (Thread 0x7efd7055f700 (LWP 26451)):
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
#1  0x00005555a169ad9a in my_write_core (sig=sig@entry=11) at /home/alice/git/10.2/mysys/stacktrace.c:477
#2  0x00005555a11da7b4 in handle_fatal_signal (sig=11) at /home/alice/git/10.2/sql/signal_handler.cc:305
#3  <signal handler called>
#4  end_sj_materialize (join=<optimized out>, join_tab=<optimized out>, end_of_records=<optimized out>) at /home/alice/git/10.2/sql/sql_select.cc:10873
#5  0x00005555a1084104 in evaluate_join_record (join=join@entry=0x7efd300107f8, join_tab=join_tab@entry=0x7efd300cb0b0, error=<optimized out>) at /home/alice/git/10.2/sql/sql_select.cc:18930
#6  0x00005555a108be2a in sub_select (join=0x7efd300107f8, join_tab=0x7efd300cb0b0, end_of_records=<optimized out>) at /home/alice/git/10.2/sql/sql_select.cc:18710
#7  0x00005555a11529ef in join_tab_execution_startup (tab=tab@entry=0x7efd30102e60) at /home/alice/git/10.2/sql/opt_subselect.cc:5341
#8  0x00005555a108bdee in sub_select (join=0x7efd300107f8, join_tab=0x7efd30102e60, end_of_records=<optimized out>) at /home/alice/git/10.2/sql/sql_select.cc:18698
#9  0x00005555a1084104 in evaluate_join_record (join=join@entry=0x7efd300107f8, join_tab=join_tab@entry=0x7efd30102ab0, error=<optimized out>) at /home/alice/git/10.2/sql/sql_select.cc:18930
#10 0x00005555a108be2a in sub_select (join=0x7efd300107f8, join_tab=0x7efd30102ab0, end_of_records=<optimized out>) at /home/alice/git/10.2/sql/sql_select.cc:18710
#11 0x00005555a10a7cb2 in do_select (procedure=<optimized out>, join=0x7efd300107f8) at /home/alice/git/10.2/sql/sql_select.cc:18254
#12 JOIN::exec_inner (this=this@entry=0x7efd300107f8) at /home/alice/git/10.2/sql/sql_select.cc:3585
#13 0x00005555a10a7e59 in JOIN::exec (this=this@entry=0x7efd300107f8) at /home/alice/git/10.2/sql/sql_select.cc:3380
#14 0x00005555a10a7f9e in mysql_select (thd=thd@entry=0x7efd300009a8, tables=0x7efd3002c0a8, wild_num=<optimized out>, fields=..., conds=0x7efd300104c8, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2416184064, result=0x7efd3003a368, unit=0x7efd30029b60, select_lex=0x7efd3002a298) at /home/alice/git/10.2/sql/sql_select.cc:3780
#15 0x00005555a10a88de in handle_select (thd=thd@entry=0x7efd300009a8, lex=lex@entry=0x7efd30029a98, result=result@entry=0x7efd3003a368, setup_tables_done_option=setup_tables_done_option@entry=0) at /home/alice/git/10.2/sql/sql_select.cc:376
#16 0x00005555a0fa7036 in execute_sqlcom_select (thd=thd@entry=0x7efd300009a8, all_tables=0x7efd3002c0a8) at /home/alice/git/10.2/sql/sql_parse.cc:6476
#17 0x00005555a105ed16 in mysql_execute_command (thd=0x7efd300009a8) at /home/alice/git/10.2/sql/sql_parse.cc:3483
#18 0x00005555a1072c07 in Prepared_statement::execute (this=this@entry=0x7efd300c0b58, expanded_query=expanded_query@entry=0x7efd7055cbd0, open_cursor=open_cursor@entry=false) at /home/alice/git/10.2/sql/sql_prepare.cc:4774
#19 0x00005555a1073343 in Prepared_statement::execute_loop (this=0x7efd300c0b58, expanded_query=0x7efd7055cbd0, open_cursor=false, packet_end=<optimized out>, packet=<optimized out>) at /home/alice/git/10.2/sql/sql_prepare.cc:4203
#20 0x00005555a107388f in mysql_sql_stmt_execute (thd=thd@entry=0x7efd300009a8) at /home/alice/git/10.2/sql/sql_prepare.cc:3311
#21 0x00005555a1059ceb in mysql_execute_command (thd=thd@entry=0x7efd300009a8) at /home/alice/git/10.2/sql/sql_parse.cc:3499
#22 0x00005555a1060ddd in mysql_parse (thd=0x7efd300009a8, rawbuf=<optimized out>, length=13, parser_state=0x7efd7055e240, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /home/alice/git/10.2/sql/sql_parse.cc:7941
#23 0x00005555a1063976 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7efd300009a8, packet=packet@entry=0x7efd30006ce9 "", packet_length=packet_length@entry=13, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /home/alice/git/10.2/sql/sql_parse.cc:1820
#24 0x00005555a106424f in do_command (thd=0x7efd300009a8) at /home/alice/git/10.2/sql/sql_parse.cc:1374
#25 0x00005555a111e734 in do_handle_one_connection (connect=connect@entry=0x5555a423e058) at /home/alice/git/10.2/sql/sql_connect.cc:1335
#26 0x00005555a111e8a4 in handle_one_connection (arg=arg@entry=0x5555a423e058) at /home/alice/git/10.2/sql/sql_connect.cc:1241
#27 0x00005555a135cb84 in pfs_spawn_thread (arg=0x5555a4207db8) at /home/alice/git/10.2/storage/perfschema/pfs.cc:1862
#28 0x00007efd805456ba in start_thread (arg=0x7efd7055f700) at pthread_create.c:333
#29 0x00007efd7f9da41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Comment by Alice Sherepa [ 2018-07-30 ]

It was fixed in scope of MDEV-15247, by 5f2a67a6c35fd0d83302465 commit of Varun Gupta, the fix is included into 10.2.16

commit 5f2a67a6c35fd0d833024652ddc33eab8bcb1ed4
Author: Varun Gupta <varunraiko1803@gmail.com>
Date:   Wed Jun 20 02:36:00 2018 +0530
 
    MDEV-15247: Crash when SET NAMES 'utf8' is set
    
    In this case we are accessing incorrect memory when we have mergeable semi-joins.
    In the case when we have mergeable semi joins parent select will have a table count
    of all the tables in that select plus all the tables involved in the IN-subquery.
    But this table count does not include the "sjm table" (only includes the inner and outer tables)
    denotes as <subquery#> in explain.

Generated at Thu Feb 08 08:26:44 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.