[MDEV-15755] Query crashing MariaDB in cleanup_after_query Created: 2018-04-02  Updated: 2018-10-04  Resolved: 2018-05-15

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.2.8, 10.2.9, 10.3.1, 10.3.2, 10.3.3, 10.3.4, 10.3.5, 10.2, 10.3
Fix Version/s: 10.2.15, 10.3.7

Type: Bug Priority: Blocker
Reporter: Lori Patrick Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: None
Environment:

CentOS 7.3


Issue Links:
Relates
relates to MDEV-15065 Item_subselect::cleanup double free o... Closed
relates to MDEV-15585 signal 11 in cleanup phase Closed
relates to MDEV-15928 MariaDB SELECT query ends with lost c... Closed
relates to MDEV-16078 Server crashes in Item::delete_self o... Closed
relates to MDEV-16118 Frequenly & unpredicted crash (buffer... Closed

 Description   

I have a server I upgraded from 5.5.34 to 10.2.8 that had been stable for a couple weeks. Someone ran a month end query and the server crashed with the following message in the error log:

*** Error in `/usr/sbin/mysqld': double free or corruption (!prev): 0x00007f355c2d6c50 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c503)[0x7f408eb51503]
/usr/sbin/mysqld(_ZN14Item_subselect7cleanupEv+0x7e)[0x7f40910bffbe]
/usr/sbin/mysqld(_ZN11Query_arena10free_itemsEv+0x30)[0x7f4090e59030]
/usr/sbin/mysqld(_ZN3THD19cleanup_after_queryEv+0x5c)[0x7f4090e5b94c]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x237)[0x7f4090e91007]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x203d)[0x7f4090e9405d]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x149)[0x7f4090e94c69]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1aa)[0x7f4090f585da]
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x7f4090f586fd]
/lib64/libpthread.so.0(+0x7dc5)[0x7f4090579dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f408ebcc76d]
======= Memory map: ========
7f3540000000-7f3550000000 rw-p 00000000 00:00 0
7f355c000000-7f355c6e4000 rw-p 00000000 00:00 0
7f355c6e4000-7f3560000000 ---p 00000000 00:00 0
7f3560000000-7f35606da000 rw-p 00000000 00:00 0
7f35606da000-7f3564000000 ---p 00000000 00:00 0
7f3564000000-7f3564430000 rw-p 00000000 00:00 0
7f3564430000-7f3568000000 ---p 00000000 00:00 0
7f3568000000-7f356861f000 rw-p 00000000 00:00 0
7f356861f000-7f356c000000 ---p 00000000 00:00 0
7f356c000000-7f356c021000 rw-p 00000000 00:00 0
7f356c021000-7f3570000000 ---p 00000000 00:00 0
7f3570000000-7f3570021000 rw-p 00000000 00:00 0
7f3570021000-7f3574000000 ---p 00000000 00:00 0
7f3574000000-7f3574023000 rw-p 00000000 00:00 0
7f3574023000-7f3578000000 ---p 00000000 00:00 0
7f3578000000-7f3578021000 rw-p 00000000 00:00 0
7f3578021000-7f357c000000 ---p 00000000 00:00 0
7f357c000000-7f357c021000 rw-p 00000000 00:00 0
7f357c021000-7f3580000000 ---p 00000000 00:00 0
7f3580000000-7f3580022000 rw-p 00000000 00:00 0
7f3580022000-7f3584000000 ---p 00000000 00:00 0
7f3584000000-7f3584420000 rw-p 00000000 00:00 0
7f3584420000-7f3588000000 ---p 00000000 00:00 0
7f3588000000-7f3588021000 rw-p 00000000 00:00 0
7f3588021000-7f358c000000 ---p 00000000 00:00 0
7f358c000000-7f358fc37000 rw-p 00000000 00:00 0
7f358fc37000-7f3590000000 ---p 00000000 00:00 0
7f3590000000-7f3590420000 rw-p 00000000 00:00 0
7f3590420000-7f3594000000 ---p 00000000 00:00 0
7f3594000000-7f3594021000 rw-p 00000000 00:00 0
7f3594021000-7f3598000000 ---p 00000000 00:00 0
7f3598000000-7f3598038000 rw-p 00000000 00:00 0
7f3598038000-7f359c000000 ---p 00000000 00:00 0
7f359c000000-7f359c021000 rw-p 00000000 00:00 0
7f359c021000-7f35a0000000 ---p 00000000 00:00 0
7f35a0000000-7f35a0021000 rw-p 00000000 00:00 0
7f35a0021000-7f35a4000000 ---p 00000000 00:00 0
7f35a4000000-7f35a4021000 rw-p 00000000 00:00 0
7f35a4021000-7f35a8000000 ---p 00000000 00:00 0
7f35a8000000-7f35a8021000 rw-p 00000000 00:00 0
7f35a8021000-7f35ac000000 ---p 00000000 00:00 0
7f35ac000000-7f35ac401000 rw-p 00000000 00:00 0
7f35ac401000-7f35b0000000 ---p 00000000 00:00 0
7f35b0000000-7f35b0021000 rw-p 00000000 00:00 0
7f35b0021000-7f35b4000000 ---p 00000000 00:00 0
7f35b4000000-7f35b4ccd000 rw-p 00000000 00:00 0
7f35b4ccd000-7f35b8000000 ---p 00000000 00:00 0
7f35bafa2000-7f35c4000000 rw-p 00000000 00:00 0
7f35c4000000-7f35c4021000 rw-p 00000000 00:00 0
7f35c4021000-7f35c8000000 ---p 00000000 00:00 0
7f35c8000000-7f35c8021000 rw-p 00000000 00:00 0
7f35c8021000-7f35cc000000 ---p 00000000 00:00 0
7f35cc000000-7f35cc021000 rw-p 00000000 00:00 0
7f35cc021000-7f35d0000000 ---p 00000000 00:00 0
7f35d0000000-7f35d0021000 rw-p 00000000 00:00 0
7f35d0021000-7f35d4000000 ---p 00000000 00:00 0
7f35d77ff000-7f35d7800000 ---p 00000000 00:00 0
7f35d7800000-7f35d8000000 rw-p 00000000 00:00 0                          [stack:4286]
7f35d8000000-7f35d8021000 rw-p 00000000 00:00 0
7f35d8021000-7f35dc000000 ---p 00000000 00:00 0
7f35dc3d5000-7f35dc3d6000 ---p 00000000 00:00 0
7f35dc3d6000-7f35dcbd6000 rw-p 00000000 00:00 0                          [stack:4285]
7f35dcbd6000-7f35dcbd7000 ---p 00000000 00:00 0
7f35dcbd7000-7f35dd3d7000 rw-p 00000000 00:00 0                          [stack:4284]
7f35dd3d7000-7f35dd3d8000 ---p 00000000 00:00 0
7f35dd3d8000-7f35ddbd8000 rw-p 00000000 00:00 0                          [stack:4283]
7f35ddbd8000-7f35ddbd9000 ---p 00000000 00:00 0
7f35ddbd9000-7f35de3d9000 rw-p 00000000 00:00 0                          [stack:4282]
7f35de3d9000-7f35de3da000 ---p 00000000 00:00 0
7f35de3da000-7f35debda000 rw-p 00000000 00:00 0                          [stack:4281]
7f35debda000-7f35debdb000 ---p 00000000 00:00 0
7f35debdb000-7f35df3db000 rw-p 00000000 00:00 0                          [stack:4280]
7f35df3db000-7f35df3dc000 ---p 00000000 00:00 0
7f35df3dc000-7f35dfbdc000 rw-p 00000000 00:00 0                          [stack:4279]
7f35dfbdc000-7f35dfbdd000 ---p 00000000 00:00 0
7f35dfbdd000-7f35e03dd000 rw-p 00000000 00:00 0                          [stack:4278]
7f35e03dd000-7f35e03de000 ---p 00000000 00:00 0
7f35e03de000-7f35e0bde000 rw-p 00000000 00:00 0                          [stack:4277]
7f35e0bde000-7f35e0bdf000 ---p 00000000 00:00 0
7f35e0bdf000-7f35e13df000 rw-p 00000000 00:00 0                          [stack:4276]
7f35e13df000-7f35e13e0000 ---p 00000000 00:00 0
7f35e13e0000-7f35e1be0000 rw-p 00000000 00:00 0                          [stack:4275]
7f35e1be0000-7f35e1be1000 ---p 00000000 00:00 0
7f35e1be1000-7f35e23e1000 rw-p 00000000 00:00 0                          [stack:4274]
7f35e23e1000-7f35e23e2000 ---p 00000000 00:00 0
7f35e23e2000-7f35f1f12000 rw-p 00000000 00:00 0                          [stack:4273]
7f35f4000000-7f35f4021000 rw-p 00000000 00:00 0
7f35f4021000-7f35f8000000 ---p 00000000 00:00 0
7f35f8000000-7f35f8ccf000 rw-p 00000000 00:00 0
7f35f8ccf000-7f35fc000000 ---p 00000000 00:00 0
7f35fc000000-7f35fccca000 rw-p 00000000 00:00 0
7f35fccca000-7f3600000000 ---p 00000000 00:00 0
7f3604000000-7f3605371000 rw-p 00000000 00:00 0
7f3605371000-7f3608000000 ---p 00000000 00:00 0
7f360ab51000-7f360ab52000 ---p 00000000 00:00 0
7f360ab52000-7f360ab9b000 rw-p 00000000 00:00 0                          [stack:55091]
7f360ab9b000-7f360ab9c000 ---p 00000000 00:00 0
7f360ab9c000-7f360abe5000 rw-p 00000000 00:00 0                          [stack:55082]
7f360abe5000-7f360abea000 r-xp 00000000 08:03 1573466                    /usr/lib64/libnss_dns-2.17.so
7f360abea000-7f360ade9000 ---p 00005000 08:03 1573466                    /usr/lib64/libnss_dns-2.17.so
7f360ade9000-7f360adea000 r--p 00004000 08:03 1573466                    /usr/lib64/libnss_dns-2.17.so
7f360adea000-7f360adeb000 rw-p 00005000 08:03 1573466                    /usr/lib64/libnss_dns-2.17.so
7f360adeb000-7f360adf7000 r-xp 00000000 08:03 1573468                    /usr/lib64/libnss_files-2.17.so
7f360adf7000-7f360aff6000 ---p 0000c000 08:03 1573468                    /usr/lib64/libnss_files-2.17.so
7f360aff6000-7f360aff7000 r--p 0000b000 08:03 1573468                    /usr/lib64/libnss_files-2.17.so
7f360aff7000-7f360aff8000 rw-p 0000c000 08:03 1573468                    /usr/lib64/libnss_files-2.17.so
7f360aff8000-7f360affe000 rw-p 00000000 00:00 0
7f360affe000-7f360afff000 ---p 00000000 00:00 0
7f360afff000-7f360b7ff000 rw-p 00000000 00:00 0                          [stack:4493]
7f360b7ff000-7f360b800000 ---p 00000000 00:00 0
7f360b800000-7f360c000000 rw-p 00000000 00:00 0                          [stack:4492]
7f360c000000-7f360c021000 rw-p 00000000 00:00 0
7f360c021000-7f3610000000 ---p 00000000 00:00 0
7f3610030000-7f3610031000 ---p 00000000 00:00 0
7f3610031000-7f361007a000 rw-p 00000000 00:00 0                          [stack:51469]
7f361007a000-7f361007b000 ---p 00000000 00:00 0
7f361007b000-7f361087b000 rw-p 00000000 00:00 0                          [stack:4491]
7f361087b000-7f361087c000 ---p 00000000 00:00 0
7f361087c000-7f361107c000 rw-p 00000000 00:00 0                          [stack:4490]
7f361107c000-7f361107d000 ---p 00000000 00:00 0
7f361107d000-7f361187d000 rw-p 00000000 00:00 0                          [stack:4489]
7f361187d000-7f361187e000 ---p 00000000 00:00 0
7f361187e000-7f361207e000 rw-p 00000000 00:00 0                          [stack:4488]
7f361207e000-7f361207f000 ---p 00000000 00:00 0
7f361207f000-7f361287f000 rw-p 00000000 00:00 0                          [stack:4487]
7f361287f000-7f3612880000 ---p 00000000 00:00 0
7f3612880000-7f3613080000 rw-p 00000000 00:00 0                          [stack:4486]
7f3613080000-7f3613081000 ---p 00000000 00:00 0
7f3613081000-7f3613881000 rw-p 00000000 00:00 0                          [stack:4485]
7f3613881000-7f3613882000 ---p 00000000 00:00 0
7f3613882000-7f3614082000 rw-p 00000000 00:00 0                          [stack:4484]
7f3614082000-7f3614083000 ---p 00000000 00:00 0
7f3614083000-7f3614883000 rw-p 00000000 00:00 0                          [stack:4482]
7f3614883000-7f3614884000 ---p 00000000 00:00 0
7f3614884000-7f3615084000 rw-p 00000000 00:00 0                          [stack:4481]
7f3615084000-7f3615085000 ---p 00000000 00:00 0
7f3615085000-7f3615885000 rw-p 00000000 00:00 0                          [stack:4480]
7f3615885000-7f3615886000 ---p 00000000 00:00 0
7f3615886000-7f3e70000000 rw-p 00000000 00:00 0                          [stack:4483]
7f3e70000000-7f3e70021000 rw-p 00000000 00:00 0
7f3e70021000-7f3e74000000 ---p 00000000 00:00 0
7f3e74033000-7f3e74034000 ---p 00000000 00:00 0
7f3e74034000-7f3e7407d000 rw-p 00000000 00:00 0                          [stack:51470]
7f3e7407d000-7f3e7407e000 ---p 00000000 00:00 0
7f3e7407e000-7f3e740c7000 rw-p 00000000 00:00 0                          [stack:4831]
7f3e740c7000-7f3e740c8000 ---p 00000000 00:00 0
7f3e740c8000-7f407c000000 rw-p 00000000 00:00 0                          [stack:4550]
7f407c000000-7f407c021000 rw-p 00000000 00:00 0
7f407c021000-7f4080000000 ---p 00000000 00:00 0
7f408002e000-7f408002f000 ---p 00000000 00:00 0
7f408002f000-7f4080078000 rw-p 00000000 00:00 0                          [stack:4502]
7f4080078000-7f4080079000 ---p 00000000 00:00 0
7f4080079000-7f408201e000 rw-p 00000000 00:00 0                          [stack:4501]
7f408201e000-7f408201f000 ---p 00000000 00:00 0
7f408201f000-7f4082f54000 rw-p 00000000 00:00 0                          [stack:4216]
7f4082f54000-7f4082f56000 r-xp 00000000 08:03 1578123                    /usr/lib64/mysql/plugin/sql_errlog.so
7f4082f56000-7f4083155000 ---p 00002000 08:03 1578123                    /usr/lib64/mysql/plugin/sql_errlog.so
7f4083155000-7f4083156000 r--p 00001000 08:03 1578123                    /usr/lib64/mysql/plugin/sql_errlog.so
7f4083156000-7f4083157000 rw-p 00002000 08:03 1578123                    /usr/lib64/mysql/plugin/sql_errlog.so
7f4083157000-7f408315b000 r-xp 00000000 08:03 1573939                    /usr/lib64/libcap-ng.so.0.0.0
7f408315b000-7f408335b000 ---p 00004000 08:03 1573939                    /usr/lib64/libcap-ng.so.0.0.0
7f408335b000-7f408335c000 r--p 00004000 08:03 1573939                    /usr/lib64/libcap-ng.so.0.0.0
7f408335c000-7f408335d000 rw-p 00005000 08:03 1573939                    /usr/lib64/libcap-ng.so.0.0.0
7f408335d000-7f4083379000 r-xp 00000000 08:03 1573728                    /usr/lib64/libaudit.so.1.0.0
7f4083379000-7f4083579000 ---p 0001c000 08:03 1573728                    /usr/lib64/libaudit.so.1.0.0
7f4083579000-7f408357a000 r--p 0001c000 08:03 1573728                    /usr/lib64/libaudit.so.1.0.0
7f408357a000-7f408357b000 rw-p 0001d000 08:03 1573728                    /usr/lib64/libaudit.so.1.0.0
7f408357b000-7f4083585000 rw-p 00000000 00:00 0
7f4083585000-7f4083592000 r-xp 00000000 08:03 1575772                    /usr/lib64/libpam.so.0.83.1
7f4083592000-7f4083792000 ---p 0000d000 08:03 1575772                    /usr/lib64/libpam.so.0.83.1
7f4083792000-7f4083793000 r--p 0000d000 08:03 1575772                    /usr/lib64/libpam.so.0.83.1
7f4083793000-7f4083794000 rw-p 0000e000 08:03 1575772                    /usr/lib64/libpam.so.0.83.1
7f4083794000-7f4083796000 r-xp 00000000 08:03 1578104                    /usr/lib64/mysql/plugin/auth_pam.so
7f4083796000-7f4083995000 ---p 00002000 08:03 1578104                    /usr/lib64/mysql/plugin/auth_pam.so
7f4083995000-7f4083996000 r--p 00001000 08:03 1578104                    /usr/lib64/mysql/plugin/auth_pam.so
7f4083996000-7f4083997000 rw-p 00002000 08:03 1578104                    /usr/lib64/mysql/plugin/auth_pam.so
7f4083997000-7f4083998000 ---p 00000000 00:00 0
7f4083998000-7f408c194000 rw-p 00000000 00:00 0                          [stack:3662]
7f408c194000-7f408c197000 r-xp 00000000 08:03 1573818                    /usr/lib64/libkeyutils.so.1.5
7f408c197000-7f408c396000 ---p 00003000 08:03 1573818                    /usr/lib64/libkeyutils.so.1.5
7f408c396000-7f408c397000 r--p 00002000 08:03 1573818                    /usr/lib64/libkeyutils.so.1.5
7f408c397000-7f408c398000 rw-p 00003000 08:03 1573818                    /usr/lib64/libkeyutils.so.1.5
7f408c398000-7f408c3a5000 r-xp 00000000 08:03 1579319                    /usr/lib64/libkrb5support.so.0.1
7f408c3a5000-7f408c5a5000 ---p 0000d000 08:03 1579319                    /usr/lib64/libkrb5support.so.0.1
7f408c5a5000-7f408c5a6000 r--p 0000d000 08:03 1579319                    /usr/lib64/libkrb5support.so.0.1
7f408c5a6000-7f408c5a7000 rw-p 0000e000 08:03 1579319                    /usr/lib64/libkrb5support.so.0.1
7f408c5a7000-7f408c5b6000 r-xp 00000000 08:03 1573733                    /usr/lib64/libbz2.so.1.0.6
7f408c5b6000-7f408c7b5000 ---p 0000f000 08:03 1573733                    /usr/lib64/libbz2.so.1.0.6
7f408c7b5000-7f408c7b6000 r--p 0000e000 08:03 1573733                    /usr/lib64/libbz2.so.1.0.6
7f408c7b6000-7f408c7b7000 rw-p 0000f000 08:03 1573733                    /usr/lib64/libbz2.so.1.0.6
7f408c7b7000-7f408c7ce000 r-xp 00000000 08:03 1573731                    /usr/lib64/libelf-0.166.so
7f408c7ce000-7f408c9cd000 ---p 00017000 08:03 1573731                    /usr/lib64/libelf-0.166.so
7f408c9cd000-7f408c9ce000 r--p 00016000 08:03 1573731                    /usr/lib64/libelf-0.166.so
7f408c9ce000-7f408c9cf000 rw-p 00017000 08:03 1573731                    /usr/lib64/libelf-0.166.so
7f408c9cf000-7f408ca2f000 r-xp 00000000 08:03 1573614                    /usr/lib64/libpcre.so.1.2.0
7f408ca2f000-7f408cc2e000 ---p 00060000 08:03 1573614                    /usr/lib64/libpcre.so.1.2.0
7f408cc2e000-7f408cc2f000 r--p 0005f000 08:03 1573614                    /usr/lib64/libpcre.so.1.2.0
7f408cc2f000-7f408cc30000 rw-p 00060000 08:03 1573614                    /usr/lib64/libpcre.so.1.2.0
7f408cc30000-7f408cc34000 r-xp 00000000 08:03 1573772                    /usr/lib64/libattr.so.1.1.0
7f408cc34000-7f408ce33000 ---p 00004000 08:03 1573772                    /usr/lib64/libattr.so.1.1.0
7f408ce33000-7f408ce34000 r--p 00003000 08:03 1573772                    /usr/lib64/libattr.so.1.1.0
7f408ce34000-7f408ce35000 rw-p 00004000 08:03 1573772                    /usr/lib64/libattr.so.1.1.0
7f408ce35000-7f408ce64000 r-xp 00000000 08:03 1575237                    /usr/lib64/libk5crypto.so.3.1
7f408ce64000-7f408d063000 ---p 0002f000 08:03 1575237                    /usr/lib64/libk5crypto.so.3.1
7f408d063000-7f408d065000 r--p 0002e000 08:03 1575237                    /usr/lib64/libk5crypto.so.3.1
7f408d065000-7f408d066000 rw-p 00030000 08:03 1575237                    /usr/lib64/libk5crypto.so.3.1
7f408d066000-7f408d067000 rw-p 00000000 00:00 0
7f408d067000-7f408d06a000 r-xp 00000000 08:03 1573638                    /usr/lib64/libcom_err.so.2.1
7f408d06a000-7f408d269000 ---p 00003000 08:03 1573638                    /usr/lib64/libcom_err.so.2.1
7f408d269000-7f408d26a000 r--p 00002000 08:03 1573638                    /usr/lib64/libcom_err.so.2.1
7f408d26a000-7f408d26b000 rw-p 00003000 08:03 1573638                    /usr/lib64/libcom_err.so.2.1
7f408d26b000-7f408d341000 r-xp 00000000 08:03 1575247                    /usr/lib64/libkrb5.so.3.3
7f408d341000-7f408d541000 ---p 000d6000 08:03 1575247                    /usr/lib64/libkrb5.so.3.3
7f408d541000-7f408d54f000 r--p 000d6000 08:03 1575247                    /usr/lib64/libkrb5.so.3.3
7f408d54f000-7f408d552000 rw-p 000e4000 08:03 1575247                    /usr/lib64/libkrb5.so.3.3
7f408d552000-7f408d59d000 r-xp 00000000 08:03 1573787                    /usr/lib64/libgssapi_krb5.so.2.2
7f408d59d000-7f408d79d000 ---p 0004b000 08:03 1573787                    /usr/lib64/libgssapi_krb5.so.2.2
7f408d79d000-7f408d79e000 r--p 0004b000 08:03 1573787                    /usr/lib64/libgssapi_krb5.so.2.2
7f408d79e000-7f408d7a0000 rw-p 0004c000 08:03 1573787                    /usr/lib64/libgssapi_krb5.so.2.2
7f408d7a0000-7f408d7b5000 r-xp 00000000 08:03 1579713                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f408d7b5000-7f408d9b4000 ---p 00015000 08:03 1579713                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f408d9b4000-7f408d9b5000 r--p 00014000 08:03 1579713                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f408d9b5000-7f408d9b6000 rw-p 00015000 08:03 1579713                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f408d9b6000-7f408d9fb000 r-xp 00000000 08:03 1573931                    /usr/lib64/libdw-0.166.so
7f408d9fb000-7f408dbfb000 ---p 00045000 08:03 1573931                    /usr/lib64/libdw-0.166.so
7f408dbfb000-7f408dbfd000 r--p 00045000 08:03 1573931                    /usr/lib64/libdw-0.166.so
7f408dbfd000-7f408dbfe000 rw-p 00047000 08:03 1573931                    /usr/lib64/libdw-0.166.so
7f408dbfe000-7f408dc14000 r-xp 00000000 08:03 1579301                    /usr/lib64/libresolv-2.17.so
7f408dc14000-7f408de14000 ---p 00016000 08:03 1579301                    /usr/lib64/libresolv-2.17.so
7f408de14000-7f408de15000 r--p 00016000 08:03 1579301                    /usr/lib64/libresolv-2.17.so
7f408de15000-7f408de16000 rw-p 00017000 08:03 1579301                    /usr/lib64/libresolv-2.17.so
7f408de16000-7f408de18000 rw-p 00000000 00:00 0
7f408de18000-7f408de1c000 r-xp 00000000 08:03 1573820                    /usr/lib64/libgpg-error.so.0.10.0
7f408de1c000-7f408e01b000 ---p 00004000 08:03 1573820                    /usr/lib64/libgpg-error.so.0.10.0
7f408e01b000-7f408e01c000 r--p 00003000 08:03 1573820                    /usr/lib64/libgpg-error.so.0.10.0
7f408e01c000-7f408e01d000 rw-p 00004000 08:03 1573820                    /usr/lib64/libgpg-error.so.0.10.0
7f408e01d000-7f408e099000 r-xp 00000000 08:03 1573828                    /usr/lib64/libgcrypt.so.11.8.2
7f408e099000-7f408e299000 ---p 0007c000 08:03 1573828                    /usr/lib64/libgcrypt.so.11.8.2
7f408e299000-7f408e29a000 r--p 0007c000 08:03 1573828                    /usr/lib64/libgcrypt.so.11.8.2
7f408e29a000-7f408e29d000 rw-p 0007d000 08:03 1573828                    /usr/lib64/libgcrypt.so.11.8.2
7f408e29d000-7f408e29e000 rw-p 00000000 00:00 0
7f408e29e000-7f408e2c2000 r-xp 00000000 08:03 1573624                    /usr/lib64/libselinux.so.1
7f408e2c2000-7f408e4c1000 ---p 00024000 08:03 1573624                    /usr/lib64/libselinux.so.1
7f408e4c1000-7f408e4c2000 r--p 00023000 08:03 1573624                    /usr/lib64/libselinux.so.1
7f408e4c2000-7f408e4c3000 rw-p 00024000 08:03 1573624                    /usr/lib64/libselinux.so.1
7f408e4c3000-7f408e4c5000 rw-p 00000000 00:00 0
7f408e4c5000-7f408e4cc000 r-xp 00000000 08:03 1579302                    /usr/lib64/librt-2.17.so
7f408e4cc000-7f408e6cb000 ---p 00007000 08:03 1579302                    /usr/lib64/librt-2.17.so
7f408e6cb000-7f408e6cc000 r--p 00006000 08:03 1579302                    /usr/lib64/librt-2.17.so
7f408e6cc000-7f408e6cd000 rw-p 00007000 08:03 1579302                    /usr/lib64/librt-2.17.so
7f408e6cd000-7f408e6d1000 r-xp 00000000 08:03 1573774                    /usr/lib64/libcap.so.2.22
7f408e6d1000-7f408e8d0000 ---p 00004000 08:03 1573774                    /usr/lib64/libcap.so.2.22
7f408e8d0000-7f408e8d1000 r--p 00003000 08:03 1573774                    /usr/lib64/libcap.so.2.22
7f408e8d1000-7f408e8d2000 rw-p 00004000 08:03 1573774                    /usr/lib64/libcap.so.2.22
7f408e8d2000-7f408e8d4000 r-xp 00000000 08:03 1573444                    /usr/lib64/libfreebl3.so
7f408e8d4000-7f408ead3000 ---p 00002000 08:03 1573444                    /usr/lib64/libfreebl3.so
7f408ead3000-7f408ead4000 r--p 00001000 08:03 1573444                    /usr/lib64/libfreebl3.so
7f408ead4000-7f408ead5000 rw-p 00002000 08:03 1573444                    /usr/lib64/libfreebl3.so
7f408ead5000-7f408ec8c000 r-xp 00000000 08:03 1573450                    /usr/lib64/libc-2.17.so
7f408ec8c000-7f408ee8b000 ---p 001b7000 08:03 1573450                    /usr/lib64/libc-2.17.so
7f408ee8b000-7f408ee8f000 r--p 001b6000 08:03 1573450                    /usr/lib64/libc-2.17.so
7f408ee8f000-7f408ee91000 rw-p 001ba000 08:03 1573450                    /usr/lib64/libc-2.17.so
7f408ee91000-7f408ee96000 rw-p 00000000 00:00 0
7f408ee96000-7f408ef96000 r-xp 00000000 08:03 1579298                    /usr/lib64/libm-2.17.so
7f408ef96000-7f408f196000 ---p 00100000 08:03 1579298                    /usr/lib64/libm-2.17.so
7f408f196000-7f408f197000 r--p 00100000 08:03 1579298                    /usr/lib64/libm-2.17.so
7f408f197000-7f408f198000 rw-p 00101000 08:03 1579298                    /usr/lib64/libm-2.17.so
7f408f198000-7f408f281000 r-xp 00000000 08:03 1573497                    /usr/lib64/libstdc++.so.6.0.19
7f408f281000-7f408f481000 ---p 000e9000 08:03 1573497                    /usr/lib64/libstdc++.so.6.0.19
7f408f481000-7f408f48a000 r--p 000e9000 08:03 1573497                    /usr/lib64/libstdc++.so.6.0.19
7f408f48a000-7f408f48c000 rw-p 000f2000 08:03 1573497                    /usr/lib64/libstdc++.so.6.0.19
7f408f48c000-7f408f4a1000 rw-p 00000000 00:00 0
7f408f4a1000-7f408f4a3000 r-xp 00000000 08:03 1579297                    /usr/lib64/libdl-2.17.so
7f408f4a3000-7f408f6a3000 ---p 00002000 08:03 1579297                    /usr/lib64/libdl-2.17.so
7f408f6a3000-7f408f6a4000 r--p 00002000 08:03 1579297                    /usr/lib64/libdl-2.17.so
7f408f6a4000-7f408f6a5000 rw-p 00003000 08:03 1579297                    /usr/lib64/libdl-2.17.so
7f408f6a5000-7f408f865000 r-xp 00000000 08:03 1575255                    /usr/lib64/libcrypto.so.1.0.1e
7f408f865000-7f408fa65000 ---p 001c0000 08:03 1575255                    /usr/lib64/libcrypto.so.1.0.1e
7f408fa65000-7f408fa7f000 r--p 001c0000 08:03 1575255                    /usr/lib64/libcrypto.so.1.0.1e
7f408fa7f000-7f408fa8b000 rw-p 001da000 08:03 1575255                    /usr/lib64/libcrypto.so.1.0.1e
7f408fa8b000-7f408fa8f000 rw-p 00000000 00:00 0
7f408fa8f000-7f408faf3000 r-xp 00000000 08:03 1579321                    /usr/lib64/libssl.so.1.0.1e
7f408faf3000-7f408fcf2000 ---p 00064000 08:03 1579321                    /usr/lib64/libssl.so.1.0.1e
7f408fcf2000-7f408fcf6000 r--p 00063000 08:03 1579321                    /usr/lib64/libssl.so.1.0.1e
7f408fcf6000-7f408fcfd000 rw-p 00067000 08:03 1579321                    /usr/lib64/libssl.so.1.0.1e
7f408fcfd000-7f408fd05000 r-xp 00000000 08:03 1573454                    /usr/lib64/libcrypt-2.17.so
7f408fd05000-7f408ff04000 ---p 00008000 08:03 1573454                    /usr/lib64/libcrypt-2.17.so
7f408ff04000-7f408ff05000 r--p 00007000 08:03 1573454                    /usr/lib64/libcrypt-2.17.so
7f408ff05000-7f408ff06000 rw-p 00008000 08:03 1573454                    /usr/lib64/libcrypt-2.17.so
7f408ff06000-7f408ff34000 rw-p 00000000 00:00 0
7f408ff34000-7f408ff49000 r-xp 00000000 08:03 1573627                    /usr/lib64/libz.so.1.2.7
7f408ff49000-7f4090148000 ---p 00015000 08:03 1573627                    /usr/lib64/libz.so.1.2.7
7f4090148000-7f4090149000 r--p 00014000 08:03 1573627                    /usr/lib64/libz.so.1.2.7
7f4090149000-7f409014a000 rw-p 00015000 08:03 1573627                    /usr/lib64/libz.so.1.2.7
7f409014a000-7f409014b000 r-xp 00000000 08:03 1576535                    /usr/lib64/libaio.so.1.0.1
7f409014b000-7f409034a000 ---p 00001000 08:03 1576535                    /usr/lib64/libaio.so.1.0.1
7f409034a000-7f409034b000 r--p 00000000 08:03 1576535                    /usr/lib64/libaio.so.1.0.1
7f409034b000-7f409034c000 rw-p 00001000 08:03 1576535                    /usr/lib64/libaio.so.1.0.1
7f409034c000-7f4090371000 r-xp 00000000 08:03 1573612                    /usr/lib64/liblzma.so.5.2.2
7f4090371000-7f4090570000 ---p 00025000 08:03 1573612                    /usr/lib64/liblzma.so.5.2.2
7f4090570000-7f4090571000 r--p 00024000 08:03 1573612                    /usr/lib64/liblzma.so.5.2.2
7f4090571000-7f4090572000 rw-p 00025000 08:03 1573612                    /usr/lib64/liblzma.so.5.2.2
7f4090572000-7f4090589000 r-xp 00000000 08:03 1573476                    /usr/lib64/libpthread-2.17.so
7f4090589000-7f4090788000 ---p 00017000 08:03 1573476                    /usr/lib64/libpthread-2.17.so
7f4090788000-7f4090789000 r--p 00016000 08:03 1573476                    /usr/lib64/libpthread-2.17.so
7f4090789000-7f409078a000 rw-p 00017000 08:03 1573476                    /usr/lib64/libpthread-2.17.so
7f409078a000-7f409078e000 rw-p 00000000 00:00 0
7f409078e000-7f40907ae000 r-xp 00000000 08:03 1573443                    /usr/lib64/ld-2.17.so
7f40907b3000-7f40907b4000 rw-p 00000000 00:00 0
7f40907b4000-7f40907b5000 ---p 00000000 00:00 0
7f40907b5000-7f40907fe000 rw-p 00000000 00:00 0                          [stack:4500]
7f40907fe000-7f40907ff000 ---p 00000000 00:00 0
7f40907ff000-7f4090848000 rw-p 00000000 00:00 0                          [stack:4499]
7f4090848000-7f4090849000 ---p 00000000 00:00 0
7f4090849000-7f4090893000 rw-p 00000000 00:00 0                          [stack:4494]
7f4090893000-7f4090896000 rw-s 00000000 00:0a 29155                      /[aio] (deleted)
7f4090896000-7f4090899000 rw-s 00000000 00:0a 29154                      /[aio] (deleted)
7f4090899000-7f409089c000 rw-s 00000000 00:0a 29153                      /[aio] (deleted)
7f409089c000-7f409089f000 rw-s 00000000 00:0a 29152                      /[aio] (deleted)
7f409089f000-7f40908a2000 rw-s 00000000 00:0a 29151                      /[aio] (deleted)
7f40908a2000-7f40908a5000 rw-s 00000000 00:0a 29150                      /[aio] (deleted)
7f40908a5000-7f4090975000 rw-p 00000000 00:00 0
7f4090975000-7f409099b000 r-xp 00000000 08:03 1575819                    /usr/lib64/libsystemd.so.0.6.0
7f409099b000-7f409099c000 r--p 00025000 08:03 1575819                    /usr/lib64/libsystemd.so.0.6.0
7f409099c000-7f409099d000 rw-p 00026000 08:03 1575819                    /usr/lib64/libsystemd.so.0.6.0
7f409099d000-7f409099f000 rw-p 00000000 00:00 0
7f409099f000-7f40909a0000 rw-s 00000000 00:0a 29156                      /[aio] (deleted)
7f40909a0000-7f40909a3000 rw-s 00000000 00:0a 29149                      /[aio] (deleted)
7f40909a3000-7f40909a6000 rw-s 00000000 00:0a 29148                      /[aio] (deleted)
7f40909a6000-7f40909a9000 rw-s 00000000 00:0a 29147                      /[aio] (deleted)
7f40909a9000-7f40909ac000 rw-s 00000000 00:0a 29146                      /[aio] (deleted)
7f40909ac000-7f40909ad000 rw-s 00000000 00:0a 29145                      /[aio] (deleted)
7f40909ad000-7f40909ae000 rw-p 00000000 00:00 0
7f40909ae000-7f40909af000 r--p 00020000 08:03 1573443                    /usr/lib64/ld-2.17.so
7f40909af000-7f40909b0000 rw-p 00021000 08:03 1573443                    /usr/lib64/ld-2.17.so
7f40909b0000-7f40909b1000 rw-p 00000000 00:00 0
7f40909b1000-7f4091b0e000 r-xp 00000000 08:03 1578125                    /usr/sbin/mysqld
7f4091d0e000-7f4091de8000 r--p 0115d000 08:03 1578125                    /usr/sbin/mysqld
7f4091de8000-7f4091e9f000 rw-p 01237000 08:03 1578125                    /usr/sbin/mysqld
7f4091e9f000-7f409272e000 rw-p 00000000 00:00 0
7f4093c8f000-7f40c4547000 rw-p 00000000 00:00 0                          [heap]
7ffe4c2d6000-7ffe4c2f7000 rw-p 00000000 00:00 0                          [stack]
7ffe4c368000-7ffe4c36a000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

180401  6:02:09 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.2.8-MariaDB-log
key_buffer_size=8589934592
read_buffer_size=2097152
max_used_connections=6
max_threads=1502
thread_count=15
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 14572555 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f355c0008c8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f360abe3d70 thread_stack 0x49000
(my_addr_resolve failure: fork)

/usr/sbin/mysqld(my_print_stacktrace+0x2e) [0x7f40915d2a7e]
/usr/sbin/mysqld(handle_fatal_signal+0x30d) [0x7f4091018dbd]
/lib64/libpthread.so.0(+0xf370) [0x7f4090581370]
/lib64/libc.so.6(gsignal+0x37) [0x7f408eb0a1d7]
/lib64/libc.so.6(abort+0x148) [0x7f408eb0b8c8]
/lib64/libc.so.6(+0x74f07) [0x7f408eb49f07]
/lib64/libc.so.6(+0x7c503) [0x7f408eb51503]
/usr/sbin/mysqld(Item_subselect::cleanup()+0x7e) [0x7f40910bffbe]
/usr/sbin/mysqld(Query_arena::free_items()+0x30) [0x7f4090e59030]
/usr/sbin/mysqld(THD::cleanup_after_query()+0x5c) [0x7f4090e5b94c]
/usr/sbin/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool)+0x237) [0x7f4090e91007]
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool)+0x203d) [0x7f4090e9405d]
/usr/sbin/mysqld(do_command(THD*)+0x149) [0x7f4090e94c69]
/usr/sbin/mysqld(do_handle_one_connection(CONNECT*)+0x1aa) [0x7f4090f585da]
/usr/sbin/mysqld(handle_one_connection+0x3d) [0x7f4090f586fd]
/lib64/libpthread.so.0(+0x7dc5) [0x7f4090579dc5]
/lib64/libc.so.6(clone+0x6d) [0x7f408ebcc76d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f355c01f120): SELECT CONCAT(exchange_ts_str,  '_', symbol) AS trade_key,             exchange, exchange_ts_str, fill_id, origin, user_id, context, symbol, side, price, quantity, order_id, leg_type, account_number              FR
OM `drop_copy_fill_events`              WHERE ts_str like '2018-03%' AND CONCAT(exchange_ts_str,  '_', symbol)              IN (                 SELECT distinct CONCAT(exchange_ts_str,  '_', symbol)                 FROM `drop_copy_fill_ev
ents`                 where ts_str like '2018-03%' AND                 exchange NOT IN ('ICE', 'NYSE_LIFFE_US')                 group by CONCAT(exchange_ts_str,  '_', symbol)                 having count(*) > 1 and                 count(d
istinct price) = 1 and                 count(distinct quantity) = 1 and                 count(distinct side) = 2 and                 count(distinct user_id) = 2                )             ORDER BY account_number
Connection ID (thread ID): 19
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,
firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache
=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.



 Comments   
Comment by Elena Stepanova [ 2018-04-02 ]

Is the crash reproducible when you re-run the query?

Comment by Lori Patrick [ 2018-04-02 ]

I have been able to reproduce the error on other servers. I stepped thru all available packages at yum.mariadb.org/ above 10.2.8 and the problem persisted.

Testing on 10.1.24 resulted in the query completing without error, so the issues appears to have been introduced somewhere between 10.1.24 and 10.2.8.

Tried changing the optimizer switches: orderby_uses_equalities=off and condition_pushdown_for_derived==off and problem persisted.

Comment by Elena Stepanova [ 2018-04-02 ]

10.2.8 is pretty old, but still, we are very interested in this particular failure.
Would you be able to provide a dataset on which the crash is reliably reproducible with this query?

Comment by Lori Patrick [ 2018-04-02 ]

table structure

CREATE TABLE `drop_copy_fill_events` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `tstamp` timestamp(6) NOT NULL DEFAULT current_timestamp(6),
  `curr_update_seq` bigint(20) unsigned NOT NULL DEFAULT 0,
  `origin` varchar(50) NOT NULL DEFAULT '',
  `order_id` varchar(64) DEFAULT NULL,
  `order_state` varchar(30) NOT NULL DEFAULT '',
  `event` varchar(30) NOT NULL DEFAULT '',
  `gv_ts` bigint(20) unsigned NOT NULL DEFAULT 0,
  `exchange_ts` bigint(20) unsigned NOT NULL DEFAULT 0,
  `sent_ts` bigint(20) unsigned NOT NULL DEFAULT 0,
  `ts` bigint(20) unsigned NOT NULL DEFAULT 0,
  `exchange_order_id` varchar(50) DEFAULT '',
  `ad_id` varchar(100) DEFAULT '',
  `exchange` varchar(30) NOT NULL DEFAULT '',
  `symbol` varchar(40) DEFAULT '',
  `root_symbol` varchar(30) NOT NULL DEFAULT '',
  `side` varchar(10) NOT NULL DEFAULT '',
  `order_type` varchar(30) NOT NULL DEFAULT '',
  `price` decimal(15,8) NOT NULL DEFAULT 0.00000000,
  `quantity` int(10) unsigned NOT NULL DEFAULT 0,
  `quantity_numerator` int(10) DEFAULT 0,
  `quantity_denominator` int(10) DEFAULT 0,
  `remaining_quantity` int(10) unsigned NOT NULL DEFAULT 0,
  `user_id` varchar(20) NOT NULL DEFAULT '',
  `context` varchar(20) NOT NULL DEFAULT '',
  `ad_text` varchar(100) DEFAULT '',
  `exchange_session` varchar(55) DEFAULT NULL,
  `trader_id` varchar(20) DEFAULT '',
  `fill_id` varchar(100) DEFAULT '',
  `trade_id` varchar(30) DEFAULT NULL,
  `match_id` varchar(20) DEFAULT NULL,
  `total_filled` int(10) unsigned NOT NULL DEFAULT 0,
  `trade_date` date DEFAULT '0000-00-00',
  `leg_type` varchar(30) DEFAULT '',
  `exchange_account` varchar(30) DEFAULT '',
  `account_number` varchar(30) DEFAULT '',
  `liquidity` varchar(20) DEFAULT '',
  `busted` tinyint(4) DEFAULT 0,
  `corrected` tinyint(4) DEFAULT 0,
  `insert_ts` bigint(20) unsigned NOT NULL DEFAULT 0,
  `ts_str` varchar(50) DEFAULT NULL,
  `gv_ts_str` varchar(50) DEFAULT NULL,
  `exchange_ts_str` varchar(50) DEFAULT NULL,
  `sent_ts_str` varchar(50) DEFAULT NULL,
  `value_date` date DEFAULT '0000-00-00',
  `short_sell` tinyint(1) NOT NULL DEFAULT 0,
  PRIMARY KEY (`id`),
  KEY `user_idx` (`user_id`),
  KEY `exchange_ts_idx` (`exchange_ts`),
  KEY `fill_id_str` (`fill_id`),
  KEY `exchange_order_id_idx` (`exchange_order_id`),
  KEY `ts_str_idx` (`ts_str`),
  KEY `tstamp_idx` (`tstamp`),
  KEY `order_id_user_id_idx` (`order_id`,`user_id`)
) ENGINE=InnoDB

query

SELECT
  CONCAT(exchange_ts_str, '_', symbol) AS trade_key,
  exchange,
  exchange_ts_str,
  fill_id,
  origin,
  user_id,
  context,
  symbol,
  side,
  price,
  quantity,
  order_id,
  leg_type,
  account_number
FROM
  `drop_copy_fill_events`
WHERE ts_str LIKE '2018-03%'
  AND CONCAT(exchange_ts_str, '_', symbol) IN
  (SELECT DISTINCT
    CONCAT(exchange_ts_str, '_', symbol)
  FROM
    `drop_copy_fill_events`
  WHERE ts_str LIKE '2018-03%'
    AND exchange NOT IN ('ICE', 'NYSE_LIFFE_US')
  GROUP BY CONCAT(exchange_ts_str, '_', symbol)
  HAVING COUNT (*) > 1
    AND COUNT(DISTINCT price) = 1
    AND COUNT(DISTINCT quantity) = 1
    AND COUNT(DISTINCT side) = 2
    AND COUNT(DISTINCT user_id) = 2)
ORDER BY account_number;

Comment by Lori Patrick [ 2018-04-02 ]

If I convert it to a cte:
WITH cte AS
(SELECT DISTINCT
CONCAT(exchange_ts_str, '_', symbol) AS symbol
FROM
`drop_copy_fill_events`
WHERE ts_str LIKE '2018-03%'
AND exchange NOT IN ('ICE', 'NYSE_LIFFE_US')
GROUP BY CONCAT(exchange_ts_str, '_', symbol)
HAVING COUNT > 1
AND COUNT(DISTINCT price) = 1
AND COUNT(DISTINCT quantity) = 1
AND COUNT(DISTINCT side) = 2
AND COUNT(DISTINCT user_id) = 2)
SELECT
CONCAT(f.exchange_ts_str, '_', f.symbol) AS trade_key,
f.exchange,
f.exchange_ts_str,
f.fill_id,
f.origin,
f.user_id,
f.context,
f.symbol,
f.side,
f.price,
f.quantity,
f.order_id,
f.leg_type,
f.account_number
FROM
`drop_copy_fill_events` f, cte
WHERE cte.symbol = CONCAT(f.exchange_ts_str, '_', f.symbol)
AND f.ts_str LIKE '2018-03%'
ORDER BY account_number;

It runs fine. However for backwards compatibility - changing all code is tricky. Not to mention a server crashing over a select query is not viable for production.

Comment by Elena Stepanova [ 2018-04-02 ]

Does it actually crash for you on the empty table?
If it does, please also attach or paste your cnf file(s). If it doesn't crash on the empty table, then the question remains – would you be able to provide the data set?

Comment by Lori Patrick [ 2018-04-02 ]

my.cnf for the server:

!include_dir /etc/mysql
 
#
# This  started with MariaDB example config file for systems with 4GB of memory
# running mostly MariaDB using InnoDB only tables and performing complex
# queries with few connections.
#
 
#
# The following options will be read by MariaDB client applications.
[client]
port            = 3306
socket          = /srv/db_trn/run/mysql.sock
 
 
[mysqld]
 
# generic configuration options
port            = 3306
socket          = /srv/db_trn/run/mysql.sock
pid-file        = /srv/db_trn/run/ch0pxdbpts05.pid
 
datadir         = /srv/dbdata_fast/mysql
back_log        = 50
max_connections = 1500
table_open_cache = 1000
table_definition_cache = 400
skip-external-locking
max_allowed_packet = 16M
optimizer_switch = default
#thread_cache_size = 16
#thread_concurrency = 8
query_cache_type = 1
query_cache_size = 128M
query_cache_limit = 16M
default_storage_engine = InnoDB
table_open_cache = 512
userstat        = 1
plugin-load     = auth_pam.so
innodb_strict_mode = 0
sql_mode = ''
 
# charset variables
init-connect='SET NAMES latin1'
character_set_client = latin1
character_set_server = latin1
skip-character-set-client-handshake
 
# cg-db-* specific setting
max_connect_errors      = 50000
net_read_timeout        = 300
 
# logging variables
slow_query_log  = 1
long_query_time = 5
slow_query_log_file = /srv/db_trn/log/mysql/ch0pxdbpts05-slow.log
log-warnings    = 2
log_error       = /srv/db_trn/log/mysql/ch0pxdbpts05-error.log
sql-error-log-filename = /srv/db_trn/log/mysql/ch0pxdbpts05_sql_errors.log
log_slow_admin_statements = 1
 
# binlog specific settings
binlog_cache_size = 1M
log_bin         = 1
log-bin         = /srv/db_trn/log/binlog/ch0pxdbpts05-bin
binlog_format   = mixed
expire_logs_days = 10
sync-binlog     = 1
max_binlog_size = 25M
 
# session variables
read_buffer_size = 2M
sort_buffer_size = 2M
read_rnd_buffer_size = 8M
join_buffer_size = 1M
 
# tmp variables
tmp_table_size = 1G
tmpdir          = /srv/db_trn/tmp
 
#  Replication related settings
server-id = 5
#master-host    = 
#master-user    = 
#master-password = 
#master-port    = 3306
read_only
log_slave_updates = 1
#skip-slave-start
log-slow-slave-statements
relay-log=/srv/db_trn/log/mysql/ch0pxdbpts05-relay-bin
 
# MyISAM Specific options
key_buffer_size = 8G
max_heap_table_size = 1G
bulk_insert_buffer_size = 32M
myisam_sort_buffer_size = 64M
myisam_max_sort_file_size = 500M
myisam_repair_threads = 4
myisam_recover_options = backup
 
# INNODB Specific options
innodb_file_per_table   = 1
#innodb_additional_mem_pool_size = 30M
innodb_buffer_pool_size = 32G
innodb_data_file_path   = ibdata1:50M:autoextend
innodb_data_home_dir    = /srv/dbdata_fast/mysql
innodb_thread_concurrency = 20
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size  = 4M
innodb_log_file_size    = 512M
innodb_log_group_home_dir = /srv/db_trn/log/mysql
innodb_max_dirty_pages_pct = 80
innodb_flush_method     = O_DIRECT
innodb_stats_on_metadata = 0
innodb_file_format      = Barracuda
innodb_file_format_check = 1
innodb_print_all_deadlocks = 1
innodb_open_files = 1000
 
[mysqldump]
quick
max_allowed_packet = 16M
 
[mysql]
no-auto-rehash
 
[myisamchk]
key_buffer_size = 512M
sort_buffer_size = 512M
read_buffer = 8M
write_buffer = 8M
 
[mysqlhotcopy]
interactive-timeout
 
[safe_mysqld]
log_error       = /srv/db_trn/log/mysql/ch0pxdbpts05-error.log
socket          = /srv/db_trn/run/mysql.sock
pid-file        = /srv/db_trn/run/chi0dbcore01.pid

Comment by Lori Patrick [ 2018-04-02 ]

sorry about the format of the my.cnf file - Jira sees hash marks meant for comments as a numbered list - I can attach instead if the format is a problem.

I am testing if it crashes on empty table - I can't provide the data itself unfortunately.

Comment by Lori Patrick [ 2018-04-02 ]

It does not crash with no data in the table. I had restored from a backup, so I wouldn't think it is a corruption issue. Seems more memory/buffer related.

Comment by Elena Stepanova [ 2018-04-02 ]

It would be extremely helpful if you could narrow down and/or obfuscate the dataset to the point when you can share it (if you so prefer, it can be done in private).
If you can't do that, please paste the output of SHOW INDEX IN drop_copy_fill_events.

Comment by Lori Patrick [ 2018-04-02 ]

Unfortunately - the data is very sentistive, and I can't share it. Output from show index:

Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment
drop_copy_fill_events 0 PRIMARY 1 id A 14205431 NULL NULL   BTREE    
drop_copy_fill_events 1 user_idx 1 user_id A 33582 NULL NULL   BTREE    
drop_copy_fill_events 1 exchange_ts_idx 1 exchange_ts A 14205431 NULL NULL   BTREE    
drop_copy_fill_events 1 fill_id_str 1 fill_id A 14205431 NULL NULL YES BTREE    
drop_copy_fill_events 1 exchange_order_id_idx 1 exchange_order_id A 14205431 NULL NULL YES BTREE    
drop_copy_fill_events 1 ts_str_idx 1 ts_str A 14205431 NULL NULL YES BTREE    
drop_copy_fill_events 1 tstamp_idx 1 tstamp A 14205431 NULL NULL   BTREE    
drop_copy_fill_events 1 order_id_user_id_idx 1 order_id A 14205431 NULL NULL YES BTREE    
drop_copy_fill_events 1 order_id_user_id_idx 2 user_id A 14205431 NULL NULL   BTREE    

We run optimize on the table weekly.

Comment by Lori Patrick [ 2018-04-02 ]

And explain of the query that crashes with data:

id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY drop_copy_fill_events ALL ts_str_idx NULL NULL NULL 14205472 Using where; Using filesort
1 PRIMARY <subquery2> eq_ref distinct_key distinct_key 94 func 1 Using where
2 MATERIALIZED drop_copy_fill_events ALL ts_str_idx NULL NULL NULL 14205472 Using where; Using filesort
Comment by Elena Stepanova [ 2018-04-02 ]

Thank you, we will try to reproduce it.

Comment by Lori Patrick [ 2018-04-02 ]

Thanks. I am rolling this server back from 10.2.8 - so it is a show stopper for us. I appreciate the assistance.

Comment by Elena Stepanova [ 2018-04-03 ]

mariadb-10.2.8 ASAN

==9883==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6240013b4158 at pc 0x5628600a9a16 bp 0x7fc44624fdc0 sp 0x7fc44624fdb8
READ of size 4 at 0x6240013b4158 thread T35
    #0 0x5628600a9a15 in sf_malloc_usable_size /data/src/10.2-bug/mysys/safemalloc.c:215
    #1 0x562860078ae0 in my_free /data/src/10.2-bug/mysys/my_malloc.c:215
    #2 0x56285f1e2564 in Item_subselect::cleanup() /data/src/10.2-bug/sql/item_subselect.cc:158
    #3 0x56285f1e2b42 in Item_in_subselect::cleanup() /data/src/10.2-bug/sql/item_subselect.cc:195
    #4 0x56285e9b6097 in Item::delete_self() /data/src/10.2-bug/sql/item.h:1813
    #5 0x56285e9a0c16 in Query_arena::free_items() /data/src/10.2-bug/sql/sql_class.cc:3533
    #6 0x56285e99458a in THD::cleanup_after_query() /data/src/10.2-bug/sql/sql_class.cc:2163
    #7 0x56285ea5f546 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.2-bug/sql/sql_parse.cc:7907
    #8 0x56285ea3ab56 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.2-bug/sql/sql_parse.cc:1812
    #9 0x56285ea37c41 in do_command(THD*) /data/src/10.2-bug/sql/sql_parse.cc:1360
    #10 0x56285ed6d3ce in do_handle_one_connection(CONNECT*) /data/src/10.2-bug/sql/sql_connect.cc:1354
    #11 0x56285ed6cde3 in handle_one_connection /data/src/10.2-bug/sql/sql_connect.cc:1260
    #12 0x7fc6f835e493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)
    #13 0x7fc6f64d693e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xe893e)

Comment by Elena Stepanova [ 2018-04-03 ]

--source include/have_innodb.inc
SET GLOBAL innodb_stats_persistent= 1;
DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (
  f1 bigint(20) DEFAULT 0,
  f2 varchar(50) DEFAULT '',
  f3 int(10) DEFAULT 0,
  f4 bigint(20) default 0,
  f5 bigint(20) DEFAULT 0,
  f6 varchar(50) DEFAULT '',
  f7 varchar(64) DEFAULT '',
  f8 varchar(30) DEFAULT '',
  f9 varchar(30) DEFAULT '',
  f10 bigint(20) DEFAULT 0,
  f11 bigint(20) DEFAULT 0,
  f12 bigint(20) DEFAULT 0,
  f13 bigint(20) DEFAULT 0,
  f14 varchar(50) DEFAULT '',
  f15 varchar(100) DEFAULT '',
  f16 varchar(30) DEFAULT '',
  f17 varchar(40) DEFAULT '',
  f18 varchar(30) DEFAULT '',
  f19 varchar(10) DEFAULT '',
  f20 varchar(30) DEFAULT '',
  f21 int(10) DEFAULT 0,
  f22 int(10) DEFAULT 0,
  f23 int(10) DEFAULT 0,
  f24 int(10) DEFAULT 0,
  f25 varchar(20) DEFAULT '',
  f26 varchar(20) DEFAULT '',
  f27 varchar(100) DEFAULT '',
  f28 varchar(55) DEFAULT '',
  f29 varchar(20) DEFAULT '',
  f30 varchar(100) DEFAULT '',
  f31 varchar(30) DEFAULT '',
  f32 varchar(20) DEFAULT '',
  f33 int(10) DEFAULT 0,
  f34 int(10) DEFAULT 0,
  f35 varchar(30) DEFAULT '',
  f36 varchar(30) DEFAULT '',
  f37 varchar(30) DEFAULT '',
  f38 varchar(20) DEFAULT '',
  f39 tinyint(4) DEFAULT 0,
  f40 tinyint(4) DEFAULT 0,
  f41 bigint(20) DEFAULT 0,
  f42 varchar(50) DEFAULT '',
  f43 varchar(50) DEFAULT '',
  f44 varchar(50) DEFAULT '',
  f45 int(10) DEFAULT 0,
  f46 tinyint(1) DEFAULT 0
) ENGINE=InnoDB ROW_FORMAT=DYNAMIC;
 
INSERT INTO t1 () VALUES (),(),(),(),(),(),(),(),(),(),(),(),(),(),(),();
INSERT INTO t1 SELECT * FROM t1;
INSERT INTO t1 SELECT * FROM t1;
INSERT INTO t1 SELECT * FROM t1;
INSERT INTO t1 SELECT * FROM t1;
 
SELECT * FROM t1 WHERE f2 IN (SELECT f2 FROM t1 GROUP BY f2 HAVING COUNT(DISTINCT f3) = 1);
 
# Cleanup
DROP TABLE t1;

10.2 ASAN bc2501453c3ab9

==22526==ERROR: AddressSanitizer: heap-use-after-free on address 0x6110000515d8 at pc 0x559259e2a69a bp 0x7f6c0d882d60 sp 0x7f6c0d882d58
READ of size 4 at 0x6110000515d8 thread T27
    #0 0x559259e2a699 in sf_malloc_usable_size /data/src/10.2/mysys/safemalloc.c:215
    #1 0x559259df984e in my_free /data/src/10.2/mysys/my_malloc.c:215
    #2 0x559258f78052 in Item_subselect::cleanup() /data/src/10.2/sql/item_subselect.cc:158
    #3 0x559258f78630 in Item_in_subselect::cleanup() /data/src/10.2/sql/item_subselect.cc:195
    #4 0x559258740881 in Item::delete_self() /data/src/10.2/sql/item.h:1834
    #5 0x55925872b35e in Query_arena::free_items() /data/src/10.2/sql/sql_class.cc:3563
    #6 0x55925871e962 in THD::cleanup_after_query() /data/src/10.2/sql/sql_class.cc:2178
    #7 0x5592587eb1b3 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.2/sql/sql_parse.cc:7935
    #8 0x5592587c60eb in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.2/sql/sql_parse.cc:1815
    #9 0x5592587c318f in do_command(THD*) /data/src/10.2/sql/sql_parse.cc:1369
    #10 0x559258afea35 in do_handle_one_connection(CONNECT*) /data/src/10.2/sql/sql_connect.cc:1335
    #11 0x559258afe44a in handle_one_connection /data/src/10.2/sql/sql_connect.cc:1241
    #12 0x55925950a129 in pfs_spawn_thread /data/src/10.2/storage/perfschema/pfs.cc:1862
    #13 0x7f6c1e28f493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)
    #14 0x7f6c1c67593e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xe893e)
 
0x6110000515d8 is located 88 bytes inside of 212-byte region [0x611000051580,0x611000051654)
freed by thread T27 here:
    #0 0x7f6c1e4f9527 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x54527)
    #1 0x559259e2af97 in free_memory /data/src/10.2/mysys/safemalloc.c:279
    #2 0x559259e2a59d in sf_free /data/src/10.2/mysys/safemalloc.c:197
    #3 0x559259df9884 in my_free /data/src/10.2/mysys/my_malloc.c:217
    #4 0x559258f78052 in Item_subselect::cleanup() /data/src/10.2/sql/item_subselect.cc:158
    #5 0x559258f78630 in Item_in_subselect::cleanup() /data/src/10.2/sql/item_subselect.cc:195
    #6 0x5592588b5111 in st_join_table::cleanup() /data/src/10.2/sql/sql_select.cc:11819
    #7 0x5592588b788d in JOIN::cleanup(bool) /data/src/10.2/sql/sql_select.cc:12200
    #8 0x5592588b6f6d in JOIN::join_free() /data/src/10.2/sql/sql_select.cc:12106
    #9 0x5592588de17f in do_select /data/src/10.2/sql/sql_select.cc:18281
    #10 0x55925887e15c in JOIN::exec_inner() /data/src/10.2/sql/sql_select.cc:3566
    #11 0x55925887bdf1 in JOIN::exec() /data/src/10.2/sql/sql_select.cc:3361
    #12 0x55925887f1d9 in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /data/src/10.2/sql/sql_select.cc:3761
    #13 0x55925885ed14 in handle_select(THD*, LEX*, select_result*, unsigned long) /data/src/10.2/sql/sql_select.cc:376
    #14 0x5592587e28eb in execute_sqlcom_select /data/src/10.2/sql/sql_parse.cc:6468
    #15 0x5592587cf73a in mysql_execute_command(THD*) /data/src/10.2/sql/sql_parse.cc:3479
    #16 0x5592587eaeda in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/src/10.2/sql/sql_parse.cc:7914
    #17 0x5592587c60eb in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/src/10.2/sql/sql_parse.cc:1815
    #18 0x5592587c318f in do_command(THD*) /data/src/10.2/sql/sql_parse.cc:1369
    #19 0x559258afea35 in do_handle_one_connection(CONNECT*) /data/src/10.2/sql/sql_connect.cc:1335
    #20 0x559258afe44a in handle_one_connection /data/src/10.2/sql/sql_connect.cc:1241
    #21 0x55925950a129 in pfs_spawn_thread /data/src/10.2/storage/perfschema/pfs.cc:1862
    #22 0x7f6c1e28f493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)
 
previously allocated by thread T27 here:
    #0 0x7f6c1e4f973f in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x5473f)
    #1 0x559259e29d0d in sf_malloc /data/src/10.2/mysys/safemalloc.c:118
    #2 0x559259df8fbc in my_malloc /data/src/10.2/mysys/my_malloc.c:101
    #3 0x559258d9bbb8 in read_buffpek_from_file /data/src/10.2/sql/filesort.cc:510
    #4 0x559258d9a178 in filesort(THD*, TABLE*, Filesort*, Filesort_tracker*, JOIN*, unsigned long long) /data/src/10.2/sql/filesort.cc:304
    #5 0x5592588f55c3 in create_sort_index(THD*, JOIN*, st_join_table*, Filesort*) /data/src/10.2/sql/sql_select.cc:21817
    #6 0x5592588e686c in st_join_table::sort_table() /data/src/10.2/sql/sql_select.cc:19652
    #7 0x5592588e5e89 in join_init_read_record(st_join_table*) /data/src/10.2/sql/sql_select.cc:19593
    #8 0x5592588df6a5 in sub_select(JOIN*, st_join_table*, bool) /data/src/10.2/sql/sql_select.cc:18688
    #9 0x5592588ddbb0 in do_select /data/src/10.2/sql/sql_select.cc:18235
    #10 0x55925887e15c in JOIN::exec_inner() /data/src/10.2/sql/sql_select.cc:3566
    #11 0x55925887bdf1 in JOIN::exec() /data/src/10.2/sql/sql_select.cc:3361
    #12 0x559258bcb939 in join_tab_execution_startup(st_join_table*) /data/src/10.2/sql/opt_subselect.cc:5309
    #13 0x5592588df596 in sub_select(JOIN*, st_join_table*, bool) /data/src/10.2/sql/sql_select.cc:18679
 
Thread T27 created by T0 here:
    #0 0x7f6c1e4c8bba in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x23bba)
    #1 0x55925950a6f1 in spawn_thread_v1 /data/src/10.2/storage/perfschema/pfs.cc:1912
    #2 0x5592585c0cde in inline_mysql_thread_create /data/src/10.2/include/mysql/psi/mysql_thread.h:1239
    #3 0x5592585d5b15 in create_thread_to_handle_connection(CONNECT*) /data/src/10.2/sql/mysqld.cc:6436
    #4 0x5592585d621a in create_new_thread /data/src/10.2/sql/mysqld.cc:6506
    #5 0x5592585d722b in handle_connections_sockets() /data/src/10.2/sql/mysqld.cc:6781
    #6 0x5592585d506a in mysqld_main(int, char**) /data/src/10.2/sql/mysqld.cc:6055
    #7 0x5592585bf07f in main /data/src/10.2/sql/main.cc:25
    #8 0x7f6c1c5ad2b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
 
SUMMARY: AddressSanitizer: heap-use-after-free /data/src/10.2/mysys/safemalloc.c:215 sf_malloc_usable_size
Shadow bytes around the buggy address:
  0x0c2280002260: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2280002270: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2280002280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2280002290: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c22800022a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c22800022b0: fd fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd
  0x0c22800022c0: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
  0x0c22800022d0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c22800022e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c22800022f0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
  0x0c2280002300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Contiguous container OOB:fc
  ASan internal:           fe
==22526==ABORTING
----------SERVER LOG END-------------

Not reproducible on 10.1.

Comment by Lori Patrick [ 2018-04-20 ]

Hi,

Just wondering if there is any update or schedule for a potential fix for this? If it is going to be a while, I will move forward with 10.1. There are features in 10.2 we would really like, so hoping it won't be too long for a fix or work around.

Thanks!
Lori

Comment by Sergei Golubchik [ 2018-04-21 ]

yes, the plan is to fix it in the next 10.2 release.
the release schedule is at the https://jira.mariadb.org and at the moment it says that 10.2.15 is planned for 2018-05-11

Comment by Lori Patrick [ 2018-04-23 ]

Great - thanks for the update!!

Generated at Thu Feb 08 08:23:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.