[MDEV-15215] main.partition_explicit_prune fails in bulidbot with assertion failures and server crashes Created: 2018-02-05  Updated: 2018-03-06  Resolved: 2018-02-13

Status: Closed
Project: MariaDB Server
Component/s: Partitioning, Tests
Affects Version/s: 10.3
Fix Version/s: 10.3.5

Type: Bug Priority: Blocker
Reporter: Elena Stepanova Assignee: Alexey Botchkov
Resolution: Fixed Votes: 0
Labels: None


 Description   

First failed on bb-10.3-hf:
http://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.3-hf

main.partition_explicit_prune 'innodb'   w4 [ fail ]
        Test ended at 2018-01-31 20:09:52
 
CURRENT_TEST: main.partition_explicit_prune
mysqltest: At line 442: query 'UPDATE t1 PARTITION(`p100-99999`) SET a = -2, b = concat(b, ', Updated from a = 100')
WHERE a = 100' failed with wrong errno 2013: 'Lost connection to MySQL server during query', instead of 1748...
 
The result from queries just before the failure was:
< snip >
 
FLUSH STATUS;
UPDATE t1 PARTITION(subp0) SET b = concat(b, ', Updated2') WHERE a = 100;
SELECT * FROM INFORMATION_SCHEMA.SESSION_STATUS
WHERE VARIABLE_NAME LIKE 'HANDLER_%' AND VARIABLE_VALUE > 0;
VARIABLE_NAME	VARIABLE_VALUE
HANDLER_COMMIT	1
HANDLER_TMP_WRITE	24
# Nothing, since impossible PARTITION+WHERE clause.
FLUSH STATUS;
UPDATE t1 PARTITION(subp0) SET a = -2, b = concat(b, ', Updated from a = 100')
WHERE a = 100;
SELECT * FROM INFORMATION_SCHEMA.SESSION_STATUS
WHERE VARIABLE_NAME LIKE 'HANDLER_%' AND VARIABLE_VALUE > 0;
VARIABLE_NAME	VARIABLE_VALUE
HANDLER_COMMIT	1
HANDLER_TMP_WRITE	24
# Nothing, since impossible PARTITION+WHERE clause.
FLUSH STATUS;
UPDATE t1 PARTITION(`p100-99999`) SET a = -2, b = concat(b, ', Updated from a = 100')
WHERE a = 100;
 
More results from queries before failure can be found in /run/shm/var/4/log/partition_explicit_prune.log

Server [mysqld.1 - pid: 7198, winpid: 7198, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
180131 20:09:42 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.3.5-MariaDB-10.3.5+maria~trusty-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=8
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63125 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f33180009a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f33680f2e40 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f337362fbce]
/usr/sbin/mysqld(handle_fatal_signal+0x357)[0x7f33730b1787]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f3371294330]
/usr/sbin/mysqld(+0x925600)[0x7f337328b600]
/usr/sbin/mysqld(+0x9254ed)[0x7f337328b4ed]
/usr/sbin/mysqld(+0xc8d9f2)[0x7f33735f39f2]
/usr/sbin/mysqld(+0x8596d7)[0x7f33731bf6d7]
/usr/sbin/mysqld(+0x859a47)[0x7f33731bfa47]
/usr/sbin/mysqld(_ZN10SQL_SELECT17test_quick_selectEP3THD6BitmapILj64EEyybbb+0x9fb)[0x7f33731ccedb]
/usr/sbin/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_ordery15enum_duplicatesbPySB_+0x878)[0x7f3372f7ba18]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x2c67)[0x7f3372ee14d7]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x2bd)[0x7f3372ee7aed]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x21a2)[0x7f3372eeaab2]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x137)[0x7f3372eeb397]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1aa)[0x7f3372fb87ea]
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x7f3372fb890d]
/usr/sbin/mysqld(+0x90ba1d)[0x7f3373271a1d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f337128c184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f33709af03d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f33180115b0): UPDATE t1 PARTITION(`p100-99999`) SET a = -2, b = concat(b, ', Updated from a = 100') WHERE a = 100
Connection ID (thread ID): 159
Status: NOT_KILLED

And after the merge numerous times in the main tree:

mysqld: /home/buildbot/buildbot/build/mariadb-10.3.5/storage/innobase/handler/ha_innodb.cc:3024: void ha_innobase::update_thd(THD*): Assertion `m_prebuilt->table->n_ref_count > 0' failed.
180201 22:18:48 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.3.5-MariaDB-debug-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=10
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61968 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0xa8309cd0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xb047b280 thread_stack 0x49000
/mnt/buildbot/build/mariadb-10.3.5/sql/mysqld(my_print_stacktrace+0x3b)[0x8e36403]
/mnt/buildbot/build/mariadb-10.3.5/sql/mysqld(handle_fatal_signal+0x404)[0x8677138]
[0xb77a7400]
[0xb77a7424]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb72831ef]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb7286835]
/lib/i386-linux-gnu/libc.so.6(+0x27095)[0xb727c095]
/lib/i386-linux-gnu/libc.so.6(+0x27147)[0xb727c147]
/mnt/buildbot/build/mariadb-10.3.5/sql/mysqld[0x89c5708]
/mnt/buildbot/build/mariadb-10.3.5/sql/mysqld[0x89b5005]
/mnt/buildbot/build/mariadb-10.3.5/sql/mysqld[0x89b52f8]
/mnt/buildbot/build/mariadb-10.3.5/sql/mysqld[0x8dfe9b1]
mysys/stacktrace.c:269(my_print_stacktrace)[0x8df6fbd]
sql/signal_handler.cc:168(handle_fatal_signal)[0x87da089]
handler/ha_innodb.cc:3026(ha_innobase::update_thd(THD*))[0x87d28c4]
sql/opt_range.cc:10371(check_quick_select)[0x87ca66c]
sql/opt_range.h:1648(SQL_SELECT::check_quick(THD*, bool, unsigned long long))[0x848ff1c]
sql/sql_update.cc:449(mysql_update(THD*, TABLE_LIST*, List<Item>&, List<Item>&, Item*, unsigned int, st_order*, unsigned long long, enum_duplicates, bool, unsigned long long*, unsigned long long*))[0x848890a]
sql/sql_parse.cc:4557(mysql_execute_command(THD*))[0x83a01f9]
sql/sql_parse.cc:7976(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x83ab702]
sql/sql_parse.cc:1827(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x839852e]
sql/sql_parse.cc:1370(do_command(THD*))[0x8396fcb]
sql/sql_connect.cc:1402(do_handle_one_connection(CONNECT*))[0x84efc7c]
sql/sql_connect.cc:1309(handle_one_connection)[0x84efa0a]
perfschema/pfs.cc:1864(pfs_spawn_thread)[0x88a79ec]
/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0xb752fd4c]
/lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xb733face]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0xa8319b48): UPDATE t1 PARTITION(`p100-99999`) SET a = -2, b = concat(b, ', Updated from a = 100') WHERE a = 100
Connection ID (thread ID): 179
Status: NOT_KILLED

https://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/11376/steps/mtr_ps/logs/stdio

2018-01-29 04:40:23 0xacabcb40  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.3.5/storage/innobase/handler/ha_innodb.cc line 14067
InnoDB: Failing assertion: m_prebuilt->table->stat_initialized
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/
InnoDB: about forcing recovery.
180129  4:40:23 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.3.5-MariaDB-debug-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=8
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61966 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0xa83006f8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xacabc280 thread_stack 0x49000
/mnt/buildbot/build/mariadb-10.3.5/sql/mysqld(my_print_stacktrace+0x3b)[0x8e35d0b]
/mnt/buildbot/build/mariadb-10.3.5/sql/mysqld(handle_fatal_signal+0x404)[0x8675fe0]
[0xb7717400]
[0xb7717424]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb71f31ef]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb71f6835]
/mnt/buildbot/build/mariadb-10.3.5/sql/mysqld[0x8bd91f8]
/mnt/buildbot/build/mariadb-10.3.5/sql/mysqld[0x89b482b]
/mnt/buildbot/build/mariadb-10.3.5/sql/mysqld[0x89b48c7]
/mnt/buildbot/build/mariadb-10.3.5/sql/mysqld[0x8dfe299]
/mnt/buildbot/build/mariadb-10.3.5/sql/mysqld[0x8df68ab]
mysys/stacktrace.c:269(my_print_stacktrace)[0x87d926d]
sql/signal_handler.cc:168(handle_fatal_signal)[0x87d1aa8]
handler/ha_innodb.cc:14070(ha_innobase::scan_time())[0x87c9850]
sql/opt_range.cc:10371(check_quick_select)[0x848f68c]
sql/sql_update.cc:449(mysql_update(THD*, TABLE_LIST*, List<Item>&, List<Item>&, Item*, unsigned int, st_order*, unsigned long long, enum_duplicates, bool, unsigned long long*, unsigned long long*))[0x84880c3]
sql/sql_parse.cc:4561(mysql_execute_command(THD*))[0x839fcfb]
sql/sql_prepare.cc:4736(Prepared_statement::execute(String*, bool))[0x83c7893]
sql/sql_prepare.cc:4165(Prepared_statement::execute_loop(String*, bool, unsigned char*, unsigned char*))[0x83c5ff9]
sql/sql_prepare.cc:3165(mysql_stmt_execute_common)[0x83c3c74]
sql/sql_prepare.cc:3064(mysqld_stmt_execute(THD*, char*, unsigned int))[0x83c3849]
sql/sql_parse.cc:1769(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x8397c7d]
sql/sql_parse.cc:1370(do_command(THD*))[0x8396ab3]
sql/sql_connect.cc:1401(do_handle_one_connection(CONNECT*))[0x84ef3a3]
sql/sql_connect.cc:1308(handle_one_connection)[0x84ef131]
perfschema/pfs.cc:1864(pfs_spawn_thread)[0x88a68e0]
/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0xb749fd4c]
/lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xb72aface]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0xa4ddb6e0): UPDATE t1 PARTITION(`p100-99999`) SET a = -2, b = concat(b, ', Updated from a = 100') WHERE a = 100
Connection ID (thread ID): 179
Status: NOT_KILLED



 Comments   
Comment by Alexey Botchkov [ 2018-02-11 ]

patch proposal
http://lists.askmonty.org/pipermail/commits/2018-February/011985.html

Comment by Sergei Golubchik [ 2018-02-13 ]

ok to push!

Comment by Alexey Botchkov [ 2018-02-13 ]

http://lists.askmonty.org/pipermail/commits/2018-February/011996.html

Generated at Thu Feb 08 08:19:33 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.