[MDEV-10356] rpl.rpl_parallel_temptable failure due to incorrect commit optimization of temptables - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Affects Version/s: 10.0(EOL), 10.1(EOL), 10.2(EOL), 10.3(EOL), 10.4(EOL), 10.5, 10.6, 10.7(EOL)
Fix Version/s: 10.4.32, 10.5.23, 10.6.16, 10.10.7, 10.11.6, 11.0.4, 11.1.3
Component/s: Replication, Tests
Labels:

Description

However, even non-transactional event groups have an optimisation that allow part of the commit of such event groups to run in parallel with following event groups. Thus, it was still possible for part of the commit codepath to run in parallel with another operation on the temporary table such as a SELECT or DML or even a DROP TEMPORARY TABLE. This can cause access of freed memory and probably other nasty effects.

The problem is the mark_start_commit() in handle_rpl_parallel_thread() just before the COMMIT event is applied, as well as calls of wakeup_subsequent_commit() in the binlog code and in innobase_commit(). These are too early at these points when temporary tables are replicated.

Since temporary table operations are not binlogged in MIXED or ROW mode, this problem affects only statement-based replication.

This is seen as occasional test failures of rpl.rpl_parallel_temptable as reported by Elena and others:

http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/5564/steps/test_6/logs/stdio

rpl.rpl_parallel_temptable 'innodb_plugin,stmt' w1 [ fail ]

        Test ended at 2016-07-09 07:04:06

CURRENT_TEST: rpl.rpl_parallel_temptable

mysqltest: At line 132: failed in 'select master_pos_wait('master-bin.000002', 558, 300, '')': 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:

< snip >

INSERT INTO t2 VALUES (13);

INSERT INTO t3 SELECT a+32 FROM t4;

INSERT INTO t2 VALUES (14);

INSERT INTO t2 VALUES (15);

INSERT INTO t2 VALUES (16);

INSERT INTO t4 SELECT a+64 FROM t3;

INSERT INTO t2 VALUES (17);

INSERT INTO t2 VALUES (18);

INSERT INTO t2 VALUES (19);

INSERT INTO t3 SELECT a+128 FROM t4;

INSERT INTO t2 VALUES (20);

INSERT INTO t1 SELECT a, a MOD 7 FROM t3;

INSERT INTO t1 SELECT a, a MOD 7 FROM t4;

INSERT INTO t1 SELECT a, COUNT(*) FROM t2 GROUP BY a;

FLUSH TABLES;

SET SESSION debug_dbug="+d,crash_dispatch_command_before";

SELECT 1;

Got one of the listed errors

INSERT INTO t1 VALUES (0, 1);

include/start_slave.inc

More results from queries before failure can be found in /mnt/buildbot/build/mariadb-10.1.16/mysql-test/var/1/log/rpl_parallel_temptable.log

Server [mysqld.2 - pid: 26230, winpid: 26230, exit: 256] failed during test run

Server log from this test:

----------SERVER LOG START-----------

2016-07-09  7:03:47 3077719808 [Note] /mnt/buildbot/build/mariadb-10.1.16/sql/mysqld (mysqld 10.1.16-MariaDB-debug) starting as process 26231 ...

2016-07-09  7:03:47 3077719808 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4162)

2016-07-09  7:03:47 3077719808 [Note] Plugin 'file_key_management' is disabled.

2016-07-09  7:03:47 3077719808 [Note] Plugin 'BLACKHOLE' is disabled.

2016-07-09  7:03:47 3077719808 [Note] Plugin 'partition' is disabled.

2016-07-09  7:03:47 3077719808 [Note] Plugin 'ARCHIVE' is disabled.

2016-07-09  7:03:48 3077719808 [Note] InnoDB: Using mutexes to ref count buffer pool pages

2016-07-09  7:03:48 3077719808 [Note] InnoDB:  InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!

2016-07-09  7:03:48 3077719808 [Note] InnoDB:  InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!

2016-07-09  7:03:48 3077719808 [Note] InnoDB: The InnoDB memory heap is disabled

2016-07-09  7:03:48 3077719808 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2016-07-09  7:03:48 3077719808 [Note] InnoDB: Memory barrier is not used

2016-07-09  7:03:48 3077719808 [Note] InnoDB: Compressed tables use zlib 1.2.3.4

2016-07-09  7:03:48 3077719808 [Note] InnoDB: Using Linux native AIO

2016-07-09  7:03:48 3077719808 [Note] InnoDB: Using generic crc32 instructions

2016-07-09  7:03:48 3077719808 [Note] InnoDB: Initializing buffer pool, size = 8.0M

2016-07-09  7:03:48 3077719808 [Note] InnoDB: Completed initialization of buffer pool

2016-07-09  7:03:48 3077719808 [Note] InnoDB: Highest supported file format is Barracuda.

2016-07-09  7:03:49 3077719808 [Note] InnoDB: 128 rollback segment(s) are active.

2016-07-09  7:03:49 3077719808 [Note] InnoDB: Waiting for purge to start

2016-07-09  7:03:49 3077719808 [Note] InnoDB: 5.6.31 started; log sequence number 1629517

2016-07-09  7:03:49 2858416960 [Note] InnoDB: Dumping buffer pool(s) not yet started

2016-07-09  7:03:49 3077719808 [Note] Plugin 'TEST_SQL_DISCOVERY' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'CONNECT' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'SPHINX' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'SEQUENCE' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'FEDERATED' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'XTRADB_READ_VIEW' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'XTRADB_RSEG' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'CHANGED_PAGE_BITMAPS' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'QUERY_CACHE_INFO' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'XTRADB_INTERNAL_HASH_TABLES' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_MUTEXES' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'FEEDBACK' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'METADATA_LOCK_INFO' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_CMP' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_CMP_RESET' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'QUERY_RESPONSE_TIME' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_FT_DELETED' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'LOCALES' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'unix_socket' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'QUERY_RESPONSE_TIME_AUDIT' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'rpl_semi_sync_slave' is disabled.

2016-07-09  7:03:49 3077719808 [Note] Plugin 'rpl_semi_sync_master' is disabled.

2016-07-09  7:03:49 3077719808 [Warning] /mnt/buildbot/build/mariadb-10.1.16/sql/mysqld: unknown option '--loose-skip-plugin-innodb-changed-pages'

2016-07-09  7:03:49 3077719808 [Note] Server socket created on IP: '127.0.0.1'.

2016-07-09  7:03:49 3077719808 [Note] /mnt/buildbot/build/mariadb-10.1.16/sql/mysqld: ready for connections.

Version: '10.1.16-MariaDB-debug'  socket: '/mnt/buildbot/build/mariadb-10.1.16/mysql-test/var/tmp/1/mysqld.2.sock'  port: 16041  Source distribution

2016-07-09  7:03:50 2918969152 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16040', master_log_file='master-bin.000001', master_log_pos='4'.

2016-07-09  7:03:50 2872782656 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16040',replication started in log 'master-bin.000001' at position 4

2016-07-09  7:03:50 2872479552 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 4, relay log './slave-relay-bin.000001' position: 4

2016-07-09  7:03:51 2872479552 [Note] Error reading relay log event: slave SQL thread was killed

2016-07-09  7:03:51 2872782656 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)

2016-07-09  7:03:51 2872782656 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 314

2016-07-09  7:03:51 2918969152 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port='16040', master_log_file='master-bin.000001', master_log_pos='314'. New state master_host='127.0.0.1', master_port='16040', master_log_file='master-bin.000001', master_log_pos='314'.

2016-07-09  7:03:51 2918969152 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos

2016-07-09  7:03:51 2872479552 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16040',replication starts at GTID position ''

2016-07-09  7:03:51 2824539968 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 314, relay log './slave-relay-bin.000001' position: 4; GTID position ''

2016-07-09  7:03:51 2824539968 [Note] Error reading relay log event: slave SQL thread was killed

2016-07-09  7:03:51 2872479552 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)

2016-07-09  7:03:51 2872479552 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 485; GTID position 0-1-1

2016-07-09  7:03:56 2824539968 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16040',replication starts at GTID position '0-1-1'

2016-07-09  7:03:56 2862300992 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 485, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-1'

160709  7:03:56 [ERROR] mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.1.16-MariaDB-debug

key_buffer_size=1048576

read_buffer_size=131072

max_used_connections=2

max_threads=153

thread_count=2

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61813 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0xaa800670

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0xaa9696b8 thread_stack 0x48400

/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(my_print_stacktrace+0x29)[0x8b0fe39]

/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(handle_fatal_signal+0x53c)[0x8438b4c]

[0xe9b400]

/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_Z19close_thread_tablesP3THD+0x48f)[0x81f7c3f]

/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_Z21mysql_execute_commandP3THD+0x45b)[0x825582b]

/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x36c)[0x8261b2c]

/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0xcef)[0x8540d3f]

/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_info+0x31)[0x8542031]

/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x63d)[0x81c41ad]

/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld[0x83ad240]

/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(handle_rpl_parallel_thread+0x99b)[0x83b1e4b]

/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld[0x8619a8a]

/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0x1a3d4c]

/lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xa38ace]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0xb2d12cd3): COMMIT

Connection ID (thread ID): 18

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

Writing a core file

It happens very rarely in buildbot, a handful of occasions over 2 years.
It's possible that the failure is caused by the same problem as was recently fixed in ~~MDEV-10219~~, but then the fix needs to be backported into 10.1 and possibly 10.0.

Attachments

Issue Links

blocks

MDEV-7069 Fix buildbot failures in main server trees

Stalled

is duplicated by

MDEV-25064 rpl.rpl_parallel_temptable failed in bb, ASAN heap-use-after-free in std::__atomic_base<long>::store

Closed

relates to

MDEV-26126 Replication delay increasing infinitely from grandpa master when enable parallel replication and use temporary table?

Open

MDEV-26152 Assertion: prebuilt->template_type == ROW_MYSQL_WHOLE_ROW fails in row0mysql.cc line 1752

Confirmed

MDEV-12459 The information_schema tables for getting temporary tables info is missing, at least for innodb there is no INNODB_TEMP_TABLE_INFO

Closed

MDEV-16539 Server crashes in THD::mark_tmp_table_as_free_for_reuse

Open

MDEV-25064 rpl.rpl_parallel_temptable failed in bb, ASAN heap-use-after-free in std::__atomic_base<long>::store

Closed

(2 relates to)

Activity

Ascending order - Click to sort in descending order

Elena Stepanova created issue - 2016-07-10 23:01

Elena Stepanova made changes - 2016-07-10 23:01

Field	Original Value	New Value
Link		This issue blocks MDEV-7069 [ MDEV-7069 ]

Elena Stepanova added a comment - 2017-10-29 21:36

Not sure if it's the same crash, but looks somewhat similar.
Logs not available.

10.2 b71623123875141f6983c2eee633c7d1
rpl.rpl_parallel_temptable 'innodb,stmt' w1 [ fail ]
Test ended at 2017-10-04 00:23:51

CURRENT_TEST: rpl.rpl_parallel_temptable
mysqltest: At line 132: failed in 'select master_pos_wait('master-bin.000002', 585, 300, '')': 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:
< snip >
INSERT INTO t2 VALUES (19);
connection default;
INSERT INTO t3 SELECT a+128 FROM t4;
connection server_1;
INSERT INTO t2 VALUES (20);
connection default;
INSERT INTO t1 SELECT a, a MOD 7 FROM t3;
INSERT INTO t1 SELECT a, a MOD 7 FROM t4;
connection server_1;
INSERT INTO t1 SELECT a, COUNT(*) FROM t2 GROUP BY a;
FLUSH TABLES;
SET SESSION debug_dbug="+d,crash_dispatch_command_before";
SELECT 1;
Got one of the listed errors
connection default;
connection default;
connection server_1;
INSERT INTO t1 VALUES (0, 1);
connection server_2;
include/start_slave.inc

More results from queries before failure can be found in /mnt/buildbot/build/mariadb-10.2.10/mysql-test/var/1/log/rpl_parallel_temptable.log


Server [mysqld.2 - pid: 13455, winpid: 13455, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
2017-10-04 0:23:44 2858822464 [Note] Deleted Master_info file '/run/shm/var_auto_mRhB/1/mysqld.2/data/master.info'.
2017-10-04 0:23:44 2858822464 [Note] Deleted Master_info file '/run/shm/var_auto_mRhB/1/mysqld.2/data/relay-log.info'.
2017-10-04 0:23:44 2858822464 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info'
2017-10-04 0:23:44 2858822464 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port='16060', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='4'.
2017-10-04 0:23:44 2896620352 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication started in log 'master-bin.000001' at position 4
2017-10-04 0:23:44 2859428672 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 4, relay log './slave-relay-bin.000001' position: 4
2017-10-04 0:23:44 2859428672 [Note] Error reading relay log event: slave SQL thread was killed
2017-10-04 0:23:44 2859428672 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329
2017-10-04 0:23:44 2896620352 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 329
2017-10-04 0:23:45 2858822464 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info'
2017-10-04 0:23:45 2858822464 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='329'. New state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='329'.
2017-10-04 0:23:45 2858822464 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos
2017-10-04 0:23:45 2859428672 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication starts at GTID position ''
2017-10-04 0:23:45 2896620352 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 329, relay log './slave-relay-bin.000001' position: 4; GTID position ''
2017-10-04 0:23:45 2896620352 [Note] Error reading relay log event: slave SQL thread was killed
2017-10-04 0:23:45 2896620352 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 508; GTID position '0-1-1'
2017-10-04 0:23:45 2859428672 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 508; GTID position 0-1-1
2017-10-04 0:23:48 2896620352 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication starts at GTID position '0-1-1'
2017-10-04 0:23:48 2768665408 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 508, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-1'
171004 0:23:48 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.2.10-MariaDB-debug-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=5
max_threads=153
thread_count=19
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61883 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0xb13259f8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xa7465278 thread_stack 0x49000
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(my_print_stacktrace+0x32)[0x8cf0c6a]
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(handle_fatal_signal+0x3c7)[0x85a37db]
[0xb7754400]
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_ZN17I_P_List_iteratorI15TMP_TABLE_SHARE8I_P_ListIS0_20All_tmp_table_shares21I_P_List_null_counter21I_P_List_no_push_backIS0_EEEC1ERKS6_+0xe)[0x8527b0a]
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_ZN3THD33mark_tmp_tables_as_free_for_reuseEv+0x9d)[0x8525c1b]
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_Z19close_thread_tablesP3THD+0x2ae)[0x82cda38]
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_Z21mysql_execute_commandP3THD+0xbc58)[0x83484ce]
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x290)[0x834c8a0]
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0xec6)[0x86be32e]
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_info+0x38)[0x86bd404]
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_ZN9Log_event11apply_eventEP14rpl_group_info+0x5f)[0x82a0dfb]
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld[0x82954ee]
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_Z39apply_event_and_update_pos_for_parallelP9Log_eventP3THDP14rpl_group_info+0xdd)[0x8295b87]
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld[0x84ebfaf]
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(handle_rpl_parallel_thread+0xa60)[0x84eeee8]
/mnt/buildbot/build/mariadb-10.2.10/sql/mysqld[0x87b45f0]
/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0xb74dcd4c]
/lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xb72ecace]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0xb137b033): COMMIT
Connection ID (thread ID): 37
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
----------SERVER LOG END-------------


- saving '/mnt/buildbot/build/mariadb-10.2.10/mysql-test/var/1/log/rpl.rpl_parallel_temptable-innodb,stmt/' to '/mnt/buildbot/build/mariadb-10.2.10/mysql-test/var/log/rpl.rpl_parallel_temptable-innodb,stmt/'

Retrying test rpl.rpl_parallel_temptable, attempt(2/3)...

Elena Stepanova added a comment - 2017-10-29 21:36 Not sure if it's the same crash, but looks somewhat similar. Logs not available. 10.2 b71623123875141f6983c2eee633c7d1 rpl.rpl_parallel_temptable 'innodb,stmt' w1 [ fail ] Test ended at 2017-10-04 00:23:51 CURRENT_TEST: rpl.rpl_parallel_temptable mysqltest: At line 132: failed in 'select master_pos_wait('master-bin.000002', 585, 300, '')': 2013: Lost connection to MySQL server during query The result from queries just before the failure was: < snip > INSERT INTO t2 VALUES (19); connection default; INSERT INTO t3 SELECT a+128 FROM t4; connection server_1; INSERT INTO t2 VALUES (20); connection default; INSERT INTO t1 SELECT a, a MOD 7 FROM t3; INSERT INTO t1 SELECT a, a MOD 7 FROM t4; connection server_1; INSERT INTO t1 SELECT a, COUNT(*) FROM t2 GROUP BY a; FLUSH TABLES; SET SESSION debug_dbug="+d,crash_dispatch_command_before"; SELECT 1; Got one of the listed errors connection default; connection default; connection server_1; INSERT INTO t1 VALUES (0, 1); connection server_2; include/start_slave.inc More results from queries before failure can be found in /mnt/buildbot/build/mariadb-10.2.10/mysql-test/var/1/log/rpl_parallel_temptable.log Server [mysqld.2 - pid: 13455, winpid: 13455, exit: 256] failed during test run Server log from this test: ----------SERVER LOG START----------- 2017-10-04 0:23:44 2858822464 [Note] Deleted Master_info file '/run/shm/var_auto_mRhB/1/mysqld.2/data/master.info'. 2017-10-04 0:23:44 2858822464 [Note] Deleted Master_info file '/run/shm/var_auto_mRhB/1/mysqld.2/data/relay-log.info'. 2017-10-04 0:23:44 2858822464 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2017-10-04 0:23:44 2858822464 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port='16060', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='4'. 2017-10-04 0:23:44 2896620352 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication started in log 'master-bin.000001' at position 4 2017-10-04 0:23:44 2859428672 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 4, relay log './slave-relay-bin.000001' position: 4 2017-10-04 0:23:44 2859428672 [Note] Error reading relay log event: slave SQL thread was killed 2017-10-04 0:23:44 2859428672 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329 2017-10-04 0:23:44 2896620352 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 329 2017-10-04 0:23:45 2858822464 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2017-10-04 0:23:45 2858822464 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='329'. New state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='329'. 2017-10-04 0:23:45 2858822464 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos 2017-10-04 0:23:45 2859428672 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication starts at GTID position '' 2017-10-04 0:23:45 2896620352 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 329, relay log './slave-relay-bin.000001' position: 4; GTID position '' 2017-10-04 0:23:45 2896620352 [Note] Error reading relay log event: slave SQL thread was killed 2017-10-04 0:23:45 2896620352 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 508; GTID position '0-1-1' 2017-10-04 0:23:45 2859428672 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 508; GTID position 0-1-1 2017-10-04 0:23:48 2896620352 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication starts at GTID position '0-1-1' 2017-10-04 0:23:48 2768665408 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 508, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-1' 171004 0:23:48 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.2.10-MariaDB-debug-log key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=5 max_threads=153 thread_count=19 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61883 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0xb13259f8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0xa7465278 thread_stack 0x49000 /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(my_print_stacktrace+0x32)[0x8cf0c6a] /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(handle_fatal_signal+0x3c7)[0x85a37db] [0xb7754400] /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_ZN17I_P_List_iteratorI15TMP_TABLE_SHARE8I_P_ListIS0_20All_tmp_table_shares21I_P_List_null_counter21I_P_List_no_push_backIS0_EEEC1ERKS6_+0xe)[0x8527b0a] /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_ZN3THD33mark_tmp_tables_as_free_for_reuseEv+0x9d)[0x8525c1b] /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_Z19close_thread_tablesP3THD+0x2ae)[0x82cda38] /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_Z21mysql_execute_commandP3THD+0xbc58)[0x83484ce] /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x290)[0x834c8a0] /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0xec6)[0x86be32e] /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_info+0x38)[0x86bd404] /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_ZN9Log_event11apply_eventEP14rpl_group_info+0x5f)[0x82a0dfb] /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld[0x82954ee] /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(_Z39apply_event_and_update_pos_for_parallelP9Log_eventP3THDP14rpl_group_info+0xdd)[0x8295b87] /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld[0x84ebfaf] /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld(handle_rpl_parallel_thread+0xa60)[0x84eeee8] /mnt/buildbot/build/mariadb-10.2.10/sql/mysqld[0x87b45f0] /lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0xb74dcd4c] /lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xb72ecace] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0xb137b033): COMMIT Connection ID (thread ID): 37 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file ----------SERVER LOG END------------- - saving '/mnt/buildbot/build/mariadb-10.2.10/mysql-test/var/1/log/rpl.rpl_parallel_temptable-innodb,stmt/' to '/mnt/buildbot/build/mariadb-10.2.10/mysql-test/var/log/rpl.rpl_parallel_temptable-innodb,stmt/' Retrying test rpl.rpl_parallel_temptable, attempt(2/3)...

Elena Stepanova made changes - 2017-10-29 21:36

Fix Version/s		10.2 [ 14601 ]
Affects Version/s		10.2 [ 14601 ]

Alice Sherepa added a comment - 2020-01-09 14:17

rpl.rpl_parallel_temptable 'innodb,stmt' w3 [ fail ]

        Test ended at 2020-01-08 18:00:54

CURRENT_TEST: rpl.rpl_parallel_temptable

mysqltest: In included file "./include/wait_for_slave_param.inc":

included from ./include/wait_for_slave_io_to_start.inc at line 40:

included from ./include/wait_for_slave_to_start.inc at line 26:

included from ./include/start_slave.inc at line 35:

included from /home/alice/git/10.2/mysql-test/suite/rpl/t/rpl_parallel_temptable.test at line 132:

At line 119: query 'let $_show_slave_status_value= query_get_value("SHOW SLAVE STATUS", $slave_param, 1)' failed: 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:

< snip >

INSERT INTO t2 VALUES (19);

connection default;

INSERT INTO t3 SELECT a+128 FROM t4;

connection server_1;

INSERT INTO t2 VALUES (20);

connection default;

INSERT INTO t1 SELECT a, a MOD 7 FROM t3;

INSERT INTO t1 SELECT a, a MOD 7 FROM t4;

connection server_1;

INSERT INTO t1 SELECT a, COUNT(*) FROM t2 GROUP BY a;

FLUSH TABLES;

SET SESSION debug_dbug="+d,crash_dispatch_command_before";

SELECT 1;

Got one of the listed errors

connection default;

connection default;

connection server_1;

INSERT INTO t1 VALUES (0, 1);

connection server_2;

include/start_slave.inc

More results from queries before failure can be found in /home/alice/git/10.2/mysql-test/var/3/log/rpl_parallel_temptable.log

Server [mysqld.2 - pid: 916, winpid: 916, exit: 256] failed during test run

Server log from this test:

----------SERVER LOG START-----------

2020-01-08 18:00:49 139847131875520 [Note] /home/alice/git/10.2/sql/mysqld (mysqld 10.2.31-MariaDB-debug-log) starting as process 917 ...

2020-01-08 18:00:49 139847131875520 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4186)

2020-01-08 18:00:49 139847131875520 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'partition' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'SEQUENCE' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Uses event mutexes

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Compressed tables use zlib 1.2.8

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Using Linux native AIO

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Number of pools: 1

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Using SSE2 crc32 instructions

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Completed initialization of buffer pool

2020-01-08 18:00:49 139846844655360 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Highest supported file format is Barracuda.

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: 128 out of 128 rollback segments are active.

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Creating shared tablespace for temporary tables

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: File './ibtmp1' size is now 12 MB.

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Waiting for purge to start

2020-01-08 18:00:49 139847131875520 [Note] InnoDB: 5.7.28 started; log sequence number 1628840

2020-01-08 18:00:49 139846534289152 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var_auto_Cizt/3/mysqld.2/data/ib_buffer_pool

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_MUTEXES' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_CMP' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_FT_DELETED' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_CMP_RESET' is disabled.

2020-01-08 18:00:49 139846534289152 [Note] InnoDB: Buffer pool(s) load completed at 200108 18:00:49

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'FEEDBACK' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'user_variables' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.

2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.

2020-01-08 18:00:49 139847131875520 [Warning] /home/alice/git/10.2/sql/mysqld: unknown option '--loose-pam-debug'

2020-01-08 18:00:49 139847131875520 [Note] Server socket created on IP: '127.0.0.1'.

2020-01-08 18:00:49 139847131875520 [Note] Reading of all Master_info entries succeeded

2020-01-08 18:00:49 139847131875520 [Note] Added new Master_info '' to hash table

2020-01-08 18:00:49 139847131875520 [Note] /home/alice/git/10.2/sql/mysqld: ready for connections.

Version: '10.2.31-MariaDB-debug-log'  socket: '/home/alice/git/10.2/mysql-test/var/tmp/3/mysqld.2.sock'  port: 16021  Source distribution

2020-01-08 18:00:50 139846956963584 [Note] Deleted Master_info file '/home/alice/git/10.2/mysql-test/var/3/mysqld.2/data/master.info'.

2020-01-08 18:00:50 139846956963584 [Note] Deleted Master_info file '/home/alice/git/10.2/mysql-test/var/3/mysqld.2/data/relay-log.info'.

2020-01-08 18:00:50 139846956963584 [Note] Master connection name: ''  Master_info_file: 'master.info'  Relay_info_file: 'relay-log.info'

2020-01-08 18:00:50 139846956963584 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16020', master_log_file='master-bin.000001', master_log_pos='4'.

2020-01-08 18:00:50 139846956054272 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 4, relay log './slave-relay-bin.000001' position: 4

2020-01-08 18:00:50 139846956357376 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16020',replication started in log 'master-bin.000001' at position 4

2020-01-08 18:00:51 139846956054272 [Note] Error reading relay log event: slave SQL thread was killed

2020-01-08 18:00:51 139846956054272 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329

2020-01-08 18:00:51 139846956357376 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 329

2020-01-08 18:00:51 139846956963584 [Note] Master connection name: ''  Master_info_file: 'master.info'  Relay_info_file: 'relay-log.info'

2020-01-08 18:00:51 139846956963584 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port='16020', master_log_file='master-bin.000001', master_log_pos='329'. New state master_host='127.0.0.1', master_port='16020', master_log_file='master-bin.000001', master_log_pos='329'.

2020-01-08 18:00:51 139846956963584 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos

2020-01-08 18:00:51 139846956357376 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16020',replication starts at GTID position ''

2020-01-08 18:00:51 139846956054272 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 329, relay log './slave-relay-bin.000001' position: 4; GTID position ''

2020-01-08 18:00:51 139846956054272 [Note] Error reading relay log event: slave SQL thread was killed

2020-01-08 18:00:51 139846956054272 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 508; GTID position '0-1-1'

2020-01-08 18:00:51 139846956357376 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 508; GTID position 0-1-1

2020-01-08 18:00:52 139846955751168 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 508, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-1'

2020-01-08 18:00:52 139846956054272 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16020',replication starts at GTID position '0-1-1'

200108 18:00:52 [ERROR] mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.2.31-MariaDB-debug-log

key_buffer_size=1048576

read_buffer_size=131072

max_used_connections=2

max_threads=153

thread_count=16

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63104 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f303c000af0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7f30a810ec78 thread_stack 0x49000

/home/alice/git/10.2/sql/mysqld(my_print_stacktrace+0x38)[0x55a2e23b0338]

/home/alice/git/10.2/sql/mysqld(handle_fatal_signal+0x3a3)[0x55a2e1c53064]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f30b1131390]

sql/sql_plist.h:181(I_P_List_iterator<TMP_TABLE_SHARE, I_P_List<TMP_TABLE_SHARE, All_tmp_table_shares, I_P_List_null_counter, I_P_List_no_push_back<TMP_TABLE_SHARE> > >::I_P_List_iterator(I_P_List<TMP_TABLE_SHARE, All_tmp_table_shares, I_P_List_null_counter, I_P_List_no_push_back<TMP_TABLE_SHARE> > const&))[0x55a2e1bd2977]

sql/temporary_tables.cc:741(THD::mark_tmp_tables_as_free_for_reuse())[0x55a2e1bd09bc]

sql/sql_base.cc:805(close_thread_tables(THD*))[0x55a2e195c7e6]

sql/sql_parse.cc:6053(mysql_execute_command(THD*))[0x55a2e19dae33]

sql/sql_parse.cc:7740(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x55a2e19df91d]

sql/log_event.cc:5350(Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int))[0x55a2e1d7b2ea]

sql/log_event.cc:5029(Query_log_event::do_apply_event(rpl_group_info*))[0x55a2e1d7a305]

sql/log_event.h:1452(Log_event::apply_event(rpl_group_info*))[0x55a2e192e3b9]

sql/slave.cc:3567(apply_event_and_update_pos_apply(Log_event*, THD*, rpl_group_info*, int))[0x55a2e19230c7]

sql/slave.cc:3743(apply_event_and_update_pos_for_parallel(Log_event*, THD*, rpl_group_info*))[0x55a2e19236ca]

sql/rpl_parallel.cc:50(rpt_handle_event(rpl_parallel_thread::queued_event*, rpl_parallel_thread*))[0x55a2e1b93964]

sql/rpl_parallel.cc:1316(handle_rpl_parallel_thread)[0x55a2e1b969ef]

perfschema/pfs.cc:1864(pfs_spawn_thread)[0x55a2e234affa]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f30b11276ba]

x86_64/clone.S:111(clone)[0x7f30b05bc41d]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x7f303802748b): COMMIT

Connection ID (thread ID): 27

Status: NOT_KILLED

#3  <signal handler called>

#4  0x000055a2e1bd2977 in I_P_List_iterator<TMP_TABLE_SHARE, I_P_List<TMP_TABLE_SHARE, All_tmp_table_shares, I_P_List_null_counter, I_P_List_no_push_back<TMP_TABLE_SHARE> > >::I_P_List_iterator (this=0x7f30a810d940, a=...) at /10.2/sql/sql_plist.h:181

#5  0x000055a2e1bd09bc in THD::mark_tmp_tables_as_free_for_reuse (this=0x7f303c000af0) at /10.2/sql/temporary_tables.cc:740

#6  0x000055a2e195c7e6 in close_thread_tables (thd=0x7f303c000af0) at /10.2/sql/sql_base.cc:803

#7  0x000055a2e19dae33 in mysql_execute_command (thd=0x7f303c000af0) at /10.2/sql/sql_parse.cc:6051

#8  0x000055a2e19df91d in mysql_parse (thd=0x7f303c000af0, rawbuf=0x7f303802748b "COMMIT", length=6, parser_state=0x7f30a810e510, is_com_multi=false, is_next_command=false) at /10.2/sql/sql_parse.cc:7740

#9  0x000055a2e1d7b2ea in Query_log_event::do_apply_event (this=0x7f30380272e0, rgi=0x7f3038022360, query_arg=0x7f303802748b "COMMIT", q_len_arg=6) at /10.2/sql/log_event.cc:5348

#10 0x000055a2e1d7a305 in Query_log_event::do_apply_event (this=0x7f30380272e0, rgi=0x7f3038022360) at /10.2/sql/log_event.cc:5028

#11 0x000055a2e192e3b9 in Log_event::apply_event (this=0x7f30380272e0, rgi=0x7f3038022360) at /10.2/sql/log_event.h:1452

#12 0x000055a2e19230c7 in apply_event_and_update_pos_apply (ev=0x7f30380272e0, thd=0x7f303c000af0, rgi=0x7f3038022360, reason=0) at /10.2/sql/slave.cc:3567

#13 0x000055a2e19236ca in apply_event_and_update_pos_for_parallel (ev=0x7f30380272e0, thd=0x7f303c000af0, rgi=0x7f3038022360) at /10.2/sql/slave.cc:3742

#14 0x000055a2e1b93964 in rpt_handle_event (qev=0x7f3038021f00, rpt=0x7f3038009640) at /10.2/sql/rpl_parallel.cc:50

#15 0x000055a2e1b969ef in handle_rpl_parallel_thread (arg=0x7f3038009640) at /10.2/sql/rpl_parallel.cc:1316

#16 0x000055a2e234affa in pfs_spawn_thread (arg=0x7f303800a150) at /10.2/storage/perfschema/pfs.cc:1862

#17 0x00007f30b11276ba in start_thread (arg=0x7f30a810f700) at pthread_create.c:333

#18 0x00007f30b05bc41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Alice Sherepa added a comment - 2020-01-09 14:17 rpl.rpl_parallel_temptable 'innodb,stmt' w3 [ fail ] Test ended at 2020-01-08 18:00:54 CURRENT_TEST: rpl.rpl_parallel_temptable mysqltest: In included file "./include/wait_for_slave_param.inc": included from ./include/wait_for_slave_io_to_start.inc at line 40: included from ./include/wait_for_slave_to_start.inc at line 26: included from ./include/start_slave.inc at line 35: included from /home/alice/git/10.2/mysql-test/suite/rpl/t/rpl_parallel_temptable.test at line 132: At line 119: query 'let $_show_slave_status_value= query_get_value("SHOW SLAVE STATUS", $slave_param, 1)' failed: 2013: Lost connection to MySQL server during query The result from queries just before the failure was: < snip > INSERT INTO t2 VALUES (19); connection default; INSERT INTO t3 SELECT a+128 FROM t4; connection server_1; INSERT INTO t2 VALUES (20); connection default; INSERT INTO t1 SELECT a, a MOD 7 FROM t3; INSERT INTO t1 SELECT a, a MOD 7 FROM t4; connection server_1; INSERT INTO t1 SELECT a, COUNT(*) FROM t2 GROUP BY a; FLUSH TABLES; SET SESSION debug_dbug="+d,crash_dispatch_command_before"; SELECT 1; Got one of the listed errors connection default; connection default; connection server_1; INSERT INTO t1 VALUES (0, 1); connection server_2; include/start_slave.inc More results from queries before failure can be found in /home/alice/git/10.2/mysql-test/var/3/log/rpl_parallel_temptable.log Server [mysqld.2 - pid: 916, winpid: 916, exit: 256] failed during test run Server log from this test: ----------SERVER LOG START----------- 2020-01-08 18:00:49 139847131875520 [Note] /home/alice/git/10.2/sql/mysqld (mysqld 10.2.31-MariaDB-debug-log) starting as process 917 ... 2020-01-08 18:00:49 139847131875520 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4186) 2020-01-08 18:00:49 139847131875520 [Warning] Changed limits: max_open_files: 1024 max_connections: 151 (was 151) table_cache: 421 (was 2000) 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'partition' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'SEQUENCE' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Uses event mutexes 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Compressed tables use zlib 1.2.8 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Using Linux native AIO 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Number of pools: 1 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Completed initialization of buffer pool 2020-01-08 18:00:49 139846844655360 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Highest supported file format is Barracuda. 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: Waiting for purge to start 2020-01-08 18:00:49 139847131875520 [Note] InnoDB: 5.7.28 started; log sequence number 1628840 2020-01-08 18:00:49 139846534289152 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var_auto_Cizt/3/mysqld.2/data/ib_buffer_pool 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_MUTEXES' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_CMP' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_FT_DELETED' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_CMP_RESET' is disabled. 2020-01-08 18:00:49 139846534289152 [Note] InnoDB: Buffer pool(s) load completed at 200108 18:00:49 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'user_variables' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_FT_CONFIG' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled. 2020-01-08 18:00:49 139847131875520 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled. 2020-01-08 18:00:49 139847131875520 [Warning] /home/alice/git/10.2/sql/mysqld: unknown option '--loose-pam-debug' 2020-01-08 18:00:49 139847131875520 [Note] Server socket created on IP: '127.0.0.1'. 2020-01-08 18:00:49 139847131875520 [Note] Reading of all Master_info entries succeeded 2020-01-08 18:00:49 139847131875520 [Note] Added new Master_info '' to hash table 2020-01-08 18:00:49 139847131875520 [Note] /home/alice/git/10.2/sql/mysqld: ready for connections. Version: '10.2.31-MariaDB-debug-log' socket: '/home/alice/git/10.2/mysql-test/var/tmp/3/mysqld.2.sock' port: 16021 Source distribution 2020-01-08 18:00:50 139846956963584 [Note] Deleted Master_info file '/home/alice/git/10.2/mysql-test/var/3/mysqld.2/data/master.info'. 2020-01-08 18:00:50 139846956963584 [Note] Deleted Master_info file '/home/alice/git/10.2/mysql-test/var/3/mysqld.2/data/relay-log.info'. 2020-01-08 18:00:50 139846956963584 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2020-01-08 18:00:50 139846956963584 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16020', master_log_file='master-bin.000001', master_log_pos='4'. 2020-01-08 18:00:50 139846956054272 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 4, relay log './slave-relay-bin.000001' position: 4 2020-01-08 18:00:50 139846956357376 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16020',replication started in log 'master-bin.000001' at position 4 2020-01-08 18:00:51 139846956054272 [Note] Error reading relay log event: slave SQL thread was killed 2020-01-08 18:00:51 139846956054272 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329 2020-01-08 18:00:51 139846956357376 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 329 2020-01-08 18:00:51 139846956963584 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2020-01-08 18:00:51 139846956963584 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port='16020', master_log_file='master-bin.000001', master_log_pos='329'. New state master_host='127.0.0.1', master_port='16020', master_log_file='master-bin.000001', master_log_pos='329'. 2020-01-08 18:00:51 139846956963584 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos 2020-01-08 18:00:51 139846956357376 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16020',replication starts at GTID position '' 2020-01-08 18:00:51 139846956054272 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 329, relay log './slave-relay-bin.000001' position: 4; GTID position '' 2020-01-08 18:00:51 139846956054272 [Note] Error reading relay log event: slave SQL thread was killed 2020-01-08 18:00:51 139846956054272 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 508; GTID position '0-1-1' 2020-01-08 18:00:51 139846956357376 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 508; GTID position 0-1-1 2020-01-08 18:00:52 139846955751168 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 508, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-1' 2020-01-08 18:00:52 139846956054272 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16020',replication starts at GTID position '0-1-1' 200108 18:00:52 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.2.31-MariaDB-debug-log key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=2 max_threads=153 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63104 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f303c000af0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f30a810ec78 thread_stack 0x49000 /home/alice/git/10.2/sql/mysqld(my_print_stacktrace+0x38)[0x55a2e23b0338] /home/alice/git/10.2/sql/mysqld(handle_fatal_signal+0x3a3)[0x55a2e1c53064] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f30b1131390] sql/sql_plist.h:181(I_P_List_iterator<TMP_TABLE_SHARE, I_P_List<TMP_TABLE_SHARE, All_tmp_table_shares, I_P_List_null_counter, I_P_List_no_push_back<TMP_TABLE_SHARE> > >::I_P_List_iterator(I_P_List<TMP_TABLE_SHARE, All_tmp_table_shares, I_P_List_null_counter, I_P_List_no_push_back<TMP_TABLE_SHARE> > const&))[0x55a2e1bd2977] sql/temporary_tables.cc:741(THD::mark_tmp_tables_as_free_for_reuse())[0x55a2e1bd09bc] sql/sql_base.cc:805(close_thread_tables(THD*))[0x55a2e195c7e6] sql/sql_parse.cc:6053(mysql_execute_command(THD*))[0x55a2e19dae33] sql/sql_parse.cc:7740(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x55a2e19df91d] sql/log_event.cc:5350(Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int))[0x55a2e1d7b2ea] sql/log_event.cc:5029(Query_log_event::do_apply_event(rpl_group_info*))[0x55a2e1d7a305] sql/log_event.h:1452(Log_event::apply_event(rpl_group_info*))[0x55a2e192e3b9] sql/slave.cc:3567(apply_event_and_update_pos_apply(Log_event*, THD*, rpl_group_info*, int))[0x55a2e19230c7] sql/slave.cc:3743(apply_event_and_update_pos_for_parallel(Log_event*, THD*, rpl_group_info*))[0x55a2e19236ca] sql/rpl_parallel.cc:50(rpt_handle_event(rpl_parallel_thread::queued_event*, rpl_parallel_thread*))[0x55a2e1b93964] sql/rpl_parallel.cc:1316(handle_rpl_parallel_thread)[0x55a2e1b969ef] perfschema/pfs.cc:1864(pfs_spawn_thread)[0x55a2e234affa] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f30b11276ba] x86_64/clone.S:111(clone)[0x7f30b05bc41d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f303802748b): COMMIT Connection ID (thread ID): 27 Status: NOT_KILLED #3 <signal handler called> #4 0x000055a2e1bd2977 in I_P_List_iterator<TMP_TABLE_SHARE, I_P_List<TMP_TABLE_SHARE, All_tmp_table_shares, I_P_List_null_counter, I_P_List_no_push_back<TMP_TABLE_SHARE> > >::I_P_List_iterator (this=0x7f30a810d940, a=...) at /10.2/sql/sql_plist.h:181 #5 0x000055a2e1bd09bc in THD::mark_tmp_tables_as_free_for_reuse (this=0x7f303c000af0) at /10.2/sql/temporary_tables.cc:740 #6 0x000055a2e195c7e6 in close_thread_tables (thd=0x7f303c000af0) at /10.2/sql/sql_base.cc:803 #7 0x000055a2e19dae33 in mysql_execute_command (thd=0x7f303c000af0) at /10.2/sql/sql_parse.cc:6051 #8 0x000055a2e19df91d in mysql_parse (thd=0x7f303c000af0, rawbuf=0x7f303802748b "COMMIT", length=6, parser_state=0x7f30a810e510, is_com_multi=false, is_next_command=false) at /10.2/sql/sql_parse.cc:7740 #9 0x000055a2e1d7b2ea in Query_log_event::do_apply_event (this=0x7f30380272e0, rgi=0x7f3038022360, query_arg=0x7f303802748b "COMMIT", q_len_arg=6) at /10.2/sql/log_event.cc:5348 #10 0x000055a2e1d7a305 in Query_log_event::do_apply_event (this=0x7f30380272e0, rgi=0x7f3038022360) at /10.2/sql/log_event.cc:5028 #11 0x000055a2e192e3b9 in Log_event::apply_event (this=0x7f30380272e0, rgi=0x7f3038022360) at /10.2/sql/log_event.h:1452 #12 0x000055a2e19230c7 in apply_event_and_update_pos_apply (ev=0x7f30380272e0, thd=0x7f303c000af0, rgi=0x7f3038022360, reason=0) at /10.2/sql/slave.cc:3567 #13 0x000055a2e19236ca in apply_event_and_update_pos_for_parallel (ev=0x7f30380272e0, thd=0x7f303c000af0, rgi=0x7f3038022360) at /10.2/sql/slave.cc:3742 #14 0x000055a2e1b93964 in rpt_handle_event (qev=0x7f3038021f00, rpt=0x7f3038009640) at /10.2/sql/rpl_parallel.cc:50 #15 0x000055a2e1b969ef in handle_rpl_parallel_thread (arg=0x7f3038009640) at /10.2/sql/rpl_parallel.cc:1316 #16 0x000055a2e234affa in pfs_spawn_thread (arg=0x7f303800a150) at /10.2/storage/perfschema/pfs.cc:1862 #17 0x00007f30b11276ba in start_thread (arg=0x7f30a810f700) at pthread_create.c:333 #18 0x00007f30b05bc41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Julien Fritsch made changes - 2020-12-01 12:34

Fix Version/s

10.0 [ 16000 ]

Julien Fritsch made changes - 2020-12-01 12:38

Fix Version/s

10.1 [ 16100 ]

Alice Sherepa made changes - 2021-03-05 12:33

Link

This issue relates to MDEV-16539 [ MDEV-16539 ]

Alice Sherepa made changes - 2021-03-05 12:33

Assignee

Michael Widenius [ monty ]

Andrei Elkin [ elkin ]

Alice Sherepa made changes - 2021-03-05 12:40

Link

This issue relates to ~~MDEV-25064~~ [ ~~MDEV-25064~~ ]

Alice Sherepa added a comment - 2021-03-05 12:42 - edited

http://buildbot.askmonty.org/buildbot/builders/kvm-bintar-centos6-amd64-debug/builds/3023/steps/mtr/logs/stdio

10.2 21a96581fd1dd11c5605
rpl.rpl_parallel_temptable 'innodb,stmt' w4 [ fail ]
Test ended at 2020-08-26 05:04:58

CURRENT_TEST: rpl.rpl_parallel_temptable
mysqltest: At line 132: failed in 'select master_pos_wait('master-bin.000002', 585, 300, '')': 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:
< snip >
INSERT INTO t2 VALUES (19);
connection default;
INSERT INTO t3 SELECT a+128 FROM t4;
connection server_1;
INSERT INTO t2 VALUES (20);
connection default;
INSERT INTO t1 SELECT a, a MOD 7 FROM t3;
INSERT INTO t1 SELECT a, a MOD 7 FROM t4;
connection server_1;
INSERT INTO t1 SELECT a, COUNT(*) FROM t2 GROUP BY a;
FLUSH TABLES;
SET SESSION debug_dbug="+d,crash_dispatch_command_before";
SELECT 1;
Got one of the listed errors
connection default;
connection default;
connection server_1;
INSERT INTO t1 VALUES (0, 1);
connection server_2;
include/start_slave.inc

More results from queries before failure can be found in /usr/local/mariadb-10.2.34-linux-x86_64/mysql-test/var/4/log/rpl_parallel_temptable.log


Server [mysqld.2 - pid: 30821, winpid: 30821, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
2020-08-26 5:04:55 139652224616224 [Note] /usr/local/mariadb-10.2.34-linux-x86_64/bin/mysqld (mysqld 10.2.34-MariaDB-debug-log) starting as process 30833 ...
2020-08-26 5:04:55 139652224616224 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32186)
2020-08-26 5:04:55 139652224616224 [Warning] Changed limits: max_open_files: 1024 max_connections: 151 (was 151) table_cache: 421 (was 2000)
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'SEQUENCE' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'partition' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Uses event mutexes
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Using Linux native AIO
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Number of pools: 1
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Using generic crc32 instructions
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Completed initialization of buffer pool
2020-08-26 5:04:55 139651960518400 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Highest supported file format is Barracuda.
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: 128 out of 128 rollback segments are active.
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Waiting for purge to start
2020-08-26 5:04:55 139652224616224 [Note] InnoDB: 5.7.31 started; log sequence number 1628836
2020-08-26 5:04:55 139651702572800 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var_auto_7Dp2/4/mysqld.2/data/ib_buffer_pool
2020-08-26 5:04:55 139651702572800 [Note] InnoDB: Buffer pool(s) load completed at 200826 5:04:55
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_MUTEXES' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_CMP' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'FEEDBACK' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'user_variables' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
2020-08-26 5:04:55 139652224616224 [Warning] /usr/local/mariadb-10.2.34-linux-x86_64/bin/mysqld: unknown option '--loose-pam-debug'
2020-08-26 5:04:55 139652224616224 [Note] Server socket created on IP: '127.0.0.1'.
2020-08-26 5:04:55 139652224616224 [Note] Reading of all Master_info entries succeeded
2020-08-26 5:04:55 139652224616224 [Note] Added new Master_info '' to hash table
2020-08-26 5:04:55 139652224616224 [Note] /usr/local/mariadb-10.2.34-linux-x86_64/bin/mysqld: ready for connections.
Version: '10.2.34-MariaDB-debug-log' socket: '/usr/local/mariadb-10.2.34-linux-x86_64/mysql-test/var/tmp/4/mysqld.2.sock' port: 16061 MariaDB Server
2020-08-26 5:04:56 139652101568256 [Note] Deleted Master_info file '/usr/local/mariadb-10.2.34-linux-x86_64/mysql-test/var/4/mysqld.2/data/master.info'.
2020-08-26 5:04:56 139652101568256 [Note] Deleted Master_info file '/usr/local/mariadb-10.2.34-linux-x86_64/mysql-test/var/4/mysqld.2/data/relay-log.info'.
2020-08-26 5:04:56 139652101568256 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info'
2020-08-26 5:04:56 139652101568256 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='4'.
2020-08-26 5:04:56 139652077000448 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 4, relay log './slave-relay-bin.000001' position: 4
2020-08-26 5:04:56 139652100962048 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication started in log 'master-bin.000001' at position 4
2020-08-26 5:04:56 139652077000448 [Note] Error reading relay log event: slave SQL thread was killed
2020-08-26 5:04:56 139652077000448 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329
2020-08-26 5:04:56 139652100962048 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 329
2020-08-26 5:04:56 139652101568256 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info'
2020-08-26 5:04:56 139652101568256 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='329'. New state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='329'.
2020-08-26 5:04:56 139652101568256 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos
2020-08-26 5:04:56 139652100962048 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication starts at GTID position ''
2020-08-26 5:04:56 139652077000448 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 329, relay log './slave-relay-bin.000001' position: 4; GTID position ''
2020-08-26 5:04:56 139652077000448 [Note] Error reading relay log event: slave SQL thread was killed
2020-08-26 5:04:56 139652077000448 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 508; GTID position '0-1-1'
2020-08-26 5:04:56 139652100962048 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 508; GTID position 0-1-1
2020-08-26 5:04:57 139652077000448 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication starts at GTID position '0-1-1'
2020-08-26 5:04:57 139652100962048 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 508, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-1'
200826 5:04:57 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.2.34-MariaDB-debug-log
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=2
max_threads=153
thread_count=16
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63104 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f02d8000af0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f03485e8da8 thread_stack 0x49000
/usr/local/mariadb-10.2.34-linux-x86_64/bin/mysqld(my_print_stacktrace+0x2b)[0x7f03521a1949]
/usr/local/mariadb-10.2.34-linux-x86_64/bin/mysqld(handle_fatal_signal+0x30c)[0x7f0351b16ea5]
sigaction.c:0(__restore_rt)[0x7f0350fb94a0]
sql/sql_plist.h:181(THD::mark_tmp_tables_as_free_for_reuse())[0x7f0351ab8c8c]
sql/sql_base.cc:805(close_thread_tables(THD*))[0x7f03518ead44]
sql/sql_parse.cc:6046(mysql_execute_command(THD*))[0x7f035194b0d0]
sql/sql_parse.cc:7733(mysql_parse(THD, char, unsigned int, Parser_state*, bool, bool))[0x7f035194c1a8]
sql/log_event.cc:5361(Query_log_event::do_apply_event(rpl_group_info, char const, unsigned int))[0x7f0351c29fd7]
sql/log_event.cc:5040(Query_log_event::do_apply_event(rpl_group_info*))[0x7f0351c2a90a]
sql/log_event.h:1452(apply_event_and_update_pos_apply(Log_event, THD, rpl_group_info*, int))[0x7f03518ab56e]
sql/slave.cc:3764(apply_event_and_update_pos_for_parallel(Log_event, THD, rpl_group_info*))[0x7f03518b1af2]
sql/mysqld.h:759(rpt_handle_event(rpl_parallel_thread::queued_event, rpl_parallel_thread))[0x7f0351a8e59c]
sql/rpl_parallel.cc:1335(handle_rpl_parallel_thread)[0x7f0351a92943]
perfschema/pfs.cc:1872(pfs_spawn_thread)[0x7f03521257f6]
pthread_create.c:0(start_thread)[0x7f0350fb17f1]
/lib64/libc.so.6(clone+0x6d)[0x7f034ff31ccd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f02e802e8ab): COMMIT

Connection ID (thread ID): 26
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /dev/shm/var_auto_7Dp2/4/mysqld.2/data
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1024 unlimited processes
Max open files 1024 1024 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 46692 46692 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core

----------SERVER LOG END-------------


- found 'core.30833' (0/0)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace from coredump /usr/local/mariadb-10.2.34-linux-x86_64/mysql-test/var/4/log/rpl.rpl_parallel_temptable-innodb,stmt/mysqld.2/data/core.30833
Core generated by '/usr/local/mariadb-10.2.34-linux-x86_64/bin/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New Thread 32239]
[New Thread 32240]
[New Thread 32241]
[New Thread 30927]
[New Thread 30874]
[New Thread 30928]
[New Thread 30833]
[New Thread 30950]
[New Thread 30967]
[New Thread 30969]
[New Thread 30918]
[New Thread 30917]
[New Thread 30970]
[New Thread 30924]
[New Thread 30977]
[New Thread 30925]
[New Thread 30949]
[New Thread 32234]
[New Thread 30978]
[New Thread 30926]
[New Thread 32235]
[New Thread 31022]
[New Thread 31033]
[New Thread 32237]
[New Thread 30948]
[New Thread 32238]
[New Thread 30968]
[New Thread 31035]
[New Thread 30976]
[New Thread 31045]
[New Thread 31023]
[New Thread 31050]
[New Thread 31024]
[New Thread 31034]
[New Thread 31025]
[New Thread 31028]
[Thread debugging using libthread_db enabled]
Core was generated by `/usr/local/mariadb-10.2.34-linux-x86_64/bin/mysqld --defaults-group-suffix=.2 -'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f0350fb663c in pthread_kill () from /lib64/libpthread.so.0
#0 0x00007f0350fb663c in pthread_kill () from /lib64/libpthread.so.0
#1 0x00007f03521a174f in my_write_core ()
#2 0x00007f0351b17154 in handle_fatal_signal ()
#3 <signal handler called>
#4 0x00007f0351ab8c8c in THD::mark_tmp_tables_as_free_for_reuse() () at /home/buildbot/buildbot/build/sql/sql_plist.h:181
#5 0x00007f03518ead44 in close_thread_tables(THD*) () at /home/buildbot/buildbot/build/sql/sql_base.cc:803
#6 0x00007f035194b0d0 in mysql_execute_command(THD*) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:6044
#7 0x00007f035194c1a8 in mysql_parse(THD, char, unsigned int, Parser_state*, bool, bool) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:7733
#8 0x00007f0351c29fd7 in Query_log_event::do_apply_event(rpl_group_info, char const, unsigned int) () at /home/buildbot/buildbot/build/sql/log_event.cc:5359
#9 0x00007f0351c2a90a in Query_log_event::do_apply_event(rpl_group_info*) () at /home/buildbot/buildbot/build/sql/log_event.cc:5039
#10 0x00007f03518ab56e in apply_event_and_update_pos_apply(Log_event, THD, rpl_group_info*, int) () at /home/buildbot/buildbot/build/sql/log_event.h:1452
#11 0x00007f03518b1af2 in apply_event_and_update_pos_for_parallel(Log_event, THD, rpl_group_info*) () at /home/buildbot/buildbot/build/sql/slave.cc:3763
#12 0x00007f0351a8e59c in rpt_handle_event(rpl_parallel_thread::queued_event, rpl_parallel_thread) () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:50
#13 0x00007f0351a92943 in handle_rpl_parallel_thread () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:1335
#14 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#15 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x7f0312e59700 (LWP 31028)):
#0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f03521a6b66 in safe_cond_timedwait ()
#2 0x00007f035205c69d in my_service_thread_sleep ()
#3 0x00007f0352054878 in ma_checkpoint_background ()
#4 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x7f0313fff700 (LWP 31025)):
#0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f03521a6974 in safe_cond_wait ()
#2 0x00007f0351cca5bb in thd_destructor_proxy () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1156
#3 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#4 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7f0349f13700 (LWP 31034)):
#0 0x00007f0350fb9245 in sigwait () from /lib64/libpthread.so.0
#1 0x00007f035188a11f in signal_hand () at /home/buildbot/buildbot/build/sql/mysqld.cc:3510
#2 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#3 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7f0330dfa700 (LWP 31024)):
#0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0351d7b940 in os_event::wait_low(long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158
#2 0x00007f0351d7bcde in os_event_wait_low(os_event*, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507
#3 0x00007f0351f1b062 in buf_resize_thread () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0buf.cc:3127
#4 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7f0349e35700 (LWP 31050)):
#0 0x00007f034ff28613 in poll () from /lib64/libc.so.6
#1 0x00007f03521efdd8 in vio_io_wait ()
#2 0x00007f03521eff29 in vio_socket_io_wait ()
#3 0x00007f03521f004a in vio_read ()
#4 0x00007f03518969ec in my_real_read(st_net, unsigned long, char) ()
#5 0x00007f035189782d in my_net_read_packet_reallen ()
#6 0x00007f0351897a96 in my_net_read_packet ()
#7 0x00007f0351950bf5 in do_command(THD*) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:1253
#8 0x00007f0351a3880d in do_handle_one_connection(CONNECT*) () at /home/buildbot/buildbot/build/sql/sql_connect.cc:1336
#9 0x00007f0351a38946 in handle_one_connection () at /home/buildbot/buildbot/build/sql/sql_connect.cc:1241
#10 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#11 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7f03317fb700 (LWP 31023)):
#0 0x00007f0350fb8ccd in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f0351d7c1ea in os_thread_sleep(unsigned long) ()
#2 0x00007f0351f01f09 in btr_defragment_thread () at /home/buildbot/buildbot/build/storage/innobase/btr/btr0defragment.cc:715
#3 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7f0349e7f700 (LWP 31045)):
#0 0x00007f0350fb7ff4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f0350fb335e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x00007f0350fb32c6 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f03521a6081 in safe_mutex_lock ()
#4 0x00007f0351a294fa in inline_mysql_mutex_lock () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:673
#5 0x00007f0351a2a9eb in Relay_log_info::wait_for_pos(THD, String, long long, long long) () at /home/buildbot/buildbot/build/sql/rpl_rli.cc:809
#6 0x00007f0351b89376 in Item_master_pos_wait::val_int() () at /home/buildbot/buildbot/build/sql/item_func.cc:3941
#7 0x00007f0351b2bd74 in Item::send(Protocol, String) () at /home/buildbot/buildbot/build/sql/item.cc:6940
#8 0x00007f035189da5f in Protocol::send_result_set_row(List<Item>*) () at /home/buildbot/buildbot/build/sql/protocol.cc:990
#9 0x00007f035190e519 in select_send::send_data(List<Item>&) () at /home/buildbot/buildbot/build/sql/sql_class.cc:2731
#10 0x00007f03519a63db in JOIN::exec_inner() () at /home/buildbot/buildbot/build/sql/sql_select.cc:3514
#11 0x00007f03519a706f in JOIN::exec() () at /home/buildbot/buildbot/build/sql/sql_select.cc:3433
#12 0x00007f03519a585a in mysql_select(THD, TABLE_LIST, unsigned int, List<Item>&, Item, unsigned int, st_order, st_order, Item, st_order, unsigned long long, select_result, st_select_lex_unit, st_select_lex) () at /home/buildbot/buildbot/build/sql/sql_select.cc:3833
#13 0x00007f03519a5ae7 in handle_select(THD, LEX, select_result*, unsigned long) () at /home/buildbot/buildbot/build/sql/sql_select.cc:373
#14 0x00007f035193a072 in execute_sqlcom_select(THD, TABLE_LIST) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:6218
#15 0x00007f035194319a in mysql_execute_command(THD*) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:3524
#16 0x00007f035194c1a8 in mysql_parse(THD, char, unsigned int, Parser_state*, bool, bool) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:7733
#17 0x00007f035194e4cb in dispatch_command(enum_server_command, THD, char, unsigned int, bool, bool) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:1824
#18 0x00007f0351950faf in do_command(THD*) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:1378
#19 0x00007f0351a3880d in do_handle_one_connection(CONNECT*) () at /home/buildbot/buildbot/build/sql/sql_connect.cc:1336
#20 0x00007f0351a38946 in handle_one_connection () at /home/buildbot/buildbot/build/sql/sql_connect.cc:1241
#21 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#22 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7f0333fff700 (LWP 30976)):
#0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0351d7b940 in os_event::wait_low(long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158
#2 0x00007f0351d7bcde in os_event_wait_low(os_event*, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507
#3 0x00007f0351e53cd0 in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:940
#4 0x00007f0351e5582b in srv_worker_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2620
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f0349ec9700 (LWP 31035)):
#0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f03521a6974 in safe_cond_wait ()
#2 0x00007f03518ab257 in handle_slave_background () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145
#3 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#4 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f033a1fc700 (LWP 30968)):
#0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0351d7bd03 in os_event::timed_wait(timespec const*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275
#2 0x00007f0351d7bf25 in os_event::wait_time_low(unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385
#3 0x00007f0351d7bfa1 in os_event_wait_time_low(os_event*, unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:490
#4 0x00007f0351f80b24 in dict_stats_thread () at /home/buildbot/buildbot/build/storage/innobase/dict/dict0stats_bg.cc:475
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f034867d700 (LWP 32238)):
#0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f03521a6974 in safe_cond_wait ()
#2 0x00007f0351a8e346 in inline_mysql_cond_wait () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145
#3 0x00007f0351a916fd in handle_rpl_parallel_thread () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:1114
#4 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f033bfff700 (LWP 30948)):
#0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0351d7bd03 in os_event::timed_wait(timespec const*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275
#2 0x00007f0351d7bf25 in os_event::wait_time_low(unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385
#3 0x00007f0351d7bfa1 in os_event_wait_time_low(os_event*, unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:490
#4 0x00007f0351d452b2 in lock_wait_timeout_thread () at /home/buildbot/buildbot/build/storage/innobase/lock/lock0wait.cc:525
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f0348633700 (LWP 32237)):
#0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f03521a6974 in safe_cond_wait ()
#2 0x00007f0351a8e346 in inline_mysql_cond_wait () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145
#3 0x00007f0351a916fd in handle_rpl_parallel_thread () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:1114
#4 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f0349f5d700 (LWP 31033)):
#0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f03521a6974 in safe_cond_wait ()
#2 0x00007f0351c00a54 in inline_mysql_cond_wait () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145
#3 0x00007f0351c10228 in binlog_background_thread () at /home/buildbot/buildbot/build/sql/log.cc:9905
#4 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f03321fc700 (LWP 31022)):
#0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0351d7b940 in os_event::wait_low(long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158
#2 0x00007f0351d7bcde in os_event_wait_low(os_event*, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507
#3 0x00007f0351f235ba in buf_dump_thread () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0dump.cc:806
#4 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f0349deb700 (LWP 32235)):
#0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f03521a6974 in safe_cond_wait ()
#2 0x00007f0351c0ae47 in MYSQL_BIN_LOG::wait_for_update_relay_log(THD*) () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145
#3 0x00007f03518a9cab in next_event(rpl_group_info, unsigned long long) () at /home/buildbot/buildbot/build/sql/slave.cc:7237
#4 0x00007f03518b2f3f in handle_slave_sql () at /home/buildbot/buildbot/build/sql/slave.cc:3884
#5 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f0342bfd700 (LWP 30926)):
#0 0x00007f0350da9614 in ?? () from /lib64/libaio.so.1
#1 0x00007f0351d75da6 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2 0x00007f0351d7879e in LinuxAIOHandler::poll(fil_node_t, void, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3 0x00007f0351d78fac in os_aio_handler(unsigned long, fil_node_t, void, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4 0x00007f0351f97d66 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5098
#5 0x00007f0351e5a03f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f0332bfd700 (LWP 30978)):
#0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0351d7b940 in os_event::wait_low(long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158
#2 0x00007f0351d7bcde in os_event_wait_low(os_event*, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507
#3 0x00007f0351e53cd0 in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:940
#4 0x00007f0351e5582b in srv_worker_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2620
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f0348711700 (LWP 32234)):
#0 0x00007f034ff28613 in poll () from /lib64/libc.so.6
#1 0x00007f03521efdd8 in vio_io_wait ()
#2 0x00007f03521eff29 in vio_socket_io_wait ()
#3 0x00007f03521f004a in vio_read ()
#4 0x00007f03521f0274 in vio_read_buff ()
#5 0x00007f03518969ec in my_real_read(st_net, unsigned long, char) ()
#6 0x00007f035189782d in my_net_read_packet_reallen ()
#7 0x00007f0351ae8a28 in cli_safe_read_reallen ()
#8 0x00007f03518b7fef in handle_slave_io () at /home/buildbot/buildbot/build/sql/slave.cc:3348
#9 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#10 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f0340dfa700 (LWP 30949)):
#0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0351d7bd03 in os_event::timed_wait(timespec const*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275
#2 0x00007f0351d7bf25 in os_event::wait_time_low(unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385
#3 0x00007f0351d7bfa1 in os_event_wait_time_low(os_event*, unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:490
#4 0x00007f0351e52e68 in srv_error_monitor_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:1915
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f03435fe700 (LWP 30925)):
#0 0x00007f0350da9614 in ?? () from /lib64/libaio.so.1
#1 0x00007f0351d75da6 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2 0x00007f0351d7879e in LinuxAIOHandler::poll(fil_node_t, void, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3 0x00007f0351d78fac in os_aio_handler(unsigned long, fil_node_t, void, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4 0x00007f0351f97d66 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5098
#5 0x00007f0351e5a03f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f03335fe700 (LWP 30977)):
#0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0351d7b940 in os_event::wait_low(long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158
#2 0x00007f0351d7bcde in os_event_wait_low(os_event*, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507
#3 0x00007f0351e53cd0 in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:940
#4 0x00007f0351e5582b in srv_worker_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2620
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f0343fff700 (LWP 30924)):
#0 0x00007f0350da9614 in ?? () from /lib64/libaio.so.1
#1 0x00007f0351d75da6 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2 0x00007f0351d7879e in LinuxAIOHandler::poll(fil_node_t, void, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3 0x00007f0351d78fac in os_aio_handler(unsigned long, fil_node_t, void, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4 0x00007f0351f97d66 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5098
#5 0x00007f0351e5a03f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f0338dfa700 (LWP 30970)):
#0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0351d7b940 in os_event::wait_low(long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158
#2 0x00007f0351d7bcde in os_event_wait_low(os_event*, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507
#3 0x00007f0351e53cd0 in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:940
#4 0x00007f0351e5651c in srv_purge_coordinator_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2781
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f0349d39700 (LWP 30917)):
#0 0x00007f0350da9614 in ?? () from /lib64/libaio.so.1
#1 0x00007f0351d75da6 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2 0x00007f0351d7879e in LinuxAIOHandler::poll(fil_node_t, void, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3 0x00007f0351d78fac in os_aio_handler(unsigned long, fil_node_t, void, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4 0x00007f0351f97d66 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5098
#5 0x00007f0351e5a03f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f0349338700 (LWP 30918)):
#0 0x00007f0350da9614 in ?? () from /lib64/libaio.so.1
#1 0x00007f0351d75da6 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2 0x00007f0351d7879e in LinuxAIOHandler::poll(fil_node_t, void, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3 0x00007f0351d78fac in os_aio_handler(unsigned long, fil_node_t, void, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4 0x00007f0351f97d66 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5098
#5 0x00007f0351e5a03f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f03397fb700 (LWP 30969)):
#0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0351d7bd03 in os_event::timed_wait(timespec const*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275
#2 0x00007f0351d7bf25 in os_event::wait_time_low(unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385
#3 0x00007f0351d7bfa1 in os_event_wait_time_low(os_event*, unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:490
#4 0x00007f0351eb6304 in ib_wqueue_timedwait(ib_wqueue_t*, unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/ut/ut0wqueue.cc:156
#5 0x00007f0351fdee21 in fts_optimize_thread(void*) () at /home/buildbot/buildbot/build/storage/innobase/fts/fts0opt.cc:2820
#6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f033abfd700 (LWP 30967)):
#0 0x00007f0350fb8ccd in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f0351d7c1ea in os_thread_sleep(unsigned long) ()
#2 0x00007f0351e54149 in srv_master_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2422
#3 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f033b5fe700 (LWP 30950)):
#0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0351d7bd03 in os_event::timed_wait(timespec const*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275
#2 0x00007f0351d7bf25 in os_event::wait_time_low(unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385
#3 0x00007f0351d7bfa1 in os_event_wait_time_low(os_event*, unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:490
#4 0x00007f0351e5923d in srv_monitor_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:1751
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f03513d8720 (LWP 30833)):
#0 0x00007f034ff28613 in poll () from /lib64/libc.so.6
#1 0x00007f03518938d5 in handle_connections_sockets() () at /home/buildbot/buildbot/build/sql/mysqld.cc:6668
#2 0x00007f03518957ab in mysqld_main(int, char**) () at /home/buildbot/buildbot/build/sql/mysqld.cc:6136
#3 0x00007f0351886dce in main ()

Thread 6 (Thread 0x7f03417fb700 (LWP 30928)):
#0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0351d7bd03 in os_event::timed_wait(timespec const*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275
#2 0x00007f0351d7bf25 in os_event::wait_time_low(unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385
#3 0x00007f0351d7bfa1 in os_event_wait_time_low(os_event*, unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:490
#4 0x00007f0351f2c396 in buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0flu.cc:2683
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f034b994700 (LWP 30874)):
#0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f03521a6b66 in safe_cond_timedwait ()
#2 0x00007f03521a721a in timer_handler ()
#3 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#4 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f03421fc700 (LWP 30927)):
#0 0x00007f0350da9614 in ?? () from /lib64/libaio.so.1
#1 0x00007f0351d75da6 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936
#2 0x00007f0351d7879e in LinuxAIOHandler::poll(fil_node_t, void, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097
#3 0x00007f0351d78fac in os_aio_handler(unsigned long, fil_node_t, void, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151
#4 0x00007f0351f97d66 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5098
#5 0x00007f0351e5a03f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332
#6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f034859f700 (LWP 32241)):
#0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f03521a6974 in safe_cond_wait ()
#2 0x00007f0351a8e346 in inline_mysql_cond_wait () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145
#3 0x00007f0351a916fd in handle_rpl_parallel_thread () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:1114
#4 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f03486c7700 (LWP 32240)):
#0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f03521a6974 in safe_cond_wait ()
#2 0x00007f0351a8e346 in inline_mysql_cond_wait () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145
#3 0x00007f0351a916fd in handle_rpl_parallel_thread () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:1114
#4 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f03485e9700 (LWP 32239)):
#0 0x00007f0350fb663c in pthread_kill () from /lib64/libpthread.so.0
#1 0x00007f03521a174f in my_write_core ()
#2 0x00007f0351b17154 in handle_fatal_signal ()
#3 <signal handler called>
#4 0x00007f0351ab8c8c in THD::mark_tmp_tables_as_free_for_reuse() () at /home/buildbot/buildbot/build/sql/sql_plist.h:181
#5 0x00007f03518ead44 in close_thread_tables(THD*) () at /home/buildbot/buildbot/build/sql/sql_base.cc:803
#6 0x00007f035194b0d0 in mysql_execute_command(THD*) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:6044
#7 0x00007f035194c1a8 in mysql_parse(THD, char, unsigned int, Parser_state*, bool, bool) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:7733
#8 0x00007f0351c29fd7 in Query_log_event::do_apply_event(rpl_group_info, char const, unsigned int) () at /home/buildbot/buildbot/build/sql/log_event.cc:5359
#9 0x00007f0351c2a90a in Query_log_event::do_apply_event(rpl_group_info*) () at /home/buildbot/buildbot/build/sql/log_event.cc:5039
#10 0x00007f03518ab56e in apply_event_and_update_pos_apply(Log_event, THD, rpl_group_info*, int) () at /home/buildbot/buildbot/build/sql/log_event.h:1452
#11 0x00007f03518b1af2 in apply_event_and_update_pos_for_parallel(Log_event, THD, rpl_group_info*) () at /home/buildbot/buildbot/build/sql/slave.cc:3763
#12 0x00007f0351a8e59c in rpt_handle_event(rpl_parallel_thread::queued_event, rpl_parallel_thread) () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:50
#13 0x00007f0351a92943 in handle_rpl_parallel_thread () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:1335
#14 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869
#15 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Alice Sherepa added a comment - 2021-03-05 12:42 - edited http://buildbot.askmonty.org/buildbot/builders/kvm-bintar-centos6-amd64-debug/builds/3023/steps/mtr/logs/stdio 10.2 21a96581fd1dd11c5605 rpl.rpl_parallel_temptable 'innodb,stmt' w4 [ fail ] Test ended at 2020-08-26 05:04:58 CURRENT_TEST: rpl.rpl_parallel_temptable mysqltest: At line 132: failed in 'select master_pos_wait('master-bin.000002', 585, 300, '')': 2013: Lost connection to MySQL server during query The result from queries just before the failure was: < snip > INSERT INTO t2 VALUES (19); connection default; INSERT INTO t3 SELECT a+128 FROM t4; connection server_1; INSERT INTO t2 VALUES (20); connection default; INSERT INTO t1 SELECT a, a MOD 7 FROM t3; INSERT INTO t1 SELECT a, a MOD 7 FROM t4; connection server_1; INSERT INTO t1 SELECT a, COUNT(*) FROM t2 GROUP BY a; FLUSH TABLES; SET SESSION debug_dbug="+d,crash_dispatch_command_before"; SELECT 1; Got one of the listed errors connection default; connection default; connection server_1; INSERT INTO t1 VALUES (0, 1); connection server_2; include/start_slave.inc More results from queries before failure can be found in /usr/local/mariadb-10.2.34-linux-x86_64/mysql-test/var/4/log/rpl_parallel_temptable.log Server [mysqld.2 - pid: 30821, winpid: 30821, exit: 256] failed during test run Server log from this test: ----------SERVER LOG START----------- 2020-08-26 5:04:55 139652224616224 [Note] /usr/local/mariadb-10.2.34-linux-x86_64/bin/mysqld (mysqld 10.2.34-MariaDB-debug-log) starting as process 30833 ... 2020-08-26 5:04:55 139652224616224 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32186) 2020-08-26 5:04:55 139652224616224 [Warning] Changed limits: max_open_files: 1024 max_connections: 151 (was 151) table_cache: 421 (was 2000) 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'SEQUENCE' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'partition' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Uses event mutexes 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Compressed tables use zlib 1.2.11 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Using Linux native AIO 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Number of pools: 1 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Using generic crc32 instructions 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Completed initialization of buffer pool 2020-08-26 5:04:55 139651960518400 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Highest supported file format is Barracuda. 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: Waiting for purge to start 2020-08-26 5:04:55 139652224616224 [Note] InnoDB: 5.7.31 started; log sequence number 1628836 2020-08-26 5:04:55 139651702572800 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var_auto_7Dp2/4/mysqld.2/data/ib_buffer_pool 2020-08-26 5:04:55 139651702572800 [Note] InnoDB: Buffer pool(s) load completed at 200826 5:04:55 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_MUTEXES' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_CMP' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_FT_DELETED' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_CMP_RESET' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'FEEDBACK' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'user_variables' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_FT_CONFIG' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled. 2020-08-26 5:04:55 139652224616224 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled. 2020-08-26 5:04:55 139652224616224 [Warning] /usr/local/mariadb-10.2.34-linux-x86_64/bin/mysqld: unknown option '--loose-pam-debug' 2020-08-26 5:04:55 139652224616224 [Note] Server socket created on IP: '127.0.0.1'. 2020-08-26 5:04:55 139652224616224 [Note] Reading of all Master_info entries succeeded 2020-08-26 5:04:55 139652224616224 [Note] Added new Master_info '' to hash table 2020-08-26 5:04:55 139652224616224 [Note] /usr/local/mariadb-10.2.34-linux-x86_64/bin/mysqld: ready for connections. Version: '10.2.34-MariaDB-debug-log' socket: '/usr/local/mariadb-10.2.34-linux-x86_64/mysql-test/var/tmp/4/mysqld.2.sock' port: 16061 MariaDB Server 2020-08-26 5:04:56 139652101568256 [Note] Deleted Master_info file '/usr/local/mariadb-10.2.34-linux-x86_64/mysql-test/var/4/mysqld.2/data/master.info'. 2020-08-26 5:04:56 139652101568256 [Note] Deleted Master_info file '/usr/local/mariadb-10.2.34-linux-x86_64/mysql-test/var/4/mysqld.2/data/relay-log.info'. 2020-08-26 5:04:56 139652101568256 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2020-08-26 5:04:56 139652101568256 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='4'. 2020-08-26 5:04:56 139652077000448 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 4, relay log './slave-relay-bin.000001' position: 4 2020-08-26 5:04:56 139652100962048 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication started in log 'master-bin.000001' at position 4 2020-08-26 5:04:56 139652077000448 [Note] Error reading relay log event: slave SQL thread was killed 2020-08-26 5:04:56 139652077000448 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329 2020-08-26 5:04:56 139652100962048 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 329 2020-08-26 5:04:56 139652101568256 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2020-08-26 5:04:56 139652101568256 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='329'. New state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='329'. 2020-08-26 5:04:56 139652101568256 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos 2020-08-26 5:04:56 139652100962048 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication starts at GTID position '' 2020-08-26 5:04:56 139652077000448 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 329, relay log './slave-relay-bin.000001' position: 4; GTID position '' 2020-08-26 5:04:56 139652077000448 [Note] Error reading relay log event: slave SQL thread was killed 2020-08-26 5:04:56 139652077000448 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 508; GTID position '0-1-1' 2020-08-26 5:04:56 139652100962048 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 508; GTID position 0-1-1 2020-08-26 5:04:57 139652077000448 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication starts at GTID position '0-1-1' 2020-08-26 5:04:57 139652100962048 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 508, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-1' 200826 5:04:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.2.34-MariaDB-debug-log key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=2 max_threads=153 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63104 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f02d8000af0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f03485e8da8 thread_stack 0x49000 /usr/local/mariadb-10.2.34-linux-x86_64/bin/mysqld(my_print_stacktrace+0x2b)[0x7f03521a1949] /usr/local/mariadb-10.2.34-linux-x86_64/bin/mysqld(handle_fatal_signal+0x30c)[0x7f0351b16ea5] sigaction.c:0(__restore_rt)[0x7f0350fb94a0] sql/sql_plist.h:181(THD::mark_tmp_tables_as_free_for_reuse())[0x7f0351ab8c8c] sql/sql_base.cc:805(close_thread_tables(THD*))[0x7f03518ead44] sql/sql_parse.cc:6046(mysql_execute_command(THD*))[0x7f035194b0d0] sql/sql_parse.cc:7733(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x7f035194c1a8] sql/log_event.cc:5361(Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int))[0x7f0351c29fd7] sql/log_event.cc:5040(Query_log_event::do_apply_event(rpl_group_info*))[0x7f0351c2a90a] sql/log_event.h:1452(apply_event_and_update_pos_apply(Log_event*, THD*, rpl_group_info*, int))[0x7f03518ab56e] sql/slave.cc:3764(apply_event_and_update_pos_for_parallel(Log_event*, THD*, rpl_group_info*))[0x7f03518b1af2] sql/mysqld.h:759(rpt_handle_event(rpl_parallel_thread::queued_event*, rpl_parallel_thread*))[0x7f0351a8e59c] sql/rpl_parallel.cc:1335(handle_rpl_parallel_thread)[0x7f0351a92943] perfschema/pfs.cc:1872(pfs_spawn_thread)[0x7f03521257f6] pthread_create.c:0(start_thread)[0x7f0350fb17f1] /lib64/libc.so.6(clone+0x6d)[0x7f034ff31ccd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f02e802e8ab): COMMIT Connection ID (thread ID): 26 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /dev/shm/var_auto_7Dp2/4/mysqld.2/data Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 10485760 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 1024 unlimited processes Max open files 1024 1024 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 46692 46692 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core ----------SERVER LOG END------------- - found 'core.30833' (0/0) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /usr/local/mariadb-10.2.34-linux-x86_64/mysql-test/var/4/log/rpl.rpl_parallel_temptable-innodb,stmt/mysqld.2/data/core.30833 Core generated by '/usr/local/mariadb-10.2.34-linux-x86_64/bin/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New Thread 32239] [New Thread 32240] [New Thread 32241] [New Thread 30927] [New Thread 30874] [New Thread 30928] [New Thread 30833] [New Thread 30950] [New Thread 30967] [New Thread 30969] [New Thread 30918] [New Thread 30917] [New Thread 30970] [New Thread 30924] [New Thread 30977] [New Thread 30925] [New Thread 30949] [New Thread 32234] [New Thread 30978] [New Thread 30926] [New Thread 32235] [New Thread 31022] [New Thread 31033] [New Thread 32237] [New Thread 30948] [New Thread 32238] [New Thread 30968] [New Thread 31035] [New Thread 30976] [New Thread 31045] [New Thread 31023] [New Thread 31050] [New Thread 31024] [New Thread 31034] [New Thread 31025] [New Thread 31028] [Thread debugging using libthread_db enabled] Core was generated by `/usr/local/mariadb-10.2.34-linux-x86_64/bin/mysqld --defaults-group-suffix=.2 -'. Program terminated with signal 11, Segmentation fault. #0 0x00007f0350fb663c in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f0350fb663c in pthread_kill () from /lib64/libpthread.so.0 #1 0x00007f03521a174f in my_write_core () #2 0x00007f0351b17154 in handle_fatal_signal () #3 <signal handler called> #4 0x00007f0351ab8c8c in THD::mark_tmp_tables_as_free_for_reuse() () at /home/buildbot/buildbot/build/sql/sql_plist.h:181 #5 0x00007f03518ead44 in close_thread_tables(THD*) () at /home/buildbot/buildbot/build/sql/sql_base.cc:803 #6 0x00007f035194b0d0 in mysql_execute_command(THD*) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:6044 #7 0x00007f035194c1a8 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:7733 #8 0x00007f0351c29fd7 in Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int) () at /home/buildbot/buildbot/build/sql/log_event.cc:5359 #9 0x00007f0351c2a90a in Query_log_event::do_apply_event(rpl_group_info*) () at /home/buildbot/buildbot/build/sql/log_event.cc:5039 #10 0x00007f03518ab56e in apply_event_and_update_pos_apply(Log_event*, THD*, rpl_group_info*, int) () at /home/buildbot/buildbot/build/sql/log_event.h:1452 #11 0x00007f03518b1af2 in apply_event_and_update_pos_for_parallel(Log_event*, THD*, rpl_group_info*) () at /home/buildbot/buildbot/build/sql/slave.cc:3763 #12 0x00007f0351a8e59c in rpt_handle_event(rpl_parallel_thread::queued_event*, rpl_parallel_thread*) () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:50 #13 0x00007f0351a92943 in handle_rpl_parallel_thread () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:1335 #14 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #15 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #16 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7f0312e59700 (LWP 31028)): #0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f03521a6b66 in safe_cond_timedwait () #2 0x00007f035205c69d in my_service_thread_sleep () #3 0x00007f0352054878 in ma_checkpoint_background () #4 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f0313fff700 (LWP 31025)): #0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f03521a6974 in safe_cond_wait () #2 0x00007f0351cca5bb in thd_destructor_proxy () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1156 #3 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #4 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f0349f13700 (LWP 31034)): #0 0x00007f0350fb9245 in sigwait () from /lib64/libpthread.so.0 #1 0x00007f035188a11f in signal_hand () at /home/buildbot/buildbot/build/sql/mysqld.cc:3510 #2 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #3 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f0330dfa700 (LWP 31024)): #0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0351d7b940 in os_event::wait_low(long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158 #2 0x00007f0351d7bcde in os_event_wait_low(os_event*, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507 #3 0x00007f0351f1b062 in buf_resize_thread () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0buf.cc:3127 #4 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f0349e35700 (LWP 31050)): #0 0x00007f034ff28613 in poll () from /lib64/libc.so.6 #1 0x00007f03521efdd8 in vio_io_wait () #2 0x00007f03521eff29 in vio_socket_io_wait () #3 0x00007f03521f004a in vio_read () #4 0x00007f03518969ec in my_real_read(st_net*, unsigned long*, char) () #5 0x00007f035189782d in my_net_read_packet_reallen () #6 0x00007f0351897a96 in my_net_read_packet () #7 0x00007f0351950bf5 in do_command(THD*) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:1253 #8 0x00007f0351a3880d in do_handle_one_connection(CONNECT*) () at /home/buildbot/buildbot/build/sql/sql_connect.cc:1336 #9 0x00007f0351a38946 in handle_one_connection () at /home/buildbot/buildbot/build/sql/sql_connect.cc:1241 #10 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #11 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f03317fb700 (LWP 31023)): #0 0x00007f0350fb8ccd in nanosleep () from /lib64/libpthread.so.0 #1 0x00007f0351d7c1ea in os_thread_sleep(unsigned long) () #2 0x00007f0351f01f09 in btr_defragment_thread () at /home/buildbot/buildbot/build/storage/innobase/btr/btr0defragment.cc:715 #3 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f0349e7f700 (LWP 31045)): #0 0x00007f0350fb7ff4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f0350fb335e in _L_lock_995 () from /lib64/libpthread.so.0 #2 0x00007f0350fb32c6 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007f03521a6081 in safe_mutex_lock () #4 0x00007f0351a294fa in inline_mysql_mutex_lock () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:673 #5 0x00007f0351a2a9eb in Relay_log_info::wait_for_pos(THD*, String*, long long, long long) () at /home/buildbot/buildbot/build/sql/rpl_rli.cc:809 #6 0x00007f0351b89376 in Item_master_pos_wait::val_int() () at /home/buildbot/buildbot/build/sql/item_func.cc:3941 #7 0x00007f0351b2bd74 in Item::send(Protocol*, String*) () at /home/buildbot/buildbot/build/sql/item.cc:6940 #8 0x00007f035189da5f in Protocol::send_result_set_row(List<Item>*) () at /home/buildbot/buildbot/build/sql/protocol.cc:990 #9 0x00007f035190e519 in select_send::send_data(List<Item>&) () at /home/buildbot/buildbot/build/sql/sql_class.cc:2731 #10 0x00007f03519a63db in JOIN::exec_inner() () at /home/buildbot/buildbot/build/sql/sql_select.cc:3514 #11 0x00007f03519a706f in JOIN::exec() () at /home/buildbot/buildbot/build/sql/sql_select.cc:3433 #12 0x00007f03519a585a in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) () at /home/buildbot/buildbot/build/sql/sql_select.cc:3833 #13 0x00007f03519a5ae7 in handle_select(THD*, LEX*, select_result*, unsigned long) () at /home/buildbot/buildbot/build/sql/sql_select.cc:373 #14 0x00007f035193a072 in execute_sqlcom_select(THD*, TABLE_LIST*) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:6218 #15 0x00007f035194319a in mysql_execute_command(THD*) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:3524 #16 0x00007f035194c1a8 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:7733 #17 0x00007f035194e4cb in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:1824 #18 0x00007f0351950faf in do_command(THD*) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:1378 #19 0x00007f0351a3880d in do_handle_one_connection(CONNECT*) () at /home/buildbot/buildbot/build/sql/sql_connect.cc:1336 #20 0x00007f0351a38946 in handle_one_connection () at /home/buildbot/buildbot/build/sql/sql_connect.cc:1241 #21 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #22 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #23 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f0333fff700 (LWP 30976)): #0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0351d7b940 in os_event::wait_low(long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158 #2 0x00007f0351d7bcde in os_event_wait_low(os_event*, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507 #3 0x00007f0351e53cd0 in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:940 #4 0x00007f0351e5582b in srv_worker_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2620 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f0349ec9700 (LWP 31035)): #0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f03521a6974 in safe_cond_wait () #2 0x00007f03518ab257 in handle_slave_background () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145 #3 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #4 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f033a1fc700 (LWP 30968)): #0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0351d7bd03 in os_event::timed_wait(timespec const*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275 #2 0x00007f0351d7bf25 in os_event::wait_time_low(unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385 #3 0x00007f0351d7bfa1 in os_event_wait_time_low(os_event*, unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:490 #4 0x00007f0351f80b24 in dict_stats_thread () at /home/buildbot/buildbot/build/storage/innobase/dict/dict0stats_bg.cc:475 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f034867d700 (LWP 32238)): #0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f03521a6974 in safe_cond_wait () #2 0x00007f0351a8e346 in inline_mysql_cond_wait () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145 #3 0x00007f0351a916fd in handle_rpl_parallel_thread () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:1114 #4 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f033bfff700 (LWP 30948)): #0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0351d7bd03 in os_event::timed_wait(timespec const*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275 #2 0x00007f0351d7bf25 in os_event::wait_time_low(unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385 #3 0x00007f0351d7bfa1 in os_event_wait_time_low(os_event*, unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:490 #4 0x00007f0351d452b2 in lock_wait_timeout_thread () at /home/buildbot/buildbot/build/storage/innobase/lock/lock0wait.cc:525 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f0348633700 (LWP 32237)): #0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f03521a6974 in safe_cond_wait () #2 0x00007f0351a8e346 in inline_mysql_cond_wait () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145 #3 0x00007f0351a916fd in handle_rpl_parallel_thread () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:1114 #4 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f0349f5d700 (LWP 31033)): #0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f03521a6974 in safe_cond_wait () #2 0x00007f0351c00a54 in inline_mysql_cond_wait () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145 #3 0x00007f0351c10228 in binlog_background_thread () at /home/buildbot/buildbot/build/sql/log.cc:9905 #4 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f03321fc700 (LWP 31022)): #0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0351d7b940 in os_event::wait_low(long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158 #2 0x00007f0351d7bcde in os_event_wait_low(os_event*, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507 #3 0x00007f0351f235ba in buf_dump_thread () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0dump.cc:806 #4 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f0349deb700 (LWP 32235)): #0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f03521a6974 in safe_cond_wait () #2 0x00007f0351c0ae47 in MYSQL_BIN_LOG::wait_for_update_relay_log(THD*) () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145 #3 0x00007f03518a9cab in next_event(rpl_group_info*, unsigned long long*) () at /home/buildbot/buildbot/build/sql/slave.cc:7237 #4 0x00007f03518b2f3f in handle_slave_sql () at /home/buildbot/buildbot/build/sql/slave.cc:3884 #5 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f0342bfd700 (LWP 30926)): #0 0x00007f0350da9614 in ?? () from /lib64/libaio.so.1 #1 0x00007f0351d75da6 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936 #2 0x00007f0351d7879e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097 #3 0x00007f0351d78fac in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151 #4 0x00007f0351f97d66 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5098 #5 0x00007f0351e5a03f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332 #6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f0332bfd700 (LWP 30978)): #0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0351d7b940 in os_event::wait_low(long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158 #2 0x00007f0351d7bcde in os_event_wait_low(os_event*, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507 #3 0x00007f0351e53cd0 in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:940 #4 0x00007f0351e5582b in srv_worker_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2620 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f0348711700 (LWP 32234)): #0 0x00007f034ff28613 in poll () from /lib64/libc.so.6 #1 0x00007f03521efdd8 in vio_io_wait () #2 0x00007f03521eff29 in vio_socket_io_wait () #3 0x00007f03521f004a in vio_read () #4 0x00007f03521f0274 in vio_read_buff () #5 0x00007f03518969ec in my_real_read(st_net*, unsigned long*, char) () #6 0x00007f035189782d in my_net_read_packet_reallen () #7 0x00007f0351ae8a28 in cli_safe_read_reallen () #8 0x00007f03518b7fef in handle_slave_io () at /home/buildbot/buildbot/build/sql/slave.cc:3348 #9 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #10 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f0340dfa700 (LWP 30949)): #0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0351d7bd03 in os_event::timed_wait(timespec const*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275 #2 0x00007f0351d7bf25 in os_event::wait_time_low(unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385 #3 0x00007f0351d7bfa1 in os_event_wait_time_low(os_event*, unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:490 #4 0x00007f0351e52e68 in srv_error_monitor_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:1915 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f03435fe700 (LWP 30925)): #0 0x00007f0350da9614 in ?? () from /lib64/libaio.so.1 #1 0x00007f0351d75da6 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936 #2 0x00007f0351d7879e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097 #3 0x00007f0351d78fac in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151 #4 0x00007f0351f97d66 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5098 #5 0x00007f0351e5a03f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332 #6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f03335fe700 (LWP 30977)): #0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0351d7b940 in os_event::wait_low(long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158 #2 0x00007f0351d7bcde in os_event_wait_low(os_event*, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507 #3 0x00007f0351e53cd0 in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:940 #4 0x00007f0351e5582b in srv_worker_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2620 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f0343fff700 (LWP 30924)): #0 0x00007f0350da9614 in ?? () from /lib64/libaio.so.1 #1 0x00007f0351d75da6 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936 #2 0x00007f0351d7879e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097 #3 0x00007f0351d78fac in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151 #4 0x00007f0351f97d66 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5098 #5 0x00007f0351e5a03f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332 #6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f0338dfa700 (LWP 30970)): #0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0351d7b940 in os_event::wait_low(long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158 #2 0x00007f0351d7bcde in os_event_wait_low(os_event*, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:507 #3 0x00007f0351e53cd0 in srv_resume_thread(srv_slot_t*, long, bool, unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:940 #4 0x00007f0351e5651c in srv_purge_coordinator_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2781 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f0349d39700 (LWP 30917)): #0 0x00007f0350da9614 in ?? () from /lib64/libaio.so.1 #1 0x00007f0351d75da6 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936 #2 0x00007f0351d7879e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097 #3 0x00007f0351d78fac in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151 #4 0x00007f0351f97d66 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5098 #5 0x00007f0351e5a03f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332 #6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f0349338700 (LWP 30918)): #0 0x00007f0350da9614 in ?? () from /lib64/libaio.so.1 #1 0x00007f0351d75da6 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936 #2 0x00007f0351d7879e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097 #3 0x00007f0351d78fac in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151 #4 0x00007f0351f97d66 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5098 #5 0x00007f0351e5a03f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332 #6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f03397fb700 (LWP 30969)): #0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0351d7bd03 in os_event::timed_wait(timespec const*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275 #2 0x00007f0351d7bf25 in os_event::wait_time_low(unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385 #3 0x00007f0351d7bfa1 in os_event_wait_time_low(os_event*, unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:490 #4 0x00007f0351eb6304 in ib_wqueue_timedwait(ib_wqueue_t*, unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/ut/ut0wqueue.cc:156 #5 0x00007f0351fdee21 in fts_optimize_thread(void*) () at /home/buildbot/buildbot/build/storage/innobase/fts/fts0opt.cc:2820 #6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f033abfd700 (LWP 30967)): #0 0x00007f0350fb8ccd in nanosleep () from /lib64/libpthread.so.0 #1 0x00007f0351d7c1ea in os_thread_sleep(unsigned long) () #2 0x00007f0351e54149 in srv_master_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2422 #3 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f033b5fe700 (LWP 30950)): #0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0351d7bd03 in os_event::timed_wait(timespec const*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275 #2 0x00007f0351d7bf25 in os_event::wait_time_low(unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385 #3 0x00007f0351d7bfa1 in os_event_wait_time_low(os_event*, unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:490 #4 0x00007f0351e5923d in srv_monitor_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:1751 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f03513d8720 (LWP 30833)): #0 0x00007f034ff28613 in poll () from /lib64/libc.so.6 #1 0x00007f03518938d5 in handle_connections_sockets() () at /home/buildbot/buildbot/build/sql/mysqld.cc:6668 #2 0x00007f03518957ab in mysqld_main(int, char**) () at /home/buildbot/buildbot/build/sql/mysqld.cc:6136 #3 0x00007f0351886dce in main () Thread 6 (Thread 0x7f03417fb700 (LWP 30928)): #0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0351d7bd03 in os_event::timed_wait(timespec const*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275 #2 0x00007f0351d7bf25 in os_event::wait_time_low(unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385 #3 0x00007f0351d7bfa1 in os_event_wait_time_low(os_event*, unsigned long, long) () at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:490 #4 0x00007f0351f2c396 in buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0flu.cc:2683 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f034b994700 (LWP 30874)): #0 0x00007f0350fb575b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f03521a6b66 in safe_cond_timedwait () #2 0x00007f03521a721a in timer_handler () #3 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #4 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f03421fc700 (LWP 30927)): #0 0x00007f0350da9614 in ?? () from /lib64/libaio.so.1 #1 0x00007f0351d75da6 in LinuxAIOHandler::collect() () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1936 #2 0x00007f0351d7879e in LinuxAIOHandler::poll(fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2097 #3 0x00007f0351d78fac in os_aio_handler(unsigned long, fil_node_t**, void**, IORequest*) () at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2151 #4 0x00007f0351f97d66 in fil_aio_wait(unsigned long) () at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:5098 #5 0x00007f0351e5a03f in io_handler_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:332 #6 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f034859f700 (LWP 32241)): #0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f03521a6974 in safe_cond_wait () #2 0x00007f0351a8e346 in inline_mysql_cond_wait () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145 #3 0x00007f0351a916fd in handle_rpl_parallel_thread () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:1114 #4 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f03486c7700 (LWP 32240)): #0 0x00007f0350fb53dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f03521a6974 in safe_cond_wait () #2 0x00007f0351a8e346 in inline_mysql_cond_wait () at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1145 #3 0x00007f0351a916fd in handle_rpl_parallel_thread () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:1114 #4 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #5 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f034ff31ccd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f03485e9700 (LWP 32239)): #0 0x00007f0350fb663c in pthread_kill () from /lib64/libpthread.so.0 #1 0x00007f03521a174f in my_write_core () #2 0x00007f0351b17154 in handle_fatal_signal () #3 <signal handler called> #4 0x00007f0351ab8c8c in THD::mark_tmp_tables_as_free_for_reuse() () at /home/buildbot/buildbot/build/sql/sql_plist.h:181 #5 0x00007f03518ead44 in close_thread_tables(THD*) () at /home/buildbot/buildbot/build/sql/sql_base.cc:803 #6 0x00007f035194b0d0 in mysql_execute_command(THD*) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:6044 #7 0x00007f035194c1a8 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:7733 #8 0x00007f0351c29fd7 in Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int) () at /home/buildbot/buildbot/build/sql/log_event.cc:5359 #9 0x00007f0351c2a90a in Query_log_event::do_apply_event(rpl_group_info*) () at /home/buildbot/buildbot/build/sql/log_event.cc:5039 #10 0x00007f03518ab56e in apply_event_and_update_pos_apply(Log_event*, THD*, rpl_group_info*, int) () at /home/buildbot/buildbot/build/sql/log_event.h:1452 #11 0x00007f03518b1af2 in apply_event_and_update_pos_for_parallel(Log_event*, THD*, rpl_group_info*) () at /home/buildbot/buildbot/build/sql/slave.cc:3763 #12 0x00007f0351a8e59c in rpt_handle_event(rpl_parallel_thread::queued_event*, rpl_parallel_thread*) () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:50 #13 0x00007f0351a92943 in handle_rpl_parallel_thread () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:1335 #14 0x00007f03521257f6 in pfs_spawn_thread () at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869 #15 0x00007f0350fb17f1 in start_thread () from /lib64/libpthread.so.0 #16 0x00007f034ff31ccd in clone () from /lib64/libc.so.6

Marko Mäkelä added a comment - 2021-05-05 05:23

There are failures on the cross-reference, with both ASAN and debug builds. Here is the most recent 10.2 failure:

10.2 21a96581fd1dd11c5605dd89e3adbaabdf6de5eb
CURRENT_TEST: rpl.rpl_parallel_temptable
mysqltest: At line 132: failed in 'select master_pos_wait('master-bin.000002', 585, 300, '')': 2013: Lost connection to MySQL server during query
…
2020-08-26 5:04:57 139652077000448 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication starts at GTID position '0-1-1'
2020-08-26 5:04:57 139652100962048 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 508, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-1'
200826 5:04:57 [ERROR] mysqld got signal 11 ;
…
#3 <signal handler called>
#4 0x00007f0351ab8c8c in THD::mark_tmp_tables_as_free_for_reuse() () at /home/buildbot/buildbot/build/sql/sql_plist.h:181
#5 0x00007f03518ead44 in close_thread_tables(THD*) () at /home/buildbot/buildbot/build/sql/sql_base.cc:803
#6 0x00007f035194b0d0 in mysql_execute_command(THD*) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:6044
#7 0x00007f035194c1a8 in mysql_parse(THD, char, unsigned int, Parser_state*, bool, bool) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:7733
#8 0x00007f0351c29fd7 in Query_log_event::do_apply_event(rpl_group_info, char const, unsigned int) () at /home/buildbot/buildbot/build/sql/log_event.cc:5359
#9 0x00007f0351c2a90a in Query_log_event::do_apply_event(rpl_group_info*) () at /home/buildbot/buildbot/build/sql/log_event.cc:5039
#10 0x00007f03518ab56e in apply_event_and_update_pos_apply(Log_event, THD, rpl_group_info*, int) () at /home/buildbot/buildbot/build/sql/log_event.h:1452
#11 0x00007f03518b1af2 in apply_event_and_update_pos_for_parallel(Log_event, THD, rpl_group_info*) () at /home/buildbot/buildbot/build/sql/slave.cc:3763
#12 0x00007f0351a8e59c in rpt_handle_event(rpl_parallel_thread::queued_event, rpl_parallel_thread) () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:50

Starting with MariaDB Server 10.2.2, InnoDB assumes that temporary tables are private to the connection that created the table. In ~~MDEV-25064~~, we have direct evidence that parallel replication is executing CREATE TEMPORARY TABLE in one thread, executing a transaction on the temporary table in another thread, and executing DROP TEMPORARY TABLE in yet another thread (while the transaction that is using the temporary table is still active).

Marko Mäkelä added a comment - 2021-05-05 05:23 There are failures on the cross-reference, with both ASAN and debug builds. Here is the most recent 10.2 failure : 10.2 21a96581fd1dd11c5605dd89e3adbaabdf6de5eb CURRENT_TEST: rpl.rpl_parallel_temptable mysqltest: At line 132: failed in 'select master_pos_wait('master-bin.000002', 585, 300, '')': 2013: Lost connection to MySQL server during query … 2020-08-26 5:04:57 139652077000448 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication starts at GTID position '0-1-1' 2020-08-26 5:04:57 139652100962048 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 508, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-1' 200826 5:04:57 [ERROR] mysqld got signal 11 ; … #3 <signal handler called> #4 0x00007f0351ab8c8c in THD::mark_tmp_tables_as_free_for_reuse() () at /home/buildbot/buildbot/build/sql/sql_plist.h:181 #5 0x00007f03518ead44 in close_thread_tables(THD*) () at /home/buildbot/buildbot/build/sql/sql_base.cc:803 #6 0x00007f035194b0d0 in mysql_execute_command(THD*) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:6044 #7 0x00007f035194c1a8 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) () at /home/buildbot/buildbot/build/sql/sql_parse.cc:7733 #8 0x00007f0351c29fd7 in Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int) () at /home/buildbot/buildbot/build/sql/log_event.cc:5359 #9 0x00007f0351c2a90a in Query_log_event::do_apply_event(rpl_group_info*) () at /home/buildbot/buildbot/build/sql/log_event.cc:5039 #10 0x00007f03518ab56e in apply_event_and_update_pos_apply(Log_event*, THD*, rpl_group_info*, int) () at /home/buildbot/buildbot/build/sql/log_event.h:1452 #11 0x00007f03518b1af2 in apply_event_and_update_pos_for_parallel(Log_event*, THD*, rpl_group_info*) () at /home/buildbot/buildbot/build/sql/slave.cc:3763 #12 0x00007f0351a8e59c in rpt_handle_event(rpl_parallel_thread::queued_event*, rpl_parallel_thread*) () at /home/buildbot/buildbot/build/sql/rpl_parallel.cc:50 Starting with MariaDB Server 10.2.2, InnoDB assumes that temporary tables are private to the connection that created the table. In MDEV-25064 , we have direct evidence that parallel replication is executing CREATE TEMPORARY TABLE in one thread, executing a transaction on the temporary table in another thread, and executing DROP TEMPORARY TABLE in yet another thread (while the transaction that is using the temporary table is still active).

Marko Mäkelä made changes - 2021-05-05 05:23

Link

This issue blocks MDEV-17805 [ MDEV-17805 ]

Marko Mäkelä made changes - 2021-05-05 05:23

Fix Version/s		10.3 [ 22126 ]
Fix Version/s		10.4 [ 22408 ]
Fix Version/s		10.5 [ 23123 ]
Fix Version/s		10.6 [ 24028 ]
Affects Version/s		10.3 [ 22126 ]
Affects Version/s		10.4 [ 22408 ]
Affects Version/s		10.5 [ 23123 ]
Affects Version/s		10.6 [ 24028 ]
Labels		ASAN race temporary-table-replication

Sergei Golubchik made changes - 2021-05-25 12:51

Fix Version/s

10.6 [ 24028 ]

Marko Mäkelä made changes - 2021-07-12 06:00

Link

This issue relates to MDEV-26126 [ MDEV-26126 ]

Marko Mäkelä made changes - 2021-07-15 12:38

Link

This issue relates to MDEV-26152 [ MDEV-26152 ]

Alice Sherepa added a comment - 2021-09-17 13:06

10.3 497b69493620eaa4cb1f2240a10be85e

rpl.rpl_parallel_temptable 'innodb,stmt' w4 [ fail ]

        Test ended at 2021-08-24 17:52:10

CURRENT_TEST: rpl.rpl_parallel_temptable

mysqltest: At line 132: failed in 'select master_pos_wait('master-bin.000002', 585, 300, '')': 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:

< snip >

INSERT INTO t2 VALUES (19);

connection default;

INSERT INTO t3 SELECT a+128 FROM t4;

connection server_1;

INSERT INTO t2 VALUES (20);

connection default;

INSERT INTO t1 SELECT a, a MOD 7 FROM t3;

INSERT INTO t1 SELECT a, a MOD 7 FROM t4;

connection server_1;

INSERT INTO t1 SELECT a, COUNT(*) FROM t2 GROUP BY a;

FLUSH TABLES;

SET SESSION debug_dbug="+d,crash_dispatch_command_before";

SELECT 1;

Got one of the listed errors

connection default;

connection default;

connection server_1;

INSERT INTO t1 VALUES (0, 1);

connection server_2;

include/start_slave.inc

More results from queries before failure can be found in /mnt/buildbot/build/mariadb-10.3.32/mysql-test/var/4/log/rpl_parallel_temptable.log

Server [mysqld.2 - pid: 4673, winpid: 4673, exit: 256] failed during test run

Server log from this test:

----------SERVER LOG START-----------

$ /mnt/buildbot/build/mariadb-10.3.32/sql/mysqld --defaults-group-suffix=.2 --defaults-file=/mnt/buildbot/build/mariadb-10.3.32/mysql-test/var/4/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --log-bin=slave-bin --binlog-format=statement --log-bin=slave-bin --core-file --loose-debug-sync-timeout=300

2021-08-24 17:52:05 0 [Note] /mnt/buildbot/build/mariadb-10.3.32/sql/mysqld (mysqld 10.3.32-MariaDB-debug-log) starting as process 4674 ...

2021-08-24 17:52:05 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32186)

2021-08-24 17:52:05 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)

2021-08-24 17:52:06 0 [Note] Plugin 'partition' is disabled.

2021-08-24 17:52:06 0 [Note] InnoDB: Using Linux native AIO

2021-08-24 17:52:06 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!

2021-08-24 17:52:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2021-08-24 17:52:06 0 [Note] InnoDB: Uses event mutexes

2021-08-24 17:52:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.8

2021-08-24 17:52:06 0 [Note] InnoDB: Number of pools: 1

2021-08-24 17:52:06 0 [Note] InnoDB: Using generic crc32 instructions

2021-08-24 17:52:06 0 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M

2021-08-24 17:52:06 0 [Note] InnoDB: Completed initialization of buffer pool

2021-08-24 17:52:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

2021-08-24 17:52:06 0 [Note] InnoDB: 128 out of 128 rollback segments are active.

2021-08-24 17:52:06 0 [Note] InnoDB: Creating shared tablespace for temporary tables

2021-08-24 17:52:06 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

2021-08-24 17:52:06 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.

2021-08-24 17:52:06 0 [Note] InnoDB: Waiting for purge to start

2021-08-24 17:52:06 0 [Note] InnoDB: 10.3.32 started; log sequence number 1636384; transaction id 33

2021-08-24 17:52:06 0 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var_auto_iuHk/4/mysqld.2/data/ib_buffer_pool

2021-08-24 17:52:06 0 [Note] Plugin 'SEQUENCE' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_CMP' is disabled.

2021-08-24 17:52:06 0 [Note] InnoDB: Buffer pool(s) load completed at 210824 17:52:06

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'FEEDBACK' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'user_variables' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_MUTEXES' is disabled.

2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.

2021-08-24 17:52:06 0 [Warning] /mnt/buildbot/build/mariadb-10.3.32/sql/mysqld: unknown option '--loose-pam-debug'

2021-08-24 17:52:06 0 [Note] Server socket created on IP: '127.0.0.1'.

2021-08-24 17:52:06 0 [Note] Reading of all Master_info entries succeeded

2021-08-24 17:52:06 0 [Note] Added new Master_info '' to hash table

2021-08-24 17:52:06 0 [Note] /mnt/buildbot/build/mariadb-10.3.32/sql/mysqld: ready for connections.

Version: '10.3.32-MariaDB-debug-log'  socket: '/mnt/buildbot/build/mariadb-10.3.32/mysql-test/var/tmp/4/mysqld.2.sock'  port: 16061  Source distribution

2021-08-24 17:52:07 10 [Note] Deleted Master_info file '/mnt/buildbot/build/mariadb-10.3.32/mysql-test/var/4/mysqld.2/data/master.info'.

2021-08-24 17:52:07 10 [Note] Deleted Master_info file '/mnt/buildbot/build/mariadb-10.3.32/mysql-test/var/4/mysqld.2/data/relay-log.info'.

2021-08-24 17:52:07 10 [Note] Master connection name: ''  Master_info_file: 'master.info'  Relay_info_file: 'relay-log.info'

2021-08-24 17:52:07 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='4'.

2021-08-24 17:52:07 12 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16060' in log 'master-bin.000001' at position 4

2021-08-24 17:52:07 13 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 4, relay log './slave-relay-bin.000001' position: 4

2021-08-24 17:52:07 12 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication started in log 'master-bin.000001' at position 4

2021-08-24 17:52:07 13 [Note] Error reading relay log event: slave SQL thread was killed

2021-08-24 17:52:07 13 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329

2021-08-24 17:52:07 13 [Note] master was 127.0.0.1:16060

2021-08-24 17:52:07 12 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 329

2021-08-24 17:52:07 12 [Note] master was 127.0.0.1:16060

2021-08-24 17:52:07 10 [Note] Master connection name: ''  Master_info_file: 'master.info'  Relay_info_file: 'relay-log.info'

2021-08-24 17:52:07 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='329'. New state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='329'.

2021-08-24 17:52:07 10 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos

2021-08-24 17:52:07 14 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16060' in log 'master-bin.000001' at position 329

2021-08-24 17:52:07 14 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication starts at GTID position ''

2021-08-24 17:52:07 15 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 329, relay log './slave-relay-bin.000001' position: 4; GTID position ''

2021-08-24 17:52:07 15 [Note] Error reading relay log event: slave SQL thread was killed

2021-08-24 17:52:07 15 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 508; GTID position '0-1-1'

2021-08-24 17:52:07 15 [Note] master was 127.0.0.1:16060

2021-08-24 17:52:07 14 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 508; GTID position 0-1-1

2021-08-24 17:52:07 14 [Note] master was 127.0.0.1:16060

2021-08-24 17:52:08 21 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16060' in log 'master-bin.000001' at position 508

2021-08-24 17:52:08 21 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication starts at GTID position '0-1-1'

2021-08-24 17:52:08 22 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 508, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-1'

210824 17:52:08 [ERROR] mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.3.32-MariaDB-debug-log

key_buffer_size=1048576

read_buffer_size=131072

max_used_connections=2

max_threads=153

thread_count=15

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62003 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0xa6f77dd0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0xa4cfb1e4 thread_stack 0x49000

mysys/stacktrace.c:174(my_print_stacktrace)[0x810672ac]

sql/signal_handler.cc:219(handle_fatal_signal)[0x8080d1c0]

addr2line: '': No such file

[0xb77b5c14]

sql/sql_plist.h:179(I_P_List_iterator<TMP_TABLE_SHARE, I_P_List<TMP_TABLE_SHARE, All_tmp_table_shares, I_P_List_null_counter, I_P_List_no_push_back<TMP_TABLE_SHARE> > >::I_P_List_iterator(I_P_List<TMP_TABLE_SHARE, All_tmp_table_shares, I_P_List_null_counter, I_P_List_no_push_back<TMP_TABLE_SHARE> > const&))[0x80735570]

sql/temporary_tables.cc:749(THD::mark_tmp_tables_as_free_for_reuse())[0x80733158]

sql/sql_base.cc:806(close_thread_tables(THD*))[0x80464f0c]

sql/sql_parse.cc:6160(mysql_execute_command(THD*))[0x80504e35]

sql/sql_parse.cc:7870(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x80509856]

sql/log_event.cc:5700(Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int))[0x8095216d]

sql/log_event.cc:5377(Query_log_event::do_apply_event(rpl_group_info*))[0x80950e1e]

sql/log_event.h:1492(Log_event::apply_event(rpl_group_info*))[0x80416c87]

sql/slave.cc:3789(apply_event_and_update_pos_apply(Log_event*, THD*, rpl_group_info*, int))[0x80409907]

sql/slave.cc:3981(apply_event_and_update_pos_for_parallel(Log_event*, THD*, rpl_group_info*))[0x8040a0bd]

sql/rpl_parallel.cc:51(rpt_handle_event(rpl_parallel_thread::queued_event*, rpl_parallel_thread*))[0x806d2e38]

sql/rpl_parallel.cc:1335(handle_rpl_parallel_thread)[0x806d633a]

perfschema/pfs.cc:1869(pfs_spawn_thread)[0x80a49677]

/lib/i386-linux-gnu/libpthread.so.0(+0x62b5)[0xb72242b5]

/lib/i386-linux-gnu/libc.so.6(clone+0x6e)[0xb714f16e]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0xb0f2e6fb): COMMIT

Connection ID (thread ID): 26

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains

information that should help you find out what is causing the crash.

Writing a core file...

Working directory at /dev/shm/var_auto_iuHk/4/mysqld.2/data

Resource Limits:

Limit                     Soft Limit           Hard Limit           Units

Max cpu time              unlimited            unlimited            seconds

Max file size             unlimited            unlimited            bytes

Max data size             unlimited            unlimited            bytes

Max stack size            8388608              unlimited            bytes

Max core file size        unlimited            unlimited            bytes

Max resident set          unlimited            unlimited            bytes

Max processes             64511                64511                processes

Max open files            1024                 1024                 files

Max locked memory         65536                65536                bytes

Max address space         unlimited            unlimited            bytes

Max file locks            unlimited            unlimited            locks

Max pending signals       64511                64511                signals

Max msgqueue size         819200               819200               bytes

Max nice priority         0                    0

Max realtime priority     0                    0

Max realtime timeout      unlimited            unlimited            us

Core pattern: |/usr/share/apport/apport %p %s %c %P

----------SERVER LOG END-------------

 - found 'core' (0/0)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace from coredump /mnt/buildbot/build/mariadb-10.3.32/mysql-test/var/4/log/rpl.rpl_parallel_temptable-innodb,stmt/mysqld.2/data/core

Core generated by '/mnt/buildbot/build/mariadb-10.3.32/sql/mysqld'

Output from gdb follows. The first stack trace is from the failing thread.

The following stack traces are from all threads (so the failing one is

duplicated).

--------------------------

[New LWP 4869]

[New LWP 4689]

[New LWP 4688]

[New LWP 4867]

[New LWP 4681]

[New LWP 4685]

[New LWP 4687]

[New LWP 4680]

[New LWP 4694]

[New LWP 4686]

[New LWP 4674]

[New LWP 4679]

[New LWP 4692]

[New LWP 4864]

[New LWP 4678]

[New LWP 4691]

[New LWP 4865]

[New LWP 4677]

[New LWP 4696]

[New LWP 4868]

[New LWP 4676]

[New LWP 4697]

[New LWP 4870]

[New LWP 4698]

[New LWP 4675]

[New LWP 4871]

[New LWP 4700]

[New LWP 4683]

[New LWP 4706]

[New LWP 4682]

[New LWP 4719]

[New LWP 4701]

[New LWP 4699]

[New LWP 4695]

[New LWP 4693]

[New LWP 4690]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".

Core was generated by `/mnt/buildbot/build/mariadb-10.3.32/sql/mysqld --defaults-group-suffix=.2 --def'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0  0xb77b5c31 in __kernel_vsyscall ()

[Current thread is 1 (Thread 0xa4cfbb40 (LWP 4869))]

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb722ace1 in pthread_kill () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x810673a6 in my_write_core (sig=11) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/stacktrace.c:386

#3  0x8080d570 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/signal_handler.cc:355

#4  <signal handler called>

#5  0x80735570 in I_P_List_iterator<TMP_TABLE_SHARE, I_P_List<TMP_TABLE_SHARE, All_tmp_table_shares, I_P_List_null_counter, I_P_List_no_push_back<TMP_TABLE_SHARE> > >::I_P_List_iterator (this=0xa4cfa84c, a=...) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_plist.h:179

#6  0x80733158 in THD::mark_tmp_tables_as_free_for_reuse (this=0xa6f77dd0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/temporary_tables.cc:749

#7  0x80464f0c in close_thread_tables (thd=0xa6f77dd0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_base.cc:806

#8  0x80504e35 in mysql_execute_command (thd=0xa6f77dd0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:6160

#9  0x80509856 in mysql_parse (thd=0xa6f77dd0, rawbuf=0xb0f2e6fb "COMMIT", length=6, parser_state=0xa4cfafa0, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:7870

#10 0x8095216d in Query_log_event::do_apply_event (this=0xb0f21788, rgi=0xb0f23bb0, query_arg=0xb0f2e6fb "COMMIT", q_len_arg=6) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.cc:5700

#11 0x80950e1e in Query_log_event::do_apply_event (this=0xb0f21788, rgi=0xb0f23bb0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.cc:5377

#12 0x80416c87 in Log_event::apply_event (this=0xb0f21788, rgi=0xb0f23bb0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.h:1492

#13 0x80409907 in apply_event_and_update_pos_apply (ev=0xb0f21788, thd=0xa6f77dd0, rgi=0xb0f23bb0, reason=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:3789

#14 0x8040a0bd in apply_event_and_update_pos_for_parallel (ev=0xb0f21788, thd=0xa6f77dd0, rgi=0xb0f23bb0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:3981

#15 0x806d2e38 in rpt_handle_event (qev=0xb0f24558, rpt=0xb0f0dd50) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:51

#16 0x806d633a in handle_rpl_parallel_thread (arg=0xb0f0dd50) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:1335

#17 0x80a49677 in pfs_spawn_thread (arg=0xb0f0e618) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#18 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#19 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 36 (Thread 0xac2fab40 (LWP 4690)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb722d8aa in nanosleep () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80c439c6 in os_thread_sleep (tm=1000000) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0thread.cc:231

#3  0x80d68013 in srv_master_sleep () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:2351

#4  0x80d68190 in srv_master_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:2391

#5  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#6  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 35 (Thread 0xa80fdb40 (LWP 4693)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80c435c6 in os_event::wait (this=0x839a5050) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:158

#3  0x80c43003 in os_event::wait_low (this=0x839a5050, reset_sig_count=5) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:325

#4  0x80c4336e in os_event_wait_low (event=0x839a5050, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:502

#5  0x80d643b2 in srv_resume_thread (slot=0x81928c30 <srv_sys+432>, sig_count=0, wait=true, timeout_usec=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:916

#6  0x80d689e5 in srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:2538

#7  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#8  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 34 (Thread 0xa6cffb40 (LWP 4695)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80c435c6 in os_event::wait (this=0x839a5230) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:158

#3  0x80c43003 in os_event::wait_low (this=0x839a5230, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:325

#4  0x80c4336e in os_event_wait_low (event=0x839a5230, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:502

#5  0x80e717cb in buf_dump_thread () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/buf/buf0dump.cc:834

#6  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 33 (Thread 0xb00cab40 (LWP 4699)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x8106c03b in safe_cond_wait (cond=0x81af21f4 <mysql_bin_log+3316>, mp=0x81af2190 <mysql_bin_log+3216>, file=0x812dea84 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1174) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:492

#3  0x809241ef in inline_mysql_cond_wait (that=0x81af21f4 <mysql_bin_log+3316>, mutex=0x81af2190 <mysql_bin_log+3216>, src_file=0x812df384 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/log.cc", src_line=9970) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1174

#4  0x80940abd in binlog_background_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log.cc:9970

#5  0x80a49677 in pfs_spawn_thread (arg=0x839058a0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#6  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 32 (Thread 0xac6c3b40 (LWP 4701)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x8106c03b in safe_cond_wait (cond=0x81ad3f80 <COND_manager>, mp=0x81ad3f00 <LOCK_manager>, file=0x811285c0 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1174) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:492

#3  0x804efa96 in inline_mysql_cond_wait (that=0x81ad3f80 <COND_manager>, mutex=0x81ad3f00 <LOCK_manager>, src_file=0x811286d0 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_manager.cc", src_line=102) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1174

#4  0x804eff81 in handle_manager (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_manager.cc:102

#5  0x80a49677 in pfs_spawn_thread (arg=0x83b8e960) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#6  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 31 (Thread 0xa92f4b40 (LWP 4719)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7144f8f in poll () from /lib/i386-linux-gnu/libc.so.6

#2  0x810c15ec in vio_io_wait (vio=0x83bd3900, event=VIO_IO_EVENT_READ, timeout=28800000) at /home/buildbot/buildbot/build/mariadb-10.3.32/vio/viosocket.c:962

#3  0x810bfdb1 in vio_socket_io_wait (vio=0x83bd3900, event=VIO_IO_EVENT_READ) at /home/buildbot/buildbot/build/mariadb-10.3.32/vio/viosocket.c:119

#4  0x810bffa1 in vio_read (vio=0x83bd3900, buf=0xa910b928 "\001", size=4) at /home/buildbot/buildbot/build/mariadb-10.3.32/vio/viosocket.c:196

#5  0x803e29eb in my_real_read (net=0xa91063b0, complen=0xa92f4118, header=1 '\001') at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/net_serv.cc:975

#6  0x803e3236 in my_net_read_packet_reallen (net=0xa91063b0, read_from_server=1 '\001', reallen=0xa92f418c) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/net_serv.cc:1256

#7  0x803e31e8 in my_net_read_packet (net=0xa91063b0, read_from_server=1 '\001') at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/net_serv.cc:1240

#8  0x804f35b3 in do_command (thd=0xa91061e8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:1274

#9  0x8065e7c2 in do_handle_one_connection (connect=0x83b9b478) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_connect.cc:1403

#10 0x8065e4f7 in handle_one_connection (arg=0x83b9b478) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_connect.cc:1308

#11 0x80a49677 in pfs_spawn_thread (arg=0x83bd3ac8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#12 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#13 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 30 (Thread 0xad6c5b40 (LWP 4682)):

#0  0xb7781570 in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb77815e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0x80c335e5 in LinuxAIOHandler::collect (this=0xad6c50f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:1936

#3  0x80c33cf1 in LinuxAIOHandler::poll (this=0xad6c50f0, m1=0xad6c5154, m2=0xad6c5158, request=0xad6c5178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2097

#4  0x80c33e55 in os_aio_linux_handler (global_segment=5, m1=0xad6c5154, m2=0xad6c5158, request=0xad6c5178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2151

#5  0x80c39502 in os_aio_handler (segment=5, m1=0xad6c5154, m2=0xad6c5158, request=0xad6c5178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:5761

#6  0x80f08eb0 in fil_aio_wait (segment=5) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/fil/fil0fil.cc:4465

#7  0x80d6c00c in io_handler_thread (arg=0x8232f2b4 <n+20>) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0start.cc:326

#8  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#9  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 29 (Thread 0xac679b40 (LWP 4706)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb722cd32 in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread.so.0

#2  0xb72269e5 in pthread_mutex_lock () from /lib/i386-linux-gnu/libpthread.so.0

#3  0x8106b88e in safe_mutex_lock (mp=0x83bd07d4, my_flags=0, file=0x8115f0e8 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_rli.cc", line=815) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:293

#4  0x806487b7 in inline_mysql_mutex_lock (that=0x83bd07d4, src_file=0x8115f0e8 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_rli.cc", src_line=815) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:702

#5  0x8064b9b7 in Relay_log_info::wait_for_pos (this=0x83bced98, thd=0xa6f06f68, log_name=0xa6f15a20, log_pos=585, timeout=300) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_rli.cc:815

#6  0x808a00b3 in Item_master_pos_wait::val_int (this=0xa6f15c90) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/item_func.cc:3760

#7  0x806ed3ee in Type_handler::Item_send_longlong (this=0x81aedbfc <type_handler_longlong>, item=0xa6f15c90, protocol=0xa6f073b0, buf=0xac677a5c) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_type.cc:5450

#8  0x806f2b89 in Type_handler_longlong::Item_send (this=0x81aedbfc <type_handler_longlong>, item=0xa6f15c90, protocol=0xa6f073b0, buf=0xac677a5c) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_type.h:2498

#9  0x803f0141 in Item::send (this=0xa6f15c90, protocol=0xa6f073b0, buffer=0xac677a5c) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/item.h:884

#10 0x803eaa10 in Protocol::send_result_set_row (this=0xa6f073b0, row_items=0xa6f09c6c) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/protocol.cc:999

#11 0x8049466e in select_send::send_data (this=0xa6f15e50, items=...) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_class.cc:2999

#12 0x80548071 in JOIN::exec_inner (this=0xa6f15e68) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_select.cc:4017

#13 0x80547a57 in JOIN::exec (this=0xa6f15e68) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_select.cc:3936

#14 0x80548e44 in mysql_select (thd=0xa6f06f68, tables=0x0, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0xa6f15e50, unit=0xa6f096a0, select_lex=0xa6f09bc8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_select.cc:4344

#15 0x8053b7f3 in handle_select (thd=0xa6f06f68, lex=0xa6f09630, result=0xa6f15e50, setup_tables_done_option=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_select.cc:384

#16 0x80505b75 in execute_sqlcom_select (thd=0xa6f06f68, all_tables=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:6339

#17 0x804fb690 in mysql_execute_command (thd=0xa6f06f68) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:3870

#18 0x80509856 in mysql_parse (thd=0xa6f06f68, rawbuf=0xa6f15908 "select master_pos_wait('master-bin.000002', 585, 300, '')", length=57, parser_state=0xac678b98, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:7870

#19 0x804f53d4 in dispatch_command (command=COM_QUERY, thd=0xa6f06f68, packet=0xa6f0c649 "", packet_length=57, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:1853

#20 0x804f3b7a in do_command (thd=0xa6f06f68) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:1399

#21 0x8065e7c2 in do_handle_one_connection (connect=0x83b8eca8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_connect.cc:1403

#22 0x8065e4f7 in handle_one_connection (arg=0x83b8eca8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_connect.cc:1308

#23 0x80a49677 in pfs_spawn_thread (arg=0x83bd3900) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#24 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#25 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 28 (Thread 0xacec4b40 (LWP 4683)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80c42e30 in os_event::timed_wait (this=0x839a5290, abstime=0xacec4078) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:275

#3  0x80c43141 in os_event::wait_time_low (this=0x839a5290, time_in_usec=935000, reset_sig_count=4) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:385

#4  0x80c4333c in os_event_wait_time_low (event=0x839a5290, time_in_usec=935000, reset_sig_count=4) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:485

#5  0x80e7ec4e in pc_sleep_if_needed (next_loop_time=1370734, sig_count=4, cur_time=1369799) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/buf/buf0flu.cc:2665

#6  0x80e7fe67 in buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/buf/buf0flu.cc:3124

#7  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#8  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 27 (Thread 0xb0080b40 (LWP 4700)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb722e1cd in do_sigwait () from /lib/i386-linux-gnu/libpthread.so.0

#2  0xb722e25e in sigwait () from /lib/i386-linux-gnu/libpthread.so.0

#3  0x803d04e1 in signal_hand (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/mysqld.cc:3590

#4  0x80a49677 in pfs_spawn_thread (arg=0x83b39ea8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#5  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#6  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 26 (Thread 0xa4cb1b40 (LWP 4871)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x8106c03b in safe_cond_wait (cond=0xb0f0e080, mp=0xb0f0e01c, file=0x811842e0 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1174) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:492

#3  0x806d29b9 in inline_mysql_cond_wait (that=0xb0f0e080, mutex=0xb0f0e01c, src_file=0x81184658 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc", src_line=1114) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1174

#4  0x806d5b7a in handle_rpl_parallel_thread (arg=0xb0f0e018) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:1114

#5  0x80a49677 in pfs_spawn_thread (arg=0xb0f0ea68) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#6  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 25 (Thread 0xb2c8fb40 (LWP 4675)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x8106c2e6 in safe_cond_timedwait (cond=0x823438c0 <COND_timer>, mp=0x82343840 <LOCK_timer>, abstime=0xb2c8f250, file=0x814c55f4 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1211) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:546

#3  0x8106d298 in inline_mysql_cond_timedwait (that=0x823438c0 <COND_timer>, mutex=0x82343840 <LOCK_timer>, abstime=0xb2c8f250, src_file=0x814c5644 "/home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_timer.c", src_line=292) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1211

#4  0x8106de77 in timer_handler (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_timer.c:292

#5  0x80a49677 in pfs_spawn_thread (arg=0x838ce578) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#6  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 24 (Thread 0xa54fcb40 (LWP 4698)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x8106c03b in safe_cond_wait (cond=0xa54fc1c4, mp=0xa54fc1f8, file=0x81344960 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1185) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:492

#3  0x80b3f3ed in inline_mysql_cond_wait (that=0xa54fc1c4, mutex=0xa54fc1f8, src_file=0x81346818 "/home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/handler/ha_innodb.cc", src_line=324) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1185

#4  0x80b45223 in thd_destructor_proxy () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/handler/ha_innodb.cc:324

#5  0x80a49677 in pfs_spawn_thread (arg=0x83b1ac58) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#6  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 23 (Thread 0xa70fbb40 (LWP 4870)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x8106c03b in safe_cond_wait (cond=0xb0f0df1c, mp=0xb0f0deb8, file=0x811842e0 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1174) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:492

#3  0x806d29b9 in inline_mysql_cond_wait (that=0xb0f0df1c, mutex=0xb0f0deb8, src_file=0x81184658 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc", src_line=1114) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1174

#4  0x806d5b7a in handle_rpl_parallel_thread (arg=0xb0f0deb4) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:1114

#5  0x80a49677 in pfs_spawn_thread (arg=0xb0f0e840) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#6  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 22 (Thread 0xa5cfdb40 (LWP 4697)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80c435c6 in os_event::wait (this=0x839a52f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:158

#3  0x80c43003 in os_event::wait_low (this=0x839a52f0, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:325

#4  0x80c4336e in os_event_wait_low (event=0x839a52f0, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:502

#5  0x80e57bdf in buf_resize_thread () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/buf/buf0buf.cc:3170

#6  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 21 (Thread 0xb1861b40 (LWP 4676)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x8106c2e6 in safe_cond_timedwait (cond=0x822fa240 <COND_checkpoint>, mp=0x822fa1c0 <LOCK_checkpoint>, abstime=0xb186119c, file=0x81336b18 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1211) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:546

#3  0x80ab0a75 in inline_mysql_cond_timedwait (that=0x822fa240 <COND_checkpoint>, mutex=0x822fa1c0 <LOCK_checkpoint>, abstime=0xb186119c, src_file=0x81336b68 "/home/buildbot/buildbot/build/mariadb-10.3.32/storage/maria/ma_servicethread.c", src_line=116) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1211

#4  0x80ab0fda in my_service_thread_sleep (control=0x81920038 <checkpoint_control>, sleep_time=29000000000) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/maria/ma_servicethread.c:115

#5  0x80aa486d in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/maria/ma_checkpoint.c:707

#6  0x80a49677 in pfs_spawn_thread (arg=0x839097a8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#7  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#8  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 20 (Thread 0xa7067b40 (LWP 4868)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x8106c03b in safe_cond_wait (cond=0xb0f0dc54, mp=0xb0f0dbf0, file=0x811842e0 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1174) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:492

#3  0x806d29b9 in inline_mysql_cond_wait (that=0xb0f0dc54, mutex=0xb0f0dbf0, src_file=0x81184658 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc", src_line=1114) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1174

#4  0x806d5b7a in handle_rpl_parallel_thread (arg=0xb0f0dbec) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:1114

#5  0x80a49677 in pfs_spawn_thread (arg=0xb0f0e3f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#6  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 19 (Thread 0xa64feb40 (LWP 4696)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb722d8aa in nanosleep () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80c439c6 in os_thread_sleep (tm=1000000) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0thread.cc:231

#3  0x80e45fa8 in btr_defragment_thread () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/btr/btr0defragment.cc:707

#4  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#5  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 18 (Thread 0xafecab40 (LWP 4677)):

#0  0xb7781570 in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb77815e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0x80c335e5 in LinuxAIOHandler::collect (this=0xafeca0f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:1936

#3  0x80c33cf1 in LinuxAIOHandler::poll (this=0xafeca0f0, m1=0xafeca154, m2=0xafeca158, request=0xafeca178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2097

#4  0x80c33e55 in os_aio_linux_handler (global_segment=0, m1=0xafeca154, m2=0xafeca158, request=0xafeca178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2151

#5  0x80c39502 in os_aio_handler (segment=0, m1=0xafeca154, m2=0xafeca158, request=0xafeca178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:5761

#6  0x80f08eb0 in fil_aio_wait (segment=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/fil/fil0fil.cc:4465

#7  0x80d6c00c in io_handler_thread (arg=0x8232f2a0 <n>) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0start.cc:326

#8  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#9  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 17 (Thread 0xa70b1b40 (LWP 4865)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x8106c03b in safe_cond_wait (cond=0x83bcf7c0, mp=0x83bcf218, file=0x812dea84 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1174) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:492

#3  0x809241ef in inline_mysql_cond_wait (that=0x83bcf7c0, mutex=0x83bcf218, src_file=0x812df384 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/log.cc", src_line=8409) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1174

#4  0x8093c5fe in MYSQL_BIN_LOG::wait_for_update_relay_log (this=0x83bcf214, thd=0xb0f082c8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log.cc:8409

#5  0x804144c3 in next_event (rgi=0xb0f07a28, event_size=0xa70b0b60) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:7574

#6  0x8040a12a in exec_relay_log_event (thd=0xb0f082c8, rli=0x83bced98, serial_rgi=0xb0f07a28) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:4102

#7  0x8040e31d in handle_slave_sql (arg=0x83bcd328) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:5429

#8  0x80a49677 in pfs_spawn_thread (arg=0xa6f758e8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#9  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#10 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 16 (Thread 0xa90ffb40 (LWP 4691)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80c435c6 in os_event::wait (this=0x839a4ff0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:158

#3  0x80c43003 in os_event::wait_low (this=0x839a4ff0, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:325

#4  0x80c4336e in os_event_wait_low (event=0x839a4ff0, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:502

#5  0x80d643b2 in srv_resume_thread (slot=0x81928ba8 <srv_sys+296>, sig_count=1, wait=true, timeout_usec=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:916

#6  0x80d68ed6 in srv_purge_coordinator_suspend (slot=0x81928ba8 <srv_sys+296>, rseg_history_len=15) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:2686

#7  0x80d69210 in srv_purge_coordinator_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:2767

#8  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#9  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 15 (Thread 0xaf6c9b40 (LWP 4678)):

#0  0xb7781570 in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb77815e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0x80c335e5 in LinuxAIOHandler::collect (this=0xaf6c90f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:1936

#3  0x80c33cf1 in LinuxAIOHandler::poll (this=0xaf6c90f0, m1=0xaf6c9154, m2=0xaf6c9158, request=0xaf6c9178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2097

#4  0x80c33e55 in os_aio_linux_handler (global_segment=1, m1=0xaf6c9154, m2=0xaf6c9158, request=0xaf6c9178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2151

#5  0x80c39502 in os_aio_handler (segment=1, m1=0xaf6c9154, m2=0xaf6c9158, request=0xaf6c9178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:5761

#6  0x80f08eb0 in fil_aio_wait (segment=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/fil/fil0fil.cc:4465

#7  0x80d6c00c in io_handler_thread (arg=0x8232f2a4 <n+4>) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0start.cc:326

#8  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#9  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 14 (Thread 0xa9260b40 (LWP 4864)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7144f8f in poll () from /lib/i386-linux-gnu/libc.so.6

#2  0x810c15ec in vio_io_wait (vio=0xac5087a8, event=VIO_IO_EVENT_READ, timeout=120000) at /home/buildbot/buildbot/build/mariadb-10.3.32/vio/viosocket.c:962

#3  0x810bfdb1 in vio_socket_io_wait (vio=0xac5087a8, event=VIO_IO_EVENT_READ) at /home/buildbot/buildbot/build/mariadb-10.3.32/vio/viosocket.c:119

#4  0x810bffa1 in vio_read (vio=0xac5087a8, buf=0xac508970 "1", size=16384) at /home/buildbot/buildbot/build/mariadb-10.3.32/vio/viosocket.c:196

#5  0x810c019d in vio_read_buff (vio=0xac5087a8, buf=0xac50c9b8 "", size=4) at /home/buildbot/buildbot/build/mariadb-10.3.32/vio/viosocket.c:244

#6  0x803e29eb in my_real_read (net=0xac508390, complen=0xa9260028, header=0 '\000') at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/net_serv.cc:975

#7  0x803e3236 in my_net_read_packet_reallen (net=0xac508390, read_from_server=0 '\000', reallen=0xa92601a4) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/net_serv.cc:1256

#8  0x807bd2df in cli_safe_read_reallen (mysql=0xac508390, reallen=0xa92601a4) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql-common/client.c:594

#9  0x80408f25 in read_event (mysql=0xac508390, mi=0x83bcd328, suppress_warnings=0xa926014a, network_read_len=0xa92601a4) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:3564

#10 0x8040be61 in handle_slave_io (arg=0x83bcd328) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:4711

#11 0x80a49677 in pfs_spawn_thread (arg=0xa6f758e8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#12 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#13 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 13 (Thread 0xa88feb40 (LWP 4692)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80c435c6 in os_event::wait (this=0x839a5110) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:158

#3  0x80c43003 in os_event::wait_low (this=0x839a5110, reset_sig_count=4) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:325

#4  0x80c4336e in os_event_wait_low (event=0x839a5110, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:502

#5  0x80d643b2 in srv_resume_thread (slot=0x81928d40 <srv_sys+704>, sig_count=0, wait=true, timeout_usec=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:916

#6  0x80d689e5 in srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:2538

#7  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#8  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 12 (Thread 0xaeec8b40 (LWP 4679)):

#0  0xb7781570 in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb77815e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0x80c335e5 in LinuxAIOHandler::collect (this=0xaeec80f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:1936

#3  0x80c33cf1 in LinuxAIOHandler::poll (this=0xaeec80f0, m1=0xaeec8154, m2=0xaeec8158, request=0xaeec8178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2097

#4  0x80c33e55 in os_aio_linux_handler (global_segment=2, m1=0xaeec8154, m2=0xaeec8158, request=0xaeec8178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2151

#5  0x80c39502 in os_aio_handler (segment=2, m1=0xaeec8154, m2=0xaeec8158, request=0xaeec8178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:5761

#6  0x80f08eb0 in fil_aio_wait (segment=2) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/fil/fil0fil.cc:4465

#7  0x80d6c00c in io_handler_thread (arg=0x8232f2a8 <n+8>) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0start.cc:326

#8  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#9  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 11 (Thread 0xb6ed5a00 (LWP 4674)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7144f8f in poll () from /lib/i386-linux-gnu/libc.so.6

#2  0x803d71c5 in handle_connections_sockets () at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/mysqld.cc:6815

#3  0x803d64b7 in mysqld_main (argc=158, argv=0x837a8ad0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/mysqld.cc:6286

#4  0x803c9b3b in main (argc=26, argv=0xbf96a5f4) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/main.cc:25

Thread 10 (Thread 0xab2f8b40 (LWP 4686)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80c42e30 in os_event::timed_wait (this=0x839a5170, abstime=0xab2f8118) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:275

#3  0x80c43141 in os_event::wait_time_low (this=0x839a5170, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:385

#4  0x80c4333c in os_event_wait_time_low (event=0x839a5170, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:485

#5  0x80d6690b in srv_error_monitor_thread () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:1861

#6  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 9 (Thread 0xa78fcb40 (LWP 4694)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80c435c6 in os_event::wait (this=0x839a50b0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:158

#3  0x80c43003 in os_event::wait_low (this=0x839a50b0, reset_sig_count=4) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:325

#4  0x80c4336e in os_event_wait_low (event=0x839a50b0, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:502

#5  0x80d643b2 in srv_resume_thread (slot=0x81928cb8 <srv_sys+568>, sig_count=0, wait=true, timeout_usec=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:916

#6  0x80d689e5 in srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:2538

#7  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#8  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 8 (Thread 0xae6c7b40 (LWP 4680)):

#0  0xb7781570 in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb77815e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0x80c335e5 in LinuxAIOHandler::collect (this=0xae6c70f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:1936

#3  0x80c33cf1 in LinuxAIOHandler::poll (this=0xae6c70f0, m1=0xae6c7154, m2=0xae6c7158, request=0xae6c7178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2097

#4  0x80c33e55 in os_aio_linux_handler (global_segment=3, m1=0xae6c7154, m2=0xae6c7158, request=0xae6c7178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2151

#5  0x80c39502 in os_aio_handler (segment=3, m1=0xae6c7154, m2=0xae6c7158, request=0xae6c7178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:5761

#6  0x80f08eb0 in fil_aio_wait (segment=3) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/fil/fil0fil.cc:4465

#7  0x80d6c00c in io_handler_thread (arg=0x8232f2ac <n+12>) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0start.cc:326

#8  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#9  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 7 (Thread 0xaaaf7b40 (LWP 4687)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80c42e30 in os_event::timed_wait (this=0x839a51d0, abstime=0xaaaf7208) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:275

#3  0x80c43141 in os_event::wait_time_low (this=0x839a51d0, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:385

#4  0x80c4333c in os_event_wait_time_low (event=0x839a51d0, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:485

#5  0x80d6634b in srv_monitor_thread () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:1697

#6  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 6 (Thread 0xabaf9b40 (LWP 4685)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80c42e30 in os_event::timed_wait (this=0x83abe8e0, abstime=0xabaf9218) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:275

#3  0x80c43141 in os_event::wait_time_low (this=0x83abe8e0, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:385

#4  0x80c4333c in os_event_wait_time_low (event=0x83abe8e0, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:485

#5  0x80c01ccb in lock_wait_timeout_thread () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/lock/lock0wait.cc:527

#6  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 5 (Thread 0xadec6b40 (LWP 4681)):

#0  0xb7781570 in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb77815e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0x80c335e5 in LinuxAIOHandler::collect (this=0xadec60f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:1936

#3  0x80c33cf1 in LinuxAIOHandler::poll (this=0xadec60f0, m1=0xadec6154, m2=0xadec6158, request=0xadec6178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2097

#4  0x80c33e55 in os_aio_linux_handler (global_segment=4, m1=0xadec6154, m2=0xadec6158, request=0xadec6178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2151

#5  0x80c39502 in os_aio_handler (segment=4, m1=0xadec6154, m2=0xadec6158, request=0xadec6178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:5761

#6  0x80f08eb0 in fil_aio_wait (segment=4) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/fil/fil0fil.cc:4465

#7  0x80d6c00c in io_handler_thread (arg=0x8232f2b0 <n+16>) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0start.cc:326

#8  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#9  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 4 (Thread 0xa92aab40 (LWP 4867)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb722cd32 in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread.so.0

#2  0xb72269e5 in pthread_mutex_lock () from /lib/i386-linux-gnu/libpthread.so.0

#3  0x8106b88e in safe_mutex_lock (mp=0x83bd07d4, my_flags=0, file=0x8115f0e8 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_rli.cc", line=988) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:293

#4  0x806487b7 in inline_mysql_mutex_lock (that=0x83bd07d4, src_file=0x8115f0e8 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_rli.cc", src_line=988) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:702

#5  0x8064c1f8 in Relay_log_info::inc_group_relay_log_pos (this=0x83bced98, log_pos=585, rgi=0xb0f2d4b8, skip_lock=false) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_rli.cc:988

#6  0x8064d726 in Relay_log_info::stmt_done (this=0x83bced98, event_master_log_pos=585, thd=0xb231eee0, rgi=0xb0f2d4b8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_rli.cc:1459

#7  0x8094ab23 in Log_event::do_update_pos (this=0xb0f0e498, rgi=0xb0f2d4b8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.cc:1433

#8  0x80416ce7 in Log_event::update_pos (this=0xb0f0e498, rgi=0xb0f2d4b8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.h:1508

#9  0x80409b9d in apply_event_and_update_pos_apply (ev=0xb0f0e498, thd=0xb231eee0, rgi=0xb0f2d4b8, reason=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:3847

#10 0x8040a0bd in apply_event_and_update_pos_for_parallel (ev=0xb0f0e498, thd=0xb231eee0, rgi=0xb0f2d4b8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:3981

#11 0x806d2e38 in rpt_handle_event (qev=0xb0f29368, rpt=0xb0f0da88) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:51

#12 0x806d633a in handle_rpl_parallel_thread (arg=0xb0f0da88) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:1335

#13 0x80a49677 in pfs_spawn_thread (arg=0xb0f0e1c8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#14 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#15 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 3 (Thread 0xaa2f6b40 (LWP 4688)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80c42e30 in os_event::timed_wait (this=0x83ae7770, abstime=0xaa2f6238) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:275

#3  0x80c43141 in os_event::wait_time_low (this=0x83ae7770, time_in_usec=10000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:385

#4  0x80c4333c in os_event_wait_time_low (event=0x83ae7770, time_in_usec=10000000, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:485

#5  0x80eef541 in dict_stats_thread () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/dict/dict0stats_bg.cc:491

#6  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#7  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 2 (Thread 0xa9af5b40 (LWP 4689)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x80c42e30 in os_event::timed_wait (this=0x83abdd88, abstime=0xa9af50f8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:275

#3  0x80c43141 in os_event::wait_time_low (this=0x83abdd88, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:385

#4  0x80c4333c in os_event_wait_time_low (event=0x83abdd88, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:485

#5  0x80dd8b17 in ib_wqueue_timedwait (wq=0x83abdcc8, wait_in_usecs=5000000) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/ut/ut0wqueue.cc:156

#6  0x80f53a72 in fts_optimize_thread (arg=0x83abdcc8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/fts/fts0opt.cc:2828

#7  0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#8  0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 1 (Thread 0xa4cfbb40 (LWP 4869)):

#0  0xb77b5c31 in __kernel_vsyscall ()

#1  0xb722ace1 in pthread_kill () from /lib/i386-linux-gnu/libpthread.so.0

#2  0x810673a6 in my_write_core (sig=11) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/stacktrace.c:386

#3  0x8080d570 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/signal_handler.cc:355

#4  <signal handler called>

#5  0x80735570 in I_P_List_iterator<TMP_TABLE_SHARE, I_P_List<TMP_TABLE_SHARE, All_tmp_table_shares, I_P_List_null_counter, I_P_List_no_push_back<TMP_TABLE_SHARE> > >::I_P_List_iterator (this=0xa4cfa84c, a=...) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_plist.h:179

#6  0x80733158 in THD::mark_tmp_tables_as_free_for_reuse (this=0xa6f77dd0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/temporary_tables.cc:749

#7  0x80464f0c in close_thread_tables (thd=0xa6f77dd0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_base.cc:806

#8  0x80504e35 in mysql_execute_command (thd=0xa6f77dd0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:6160

#9  0x80509856 in mysql_parse (thd=0xa6f77dd0, rawbuf=0xb0f2e6fb "COMMIT", length=6, parser_state=0xa4cfafa0, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:7870

#10 0x8095216d in Query_log_event::do_apply_event (this=0xb0f21788, rgi=0xb0f23bb0, query_arg=0xb0f2e6fb "COMMIT", q_len_arg=6) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.cc:5700

#11 0x80950e1e in Query_log_event::do_apply_event (this=0xb0f21788, rgi=0xb0f23bb0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.cc:5377

#12 0x80416c87 in Log_event::apply_event (this=0xb0f21788, rgi=0xb0f23bb0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.h:1492

#13 0x80409907 in apply_event_and_update_pos_apply (ev=0xb0f21788, thd=0xa6f77dd0, rgi=0xb0f23bb0, reason=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:3789

#14 0x8040a0bd in apply_event_and_update_pos_for_parallel (ev=0xb0f21788, thd=0xa6f77dd0, rgi=0xb0f23bb0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:3981

#15 0x806d2e38 in rpt_handle_event (qev=0xb0f24558, rpt=0xb0f0dd50) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:51

#16 0x806d633a in handle_rpl_parallel_thread (arg=0xb0f0dd50) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:1335

#17 0x80a49677 in pfs_spawn_thread (arg=0xb0f0e618) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869

#18 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0

#19 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Alice Sherepa added a comment - 2021-09-17 13:06 10.3 497b69493620eaa4cb1f2240a10be85e rpl.rpl_parallel_temptable 'innodb,stmt' w4 [ fail ] Test ended at 2021-08-24 17:52:10 CURRENT_TEST: rpl.rpl_parallel_temptable mysqltest: At line 132: failed in 'select master_pos_wait('master-bin.000002', 585, 300, '')': 2013: Lost connection to MySQL server during query The result from queries just before the failure was: < snip > INSERT INTO t2 VALUES (19); connection default; INSERT INTO t3 SELECT a+128 FROM t4; connection server_1; INSERT INTO t2 VALUES (20); connection default; INSERT INTO t1 SELECT a, a MOD 7 FROM t3; INSERT INTO t1 SELECT a, a MOD 7 FROM t4; connection server_1; INSERT INTO t1 SELECT a, COUNT(*) FROM t2 GROUP BY a; FLUSH TABLES; SET SESSION debug_dbug="+d,crash_dispatch_command_before"; SELECT 1; Got one of the listed errors connection default; connection default; connection server_1; INSERT INTO t1 VALUES (0, 1); connection server_2; include/start_slave.inc More results from queries before failure can be found in /mnt/buildbot/build/mariadb-10.3.32/mysql-test/var/4/log/rpl_parallel_temptable.log Server [mysqld.2 - pid: 4673, winpid: 4673, exit: 256] failed during test run Server log from this test: ----------SERVER LOG START----------- $ /mnt/buildbot/build/mariadb-10.3.32/sql/mysqld --defaults-group-suffix=.2 --defaults-file=/mnt/buildbot/build/mariadb-10.3.32/mysql-test/var/4/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --log-bin=slave-bin --binlog-format=statement --log-bin=slave-bin --core-file --loose-debug-sync-timeout=300 2021-08-24 17:52:05 0 [Note] /mnt/buildbot/build/mariadb-10.3.32/sql/mysqld (mysqld 10.3.32-MariaDB-debug-log) starting as process 4674 ... 2021-08-24 17:52:05 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32186) 2021-08-24 17:52:05 0 [Warning] Changed limits: max_open_files: 1024 max_connections: 151 (was 151) table_cache: 421 (was 2000) 2021-08-24 17:52:06 0 [Note] Plugin 'partition' is disabled. 2021-08-24 17:52:06 0 [Note] InnoDB: Using Linux native AIO 2021-08-24 17:52:06 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2021-08-24 17:52:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2021-08-24 17:52:06 0 [Note] InnoDB: Uses event mutexes 2021-08-24 17:52:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.8 2021-08-24 17:52:06 0 [Note] InnoDB: Number of pools: 1 2021-08-24 17:52:06 0 [Note] InnoDB: Using generic crc32 instructions 2021-08-24 17:52:06 0 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M 2021-08-24 17:52:06 0 [Note] InnoDB: Completed initialization of buffer pool 2021-08-24 17:52:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2021-08-24 17:52:06 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2021-08-24 17:52:06 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2021-08-24 17:52:06 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2021-08-24 17:52:06 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2021-08-24 17:52:06 0 [Note] InnoDB: Waiting for purge to start 2021-08-24 17:52:06 0 [Note] InnoDB: 10.3.32 started; log sequence number 1636384; transaction id 33 2021-08-24 17:52:06 0 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var_auto_iuHk/4/mysqld.2/data/ib_buffer_pool 2021-08-24 17:52:06 0 [Note] Plugin 'SEQUENCE' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_CMP' is disabled. 2021-08-24 17:52:06 0 [Note] InnoDB: Buffer pool(s) load completed at 210824 17:52:06 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'user_variables' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_MUTEXES' is disabled. 2021-08-24 17:52:06 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled. 2021-08-24 17:52:06 0 [Warning] /mnt/buildbot/build/mariadb-10.3.32/sql/mysqld: unknown option '--loose-pam-debug' 2021-08-24 17:52:06 0 [Note] Server socket created on IP: '127.0.0.1'. 2021-08-24 17:52:06 0 [Note] Reading of all Master_info entries succeeded 2021-08-24 17:52:06 0 [Note] Added new Master_info '' to hash table 2021-08-24 17:52:06 0 [Note] /mnt/buildbot/build/mariadb-10.3.32/sql/mysqld: ready for connections. Version: '10.3.32-MariaDB-debug-log' socket: '/mnt/buildbot/build/mariadb-10.3.32/mysql-test/var/tmp/4/mysqld.2.sock' port: 16061 Source distribution 2021-08-24 17:52:07 10 [Note] Deleted Master_info file '/mnt/buildbot/build/mariadb-10.3.32/mysql-test/var/4/mysqld.2/data/master.info'. 2021-08-24 17:52:07 10 [Note] Deleted Master_info file '/mnt/buildbot/build/mariadb-10.3.32/mysql-test/var/4/mysqld.2/data/relay-log.info'. 2021-08-24 17:52:07 10 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2021-08-24 17:52:07 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='4'. 2021-08-24 17:52:07 12 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16060' in log 'master-bin.000001' at position 4 2021-08-24 17:52:07 13 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 4, relay log './slave-relay-bin.000001' position: 4 2021-08-24 17:52:07 12 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication started in log 'master-bin.000001' at position 4 2021-08-24 17:52:07 13 [Note] Error reading relay log event: slave SQL thread was killed 2021-08-24 17:52:07 13 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 329 2021-08-24 17:52:07 13 [Note] master was 127.0.0.1:16060 2021-08-24 17:52:07 12 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 329 2021-08-24 17:52:07 12 [Note] master was 127.0.0.1:16060 2021-08-24 17:52:07 10 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2021-08-24 17:52:07 10 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='329'. New state master_host='127.0.0.1', master_port='16060', master_log_file='master-bin.000001', master_log_pos='329'. 2021-08-24 17:52:07 10 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos 2021-08-24 17:52:07 14 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16060' in log 'master-bin.000001' at position 329 2021-08-24 17:52:07 14 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication starts at GTID position '' 2021-08-24 17:52:07 15 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 329, relay log './slave-relay-bin.000001' position: 4; GTID position '' 2021-08-24 17:52:07 15 [Note] Error reading relay log event: slave SQL thread was killed 2021-08-24 17:52:07 15 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 508; GTID position '0-1-1' 2021-08-24 17:52:07 15 [Note] master was 127.0.0.1:16060 2021-08-24 17:52:07 14 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 508; GTID position 0-1-1 2021-08-24 17:52:07 14 [Note] master was 127.0.0.1:16060 2021-08-24 17:52:08 21 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16060' in log 'master-bin.000001' at position 508 2021-08-24 17:52:08 21 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16060',replication starts at GTID position '0-1-1' 2021-08-24 17:52:08 22 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 508, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-1' 210824 17:52:08 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.32-MariaDB-debug-log key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=2 max_threads=153 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62003 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0xa6f77dd0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0xa4cfb1e4 thread_stack 0x49000 mysys/stacktrace.c:174(my_print_stacktrace)[0x810672ac] sql/signal_handler.cc:219(handle_fatal_signal)[0x8080d1c0] addr2line: '': No such file [0xb77b5c14] sql/sql_plist.h:179(I_P_List_iterator<TMP_TABLE_SHARE, I_P_List<TMP_TABLE_SHARE, All_tmp_table_shares, I_P_List_null_counter, I_P_List_no_push_back<TMP_TABLE_SHARE> > >::I_P_List_iterator(I_P_List<TMP_TABLE_SHARE, All_tmp_table_shares, I_P_List_null_counter, I_P_List_no_push_back<TMP_TABLE_SHARE> > const&))[0x80735570] sql/temporary_tables.cc:749(THD::mark_tmp_tables_as_free_for_reuse())[0x80733158] sql/sql_base.cc:806(close_thread_tables(THD*))[0x80464f0c] sql/sql_parse.cc:6160(mysql_execute_command(THD*))[0x80504e35] sql/sql_parse.cc:7870(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x80509856] sql/log_event.cc:5700(Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int))[0x8095216d] sql/log_event.cc:5377(Query_log_event::do_apply_event(rpl_group_info*))[0x80950e1e] sql/log_event.h:1492(Log_event::apply_event(rpl_group_info*))[0x80416c87] sql/slave.cc:3789(apply_event_and_update_pos_apply(Log_event*, THD*, rpl_group_info*, int))[0x80409907] sql/slave.cc:3981(apply_event_and_update_pos_for_parallel(Log_event*, THD*, rpl_group_info*))[0x8040a0bd] sql/rpl_parallel.cc:51(rpt_handle_event(rpl_parallel_thread::queued_event*, rpl_parallel_thread*))[0x806d2e38] sql/rpl_parallel.cc:1335(handle_rpl_parallel_thread)[0x806d633a] perfschema/pfs.cc:1869(pfs_spawn_thread)[0x80a49677] /lib/i386-linux-gnu/libpthread.so.0(+0x62b5)[0xb72242b5] /lib/i386-linux-gnu/libc.so.6(clone+0x6e)[0xb714f16e] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0xb0f2e6fb): COMMIT Connection ID (thread ID): 26 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /dev/shm/var_auto_iuHk/4/mysqld.2/data Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 64511 64511 processes Max open files 1024 1024 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64511 64511 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: |/usr/share/apport/apport %p %s %c %P ----------SERVER LOG END------------- - found 'core' (0/0) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /mnt/buildbot/build/mariadb-10.3.32/mysql-test/var/4/log/rpl.rpl_parallel_temptable-innodb,stmt/mysqld.2/data/core Core generated by '/mnt/buildbot/build/mariadb-10.3.32/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 4869] [New LWP 4689] [New LWP 4688] [New LWP 4867] [New LWP 4681] [New LWP 4685] [New LWP 4687] [New LWP 4680] [New LWP 4694] [New LWP 4686] [New LWP 4674] [New LWP 4679] [New LWP 4692] [New LWP 4864] [New LWP 4678] [New LWP 4691] [New LWP 4865] [New LWP 4677] [New LWP 4696] [New LWP 4868] [New LWP 4676] [New LWP 4697] [New LWP 4870] [New LWP 4698] [New LWP 4675] [New LWP 4871] [New LWP 4700] [New LWP 4683] [New LWP 4706] [New LWP 4682] [New LWP 4719] [New LWP 4701] [New LWP 4699] [New LWP 4695] [New LWP 4693] [New LWP 4690] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1". Core was generated by `/mnt/buildbot/build/mariadb-10.3.32/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0xb77b5c31 in __kernel_vsyscall () [Current thread is 1 (Thread 0xa4cfbb40 (LWP 4869))] #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb722ace1 in pthread_kill () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x810673a6 in my_write_core (sig=11) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/stacktrace.c:386 #3 0x8080d570 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/signal_handler.cc:355 #4 <signal handler called> #5 0x80735570 in I_P_List_iterator<TMP_TABLE_SHARE, I_P_List<TMP_TABLE_SHARE, All_tmp_table_shares, I_P_List_null_counter, I_P_List_no_push_back<TMP_TABLE_SHARE> > >::I_P_List_iterator (this=0xa4cfa84c, a=...) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_plist.h:179 #6 0x80733158 in THD::mark_tmp_tables_as_free_for_reuse (this=0xa6f77dd0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/temporary_tables.cc:749 #7 0x80464f0c in close_thread_tables (thd=0xa6f77dd0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_base.cc:806 #8 0x80504e35 in mysql_execute_command (thd=0xa6f77dd0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:6160 #9 0x80509856 in mysql_parse (thd=0xa6f77dd0, rawbuf=0xb0f2e6fb "COMMIT", length=6, parser_state=0xa4cfafa0, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:7870 #10 0x8095216d in Query_log_event::do_apply_event (this=0xb0f21788, rgi=0xb0f23bb0, query_arg=0xb0f2e6fb "COMMIT", q_len_arg=6) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.cc:5700 #11 0x80950e1e in Query_log_event::do_apply_event (this=0xb0f21788, rgi=0xb0f23bb0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.cc:5377 #12 0x80416c87 in Log_event::apply_event (this=0xb0f21788, rgi=0xb0f23bb0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.h:1492 #13 0x80409907 in apply_event_and_update_pos_apply (ev=0xb0f21788, thd=0xa6f77dd0, rgi=0xb0f23bb0, reason=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:3789 #14 0x8040a0bd in apply_event_and_update_pos_for_parallel (ev=0xb0f21788, thd=0xa6f77dd0, rgi=0xb0f23bb0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:3981 #15 0x806d2e38 in rpt_handle_event (qev=0xb0f24558, rpt=0xb0f0dd50) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:51 #16 0x806d633a in handle_rpl_parallel_thread (arg=0xb0f0dd50) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:1335 #17 0x80a49677 in pfs_spawn_thread (arg=0xb0f0e618) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #18 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #19 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 36 (Thread 0xac2fab40 (LWP 4690)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb722d8aa in nanosleep () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x80c439c6 in os_thread_sleep (tm=1000000) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0thread.cc:231 #3 0x80d68013 in srv_master_sleep () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:2351 #4 0x80d68190 in srv_master_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:2391 #5 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #6 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 35 (Thread 0xa80fdb40 (LWP 4693)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x80c435c6 in os_event::wait (this=0x839a5050) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:158 #3 0x80c43003 in os_event::wait_low (this=0x839a5050, reset_sig_count=5) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:325 #4 0x80c4336e in os_event_wait_low (event=0x839a5050, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:502 #5 0x80d643b2 in srv_resume_thread (slot=0x81928c30 <srv_sys+432>, sig_count=0, wait=true, timeout_usec=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:916 #6 0x80d689e5 in srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:2538 #7 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #8 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 34 (Thread 0xa6cffb40 (LWP 4695)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x80c435c6 in os_event::wait (this=0x839a5230) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:158 #3 0x80c43003 in os_event::wait_low (this=0x839a5230, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:325 #4 0x80c4336e in os_event_wait_low (event=0x839a5230, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:502 #5 0x80e717cb in buf_dump_thread () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/buf/buf0dump.cc:834 #6 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #7 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 33 (Thread 0xb00cab40 (LWP 4699)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x8106c03b in safe_cond_wait (cond=0x81af21f4 <mysql_bin_log+3316>, mp=0x81af2190 <mysql_bin_log+3216>, file=0x812dea84 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1174) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:492 #3 0x809241ef in inline_mysql_cond_wait (that=0x81af21f4 <mysql_bin_log+3316>, mutex=0x81af2190 <mysql_bin_log+3216>, src_file=0x812df384 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/log.cc", src_line=9970) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1174 #4 0x80940abd in binlog_background_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log.cc:9970 #5 0x80a49677 in pfs_spawn_thread (arg=0x839058a0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #6 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #7 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 32 (Thread 0xac6c3b40 (LWP 4701)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x8106c03b in safe_cond_wait (cond=0x81ad3f80 <COND_manager>, mp=0x81ad3f00 <LOCK_manager>, file=0x811285c0 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1174) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:492 #3 0x804efa96 in inline_mysql_cond_wait (that=0x81ad3f80 <COND_manager>, mutex=0x81ad3f00 <LOCK_manager>, src_file=0x811286d0 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_manager.cc", src_line=102) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1174 #4 0x804eff81 in handle_manager (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_manager.cc:102 #5 0x80a49677 in pfs_spawn_thread (arg=0x83b8e960) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #6 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #7 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 31 (Thread 0xa92f4b40 (LWP 4719)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7144f8f in poll () from /lib/i386-linux-gnu/libc.so.6 #2 0x810c15ec in vio_io_wait (vio=0x83bd3900, event=VIO_IO_EVENT_READ, timeout=28800000) at /home/buildbot/buildbot/build/mariadb-10.3.32/vio/viosocket.c:962 #3 0x810bfdb1 in vio_socket_io_wait (vio=0x83bd3900, event=VIO_IO_EVENT_READ) at /home/buildbot/buildbot/build/mariadb-10.3.32/vio/viosocket.c:119 #4 0x810bffa1 in vio_read (vio=0x83bd3900, buf=0xa910b928 "\001", size=4) at /home/buildbot/buildbot/build/mariadb-10.3.32/vio/viosocket.c:196 #5 0x803e29eb in my_real_read (net=0xa91063b0, complen=0xa92f4118, header=1 '\001') at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/net_serv.cc:975 #6 0x803e3236 in my_net_read_packet_reallen (net=0xa91063b0, read_from_server=1 '\001', reallen=0xa92f418c) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/net_serv.cc:1256 #7 0x803e31e8 in my_net_read_packet (net=0xa91063b0, read_from_server=1 '\001') at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/net_serv.cc:1240 #8 0x804f35b3 in do_command (thd=0xa91061e8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:1274 #9 0x8065e7c2 in do_handle_one_connection (connect=0x83b9b478) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_connect.cc:1403 #10 0x8065e4f7 in handle_one_connection (arg=0x83b9b478) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_connect.cc:1308 #11 0x80a49677 in pfs_spawn_thread (arg=0x83bd3ac8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #12 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #13 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 30 (Thread 0xad6c5b40 (LWP 4682)): #0 0xb7781570 in ?? () from /lib/i386-linux-gnu/libaio.so.1 #1 0xb77815e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1 #2 0x80c335e5 in LinuxAIOHandler::collect (this=0xad6c50f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:1936 #3 0x80c33cf1 in LinuxAIOHandler::poll (this=0xad6c50f0, m1=0xad6c5154, m2=0xad6c5158, request=0xad6c5178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2097 #4 0x80c33e55 in os_aio_linux_handler (global_segment=5, m1=0xad6c5154, m2=0xad6c5158, request=0xad6c5178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2151 #5 0x80c39502 in os_aio_handler (segment=5, m1=0xad6c5154, m2=0xad6c5158, request=0xad6c5178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:5761 #6 0x80f08eb0 in fil_aio_wait (segment=5) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/fil/fil0fil.cc:4465 #7 0x80d6c00c in io_handler_thread (arg=0x8232f2b4 <n+20>) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0start.cc:326 #8 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #9 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 29 (Thread 0xac679b40 (LWP 4706)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb722cd32 in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread.so.0 #2 0xb72269e5 in pthread_mutex_lock () from /lib/i386-linux-gnu/libpthread.so.0 #3 0x8106b88e in safe_mutex_lock (mp=0x83bd07d4, my_flags=0, file=0x8115f0e8 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_rli.cc", line=815) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:293 #4 0x806487b7 in inline_mysql_mutex_lock (that=0x83bd07d4, src_file=0x8115f0e8 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_rli.cc", src_line=815) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:702 #5 0x8064b9b7 in Relay_log_info::wait_for_pos (this=0x83bced98, thd=0xa6f06f68, log_name=0xa6f15a20, log_pos=585, timeout=300) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_rli.cc:815 #6 0x808a00b3 in Item_master_pos_wait::val_int (this=0xa6f15c90) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/item_func.cc:3760 #7 0x806ed3ee in Type_handler::Item_send_longlong (this=0x81aedbfc <type_handler_longlong>, item=0xa6f15c90, protocol=0xa6f073b0, buf=0xac677a5c) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_type.cc:5450 #8 0x806f2b89 in Type_handler_longlong::Item_send (this=0x81aedbfc <type_handler_longlong>, item=0xa6f15c90, protocol=0xa6f073b0, buf=0xac677a5c) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_type.h:2498 #9 0x803f0141 in Item::send (this=0xa6f15c90, protocol=0xa6f073b0, buffer=0xac677a5c) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/item.h:884 #10 0x803eaa10 in Protocol::send_result_set_row (this=0xa6f073b0, row_items=0xa6f09c6c) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/protocol.cc:999 #11 0x8049466e in select_send::send_data (this=0xa6f15e50, items=...) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_class.cc:2999 #12 0x80548071 in JOIN::exec_inner (this=0xa6f15e68) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_select.cc:4017 #13 0x80547a57 in JOIN::exec (this=0xa6f15e68) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_select.cc:3936 #14 0x80548e44 in mysql_select (thd=0xa6f06f68, tables=0x0, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0xa6f15e50, unit=0xa6f096a0, select_lex=0xa6f09bc8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_select.cc:4344 #15 0x8053b7f3 in handle_select (thd=0xa6f06f68, lex=0xa6f09630, result=0xa6f15e50, setup_tables_done_option=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_select.cc:384 #16 0x80505b75 in execute_sqlcom_select (thd=0xa6f06f68, all_tables=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:6339 #17 0x804fb690 in mysql_execute_command (thd=0xa6f06f68) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:3870 #18 0x80509856 in mysql_parse (thd=0xa6f06f68, rawbuf=0xa6f15908 "select master_pos_wait('master-bin.000002', 585, 300, '')", length=57, parser_state=0xac678b98, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:7870 #19 0x804f53d4 in dispatch_command (command=COM_QUERY, thd=0xa6f06f68, packet=0xa6f0c649 "", packet_length=57, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:1853 #20 0x804f3b7a in do_command (thd=0xa6f06f68) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:1399 #21 0x8065e7c2 in do_handle_one_connection (connect=0x83b8eca8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_connect.cc:1403 #22 0x8065e4f7 in handle_one_connection (arg=0x83b8eca8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_connect.cc:1308 #23 0x80a49677 in pfs_spawn_thread (arg=0x83bd3900) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #24 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #25 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 28 (Thread 0xacec4b40 (LWP 4683)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x80c42e30 in os_event::timed_wait (this=0x839a5290, abstime=0xacec4078) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:275 #3 0x80c43141 in os_event::wait_time_low (this=0x839a5290, time_in_usec=935000, reset_sig_count=4) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:385 #4 0x80c4333c in os_event_wait_time_low (event=0x839a5290, time_in_usec=935000, reset_sig_count=4) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:485 #5 0x80e7ec4e in pc_sleep_if_needed (next_loop_time=1370734, sig_count=4, cur_time=1369799) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/buf/buf0flu.cc:2665 #6 0x80e7fe67 in buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/buf/buf0flu.cc:3124 #7 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #8 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 27 (Thread 0xb0080b40 (LWP 4700)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb722e1cd in do_sigwait () from /lib/i386-linux-gnu/libpthread.so.0 #2 0xb722e25e in sigwait () from /lib/i386-linux-gnu/libpthread.so.0 #3 0x803d04e1 in signal_hand (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/mysqld.cc:3590 #4 0x80a49677 in pfs_spawn_thread (arg=0x83b39ea8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #5 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #6 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 26 (Thread 0xa4cb1b40 (LWP 4871)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x8106c03b in safe_cond_wait (cond=0xb0f0e080, mp=0xb0f0e01c, file=0x811842e0 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1174) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:492 #3 0x806d29b9 in inline_mysql_cond_wait (that=0xb0f0e080, mutex=0xb0f0e01c, src_file=0x81184658 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc", src_line=1114) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1174 #4 0x806d5b7a in handle_rpl_parallel_thread (arg=0xb0f0e018) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:1114 #5 0x80a49677 in pfs_spawn_thread (arg=0xb0f0ea68) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #6 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #7 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 25 (Thread 0xb2c8fb40 (LWP 4675)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x8106c2e6 in safe_cond_timedwait (cond=0x823438c0 <COND_timer>, mp=0x82343840 <LOCK_timer>, abstime=0xb2c8f250, file=0x814c55f4 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1211) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:546 #3 0x8106d298 in inline_mysql_cond_timedwait (that=0x823438c0 <COND_timer>, mutex=0x82343840 <LOCK_timer>, abstime=0xb2c8f250, src_file=0x814c5644 "/home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_timer.c", src_line=292) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1211 #4 0x8106de77 in timer_handler (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_timer.c:292 #5 0x80a49677 in pfs_spawn_thread (arg=0x838ce578) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #6 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #7 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 24 (Thread 0xa54fcb40 (LWP 4698)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x8106c03b in safe_cond_wait (cond=0xa54fc1c4, mp=0xa54fc1f8, file=0x81344960 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1185) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:492 #3 0x80b3f3ed in inline_mysql_cond_wait (that=0xa54fc1c4, mutex=0xa54fc1f8, src_file=0x81346818 "/home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/handler/ha_innodb.cc", src_line=324) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1185 #4 0x80b45223 in thd_destructor_proxy () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/handler/ha_innodb.cc:324 #5 0x80a49677 in pfs_spawn_thread (arg=0x83b1ac58) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #6 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #7 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 23 (Thread 0xa70fbb40 (LWP 4870)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x8106c03b in safe_cond_wait (cond=0xb0f0df1c, mp=0xb0f0deb8, file=0x811842e0 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1174) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:492 #3 0x806d29b9 in inline_mysql_cond_wait (that=0xb0f0df1c, mutex=0xb0f0deb8, src_file=0x81184658 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc", src_line=1114) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1174 #4 0x806d5b7a in handle_rpl_parallel_thread (arg=0xb0f0deb4) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:1114 #5 0x80a49677 in pfs_spawn_thread (arg=0xb0f0e840) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #6 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #7 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 22 (Thread 0xa5cfdb40 (LWP 4697)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x80c435c6 in os_event::wait (this=0x839a52f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:158 #3 0x80c43003 in os_event::wait_low (this=0x839a52f0, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:325 #4 0x80c4336e in os_event_wait_low (event=0x839a52f0, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:502 #5 0x80e57bdf in buf_resize_thread () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/buf/buf0buf.cc:3170 #6 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #7 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 21 (Thread 0xb1861b40 (LWP 4676)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x8106c2e6 in safe_cond_timedwait (cond=0x822fa240 <COND_checkpoint>, mp=0x822fa1c0 <LOCK_checkpoint>, abstime=0xb186119c, file=0x81336b18 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1211) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:546 #3 0x80ab0a75 in inline_mysql_cond_timedwait (that=0x822fa240 <COND_checkpoint>, mutex=0x822fa1c0 <LOCK_checkpoint>, abstime=0xb186119c, src_file=0x81336b68 "/home/buildbot/buildbot/build/mariadb-10.3.32/storage/maria/ma_servicethread.c", src_line=116) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1211 #4 0x80ab0fda in my_service_thread_sleep (control=0x81920038 <checkpoint_control>, sleep_time=29000000000) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/maria/ma_servicethread.c:115 #5 0x80aa486d in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/maria/ma_checkpoint.c:707 #6 0x80a49677 in pfs_spawn_thread (arg=0x839097a8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #7 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #8 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 20 (Thread 0xa7067b40 (LWP 4868)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x8106c03b in safe_cond_wait (cond=0xb0f0dc54, mp=0xb0f0dbf0, file=0x811842e0 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1174) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:492 #3 0x806d29b9 in inline_mysql_cond_wait (that=0xb0f0dc54, mutex=0xb0f0dbf0, src_file=0x81184658 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc", src_line=1114) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1174 #4 0x806d5b7a in handle_rpl_parallel_thread (arg=0xb0f0dbec) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:1114 #5 0x80a49677 in pfs_spawn_thread (arg=0xb0f0e3f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #6 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #7 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 19 (Thread 0xa64feb40 (LWP 4696)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb722d8aa in nanosleep () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x80c439c6 in os_thread_sleep (tm=1000000) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0thread.cc:231 #3 0x80e45fa8 in btr_defragment_thread () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/btr/btr0defragment.cc:707 #4 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #5 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 18 (Thread 0xafecab40 (LWP 4677)): #0 0xb7781570 in ?? () from /lib/i386-linux-gnu/libaio.so.1 #1 0xb77815e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1 #2 0x80c335e5 in LinuxAIOHandler::collect (this=0xafeca0f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:1936 #3 0x80c33cf1 in LinuxAIOHandler::poll (this=0xafeca0f0, m1=0xafeca154, m2=0xafeca158, request=0xafeca178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2097 #4 0x80c33e55 in os_aio_linux_handler (global_segment=0, m1=0xafeca154, m2=0xafeca158, request=0xafeca178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2151 #5 0x80c39502 in os_aio_handler (segment=0, m1=0xafeca154, m2=0xafeca158, request=0xafeca178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:5761 #6 0x80f08eb0 in fil_aio_wait (segment=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/fil/fil0fil.cc:4465 #7 0x80d6c00c in io_handler_thread (arg=0x8232f2a0 <n>) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0start.cc:326 #8 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #9 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 17 (Thread 0xa70b1b40 (LWP 4865)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x8106c03b in safe_cond_wait (cond=0x83bcf7c0, mp=0x83bcf218, file=0x812dea84 "/home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h", line=1174) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:492 #3 0x809241ef in inline_mysql_cond_wait (that=0x83bcf7c0, mutex=0x83bcf218, src_file=0x812df384 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/log.cc", src_line=8409) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:1174 #4 0x8093c5fe in MYSQL_BIN_LOG::wait_for_update_relay_log (this=0x83bcf214, thd=0xb0f082c8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log.cc:8409 #5 0x804144c3 in next_event (rgi=0xb0f07a28, event_size=0xa70b0b60) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:7574 #6 0x8040a12a in exec_relay_log_event (thd=0xb0f082c8, rli=0x83bced98, serial_rgi=0xb0f07a28) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:4102 #7 0x8040e31d in handle_slave_sql (arg=0x83bcd328) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:5429 #8 0x80a49677 in pfs_spawn_thread (arg=0xa6f758e8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #9 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #10 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 16 (Thread 0xa90ffb40 (LWP 4691)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x80c435c6 in os_event::wait (this=0x839a4ff0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:158 #3 0x80c43003 in os_event::wait_low (this=0x839a4ff0, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:325 #4 0x80c4336e in os_event_wait_low (event=0x839a4ff0, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:502 #5 0x80d643b2 in srv_resume_thread (slot=0x81928ba8 <srv_sys+296>, sig_count=1, wait=true, timeout_usec=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:916 #6 0x80d68ed6 in srv_purge_coordinator_suspend (slot=0x81928ba8 <srv_sys+296>, rseg_history_len=15) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:2686 #7 0x80d69210 in srv_purge_coordinator_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:2767 #8 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #9 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 15 (Thread 0xaf6c9b40 (LWP 4678)): #0 0xb7781570 in ?? () from /lib/i386-linux-gnu/libaio.so.1 #1 0xb77815e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1 #2 0x80c335e5 in LinuxAIOHandler::collect (this=0xaf6c90f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:1936 #3 0x80c33cf1 in LinuxAIOHandler::poll (this=0xaf6c90f0, m1=0xaf6c9154, m2=0xaf6c9158, request=0xaf6c9178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2097 #4 0x80c33e55 in os_aio_linux_handler (global_segment=1, m1=0xaf6c9154, m2=0xaf6c9158, request=0xaf6c9178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2151 #5 0x80c39502 in os_aio_handler (segment=1, m1=0xaf6c9154, m2=0xaf6c9158, request=0xaf6c9178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:5761 #6 0x80f08eb0 in fil_aio_wait (segment=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/fil/fil0fil.cc:4465 #7 0x80d6c00c in io_handler_thread (arg=0x8232f2a4 <n+4>) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0start.cc:326 #8 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #9 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 14 (Thread 0xa9260b40 (LWP 4864)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7144f8f in poll () from /lib/i386-linux-gnu/libc.so.6 #2 0x810c15ec in vio_io_wait (vio=0xac5087a8, event=VIO_IO_EVENT_READ, timeout=120000) at /home/buildbot/buildbot/build/mariadb-10.3.32/vio/viosocket.c:962 #3 0x810bfdb1 in vio_socket_io_wait (vio=0xac5087a8, event=VIO_IO_EVENT_READ) at /home/buildbot/buildbot/build/mariadb-10.3.32/vio/viosocket.c:119 #4 0x810bffa1 in vio_read (vio=0xac5087a8, buf=0xac508970 "1", size=16384) at /home/buildbot/buildbot/build/mariadb-10.3.32/vio/viosocket.c:196 #5 0x810c019d in vio_read_buff (vio=0xac5087a8, buf=0xac50c9b8 "", size=4) at /home/buildbot/buildbot/build/mariadb-10.3.32/vio/viosocket.c:244 #6 0x803e29eb in my_real_read (net=0xac508390, complen=0xa9260028, header=0 '\000') at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/net_serv.cc:975 #7 0x803e3236 in my_net_read_packet_reallen (net=0xac508390, read_from_server=0 '\000', reallen=0xa92601a4) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/net_serv.cc:1256 #8 0x807bd2df in cli_safe_read_reallen (mysql=0xac508390, reallen=0xa92601a4) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql-common/client.c:594 #9 0x80408f25 in read_event (mysql=0xac508390, mi=0x83bcd328, suppress_warnings=0xa926014a, network_read_len=0xa92601a4) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:3564 #10 0x8040be61 in handle_slave_io (arg=0x83bcd328) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:4711 #11 0x80a49677 in pfs_spawn_thread (arg=0xa6f758e8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #12 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #13 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 13 (Thread 0xa88feb40 (LWP 4692)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x80c435c6 in os_event::wait (this=0x839a5110) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:158 #3 0x80c43003 in os_event::wait_low (this=0x839a5110, reset_sig_count=4) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:325 #4 0x80c4336e in os_event_wait_low (event=0x839a5110, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:502 #5 0x80d643b2 in srv_resume_thread (slot=0x81928d40 <srv_sys+704>, sig_count=0, wait=true, timeout_usec=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:916 #6 0x80d689e5 in srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:2538 #7 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #8 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 12 (Thread 0xaeec8b40 (LWP 4679)): #0 0xb7781570 in ?? () from /lib/i386-linux-gnu/libaio.so.1 #1 0xb77815e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1 #2 0x80c335e5 in LinuxAIOHandler::collect (this=0xaeec80f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:1936 #3 0x80c33cf1 in LinuxAIOHandler::poll (this=0xaeec80f0, m1=0xaeec8154, m2=0xaeec8158, request=0xaeec8178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2097 #4 0x80c33e55 in os_aio_linux_handler (global_segment=2, m1=0xaeec8154, m2=0xaeec8158, request=0xaeec8178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2151 #5 0x80c39502 in os_aio_handler (segment=2, m1=0xaeec8154, m2=0xaeec8158, request=0xaeec8178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:5761 #6 0x80f08eb0 in fil_aio_wait (segment=2) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/fil/fil0fil.cc:4465 #7 0x80d6c00c in io_handler_thread (arg=0x8232f2a8 <n+8>) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0start.cc:326 #8 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #9 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 11 (Thread 0xb6ed5a00 (LWP 4674)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7144f8f in poll () from /lib/i386-linux-gnu/libc.so.6 #2 0x803d71c5 in handle_connections_sockets () at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/mysqld.cc:6815 #3 0x803d64b7 in mysqld_main (argc=158, argv=0x837a8ad0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/mysqld.cc:6286 #4 0x803c9b3b in main (argc=26, argv=0xbf96a5f4) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/main.cc:25 Thread 10 (Thread 0xab2f8b40 (LWP 4686)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x80c42e30 in os_event::timed_wait (this=0x839a5170, abstime=0xab2f8118) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:275 #3 0x80c43141 in os_event::wait_time_low (this=0x839a5170, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:385 #4 0x80c4333c in os_event_wait_time_low (event=0x839a5170, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:485 #5 0x80d6690b in srv_error_monitor_thread () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:1861 #6 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #7 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 9 (Thread 0xa78fcb40 (LWP 4694)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x80c435c6 in os_event::wait (this=0x839a50b0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:158 #3 0x80c43003 in os_event::wait_low (this=0x839a50b0, reset_sig_count=4) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:325 #4 0x80c4336e in os_event_wait_low (event=0x839a50b0, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:502 #5 0x80d643b2 in srv_resume_thread (slot=0x81928cb8 <srv_sys+568>, sig_count=0, wait=true, timeout_usec=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:916 #6 0x80d689e5 in srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:2538 #7 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #8 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 8 (Thread 0xae6c7b40 (LWP 4680)): #0 0xb7781570 in ?? () from /lib/i386-linux-gnu/libaio.so.1 #1 0xb77815e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1 #2 0x80c335e5 in LinuxAIOHandler::collect (this=0xae6c70f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:1936 #3 0x80c33cf1 in LinuxAIOHandler::poll (this=0xae6c70f0, m1=0xae6c7154, m2=0xae6c7158, request=0xae6c7178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2097 #4 0x80c33e55 in os_aio_linux_handler (global_segment=3, m1=0xae6c7154, m2=0xae6c7158, request=0xae6c7178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2151 #5 0x80c39502 in os_aio_handler (segment=3, m1=0xae6c7154, m2=0xae6c7158, request=0xae6c7178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:5761 #6 0x80f08eb0 in fil_aio_wait (segment=3) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/fil/fil0fil.cc:4465 #7 0x80d6c00c in io_handler_thread (arg=0x8232f2ac <n+12>) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0start.cc:326 #8 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #9 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 7 (Thread 0xaaaf7b40 (LWP 4687)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x80c42e30 in os_event::timed_wait (this=0x839a51d0, abstime=0xaaaf7208) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:275 #3 0x80c43141 in os_event::wait_time_low (this=0x839a51d0, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:385 #4 0x80c4333c in os_event_wait_time_low (event=0x839a51d0, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:485 #5 0x80d6634b in srv_monitor_thread () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0srv.cc:1697 #6 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #7 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 6 (Thread 0xabaf9b40 (LWP 4685)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x80c42e30 in os_event::timed_wait (this=0x83abe8e0, abstime=0xabaf9218) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:275 #3 0x80c43141 in os_event::wait_time_low (this=0x83abe8e0, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:385 #4 0x80c4333c in os_event_wait_time_low (event=0x83abe8e0, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:485 #5 0x80c01ccb in lock_wait_timeout_thread () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/lock/lock0wait.cc:527 #6 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #7 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 5 (Thread 0xadec6b40 (LWP 4681)): #0 0xb7781570 in ?? () from /lib/i386-linux-gnu/libaio.so.1 #1 0xb77815e6 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1 #2 0x80c335e5 in LinuxAIOHandler::collect (this=0xadec60f0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:1936 #3 0x80c33cf1 in LinuxAIOHandler::poll (this=0xadec60f0, m1=0xadec6154, m2=0xadec6158, request=0xadec6178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2097 #4 0x80c33e55 in os_aio_linux_handler (global_segment=4, m1=0xadec6154, m2=0xadec6158, request=0xadec6178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:2151 #5 0x80c39502 in os_aio_handler (segment=4, m1=0xadec6154, m2=0xadec6158, request=0xadec6178) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0file.cc:5761 #6 0x80f08eb0 in fil_aio_wait (segment=4) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/fil/fil0fil.cc:4465 #7 0x80d6c00c in io_handler_thread (arg=0x8232f2b0 <n+16>) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/srv/srv0start.cc:326 #8 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #9 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 4 (Thread 0xa92aab40 (LWP 4867)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb722cd32 in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread.so.0 #2 0xb72269e5 in pthread_mutex_lock () from /lib/i386-linux-gnu/libpthread.so.0 #3 0x8106b88e in safe_mutex_lock (mp=0x83bd07d4, my_flags=0, file=0x8115f0e8 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_rli.cc", line=988) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/thr_mutex.c:293 #4 0x806487b7 in inline_mysql_mutex_lock (that=0x83bd07d4, src_file=0x8115f0e8 "/home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_rli.cc", src_line=988) at /home/buildbot/buildbot/build/mariadb-10.3.32/include/mysql/psi/mysql_thread.h:702 #5 0x8064c1f8 in Relay_log_info::inc_group_relay_log_pos (this=0x83bced98, log_pos=585, rgi=0xb0f2d4b8, skip_lock=false) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_rli.cc:988 #6 0x8064d726 in Relay_log_info::stmt_done (this=0x83bced98, event_master_log_pos=585, thd=0xb231eee0, rgi=0xb0f2d4b8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_rli.cc:1459 #7 0x8094ab23 in Log_event::do_update_pos (this=0xb0f0e498, rgi=0xb0f2d4b8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.cc:1433 #8 0x80416ce7 in Log_event::update_pos (this=0xb0f0e498, rgi=0xb0f2d4b8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.h:1508 #9 0x80409b9d in apply_event_and_update_pos_apply (ev=0xb0f0e498, thd=0xb231eee0, rgi=0xb0f2d4b8, reason=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:3847 #10 0x8040a0bd in apply_event_and_update_pos_for_parallel (ev=0xb0f0e498, thd=0xb231eee0, rgi=0xb0f2d4b8) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:3981 #11 0x806d2e38 in rpt_handle_event (qev=0xb0f29368, rpt=0xb0f0da88) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:51 #12 0x806d633a in handle_rpl_parallel_thread (arg=0xb0f0da88) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:1335 #13 0x80a49677 in pfs_spawn_thread (arg=0xb0f0e1c8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #14 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #15 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 3 (Thread 0xaa2f6b40 (LWP 4688)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x80c42e30 in os_event::timed_wait (this=0x83ae7770, abstime=0xaa2f6238) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:275 #3 0x80c43141 in os_event::wait_time_low (this=0x83ae7770, time_in_usec=10000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:385 #4 0x80c4333c in os_event_wait_time_low (event=0x83ae7770, time_in_usec=10000000, reset_sig_count=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:485 #5 0x80eef541 in dict_stats_thread () at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/dict/dict0stats_bg.cc:491 #6 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #7 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 2 (Thread 0xa9af5b40 (LWP 4689)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb7229e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x80c42e30 in os_event::timed_wait (this=0x83abdd88, abstime=0xa9af50f8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:275 #3 0x80c43141 in os_event::wait_time_low (this=0x83abdd88, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:385 #4 0x80c4333c in os_event_wait_time_low (event=0x83abdd88, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/os/os0event.cc:485 #5 0x80dd8b17 in ib_wqueue_timedwait (wq=0x83abdcc8, wait_in_usecs=5000000) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/ut/ut0wqueue.cc:156 #6 0x80f53a72 in fts_optimize_thread (arg=0x83abdcc8) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/innobase/fts/fts0opt.cc:2828 #7 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #8 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6 Thread 1 (Thread 0xa4cfbb40 (LWP 4869)): #0 0xb77b5c31 in __kernel_vsyscall () #1 0xb722ace1 in pthread_kill () from /lib/i386-linux-gnu/libpthread.so.0 #2 0x810673a6 in my_write_core (sig=11) at /home/buildbot/buildbot/build/mariadb-10.3.32/mysys/stacktrace.c:386 #3 0x8080d570 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/signal_handler.cc:355 #4 <signal handler called> #5 0x80735570 in I_P_List_iterator<TMP_TABLE_SHARE, I_P_List<TMP_TABLE_SHARE, All_tmp_table_shares, I_P_List_null_counter, I_P_List_no_push_back<TMP_TABLE_SHARE> > >::I_P_List_iterator (this=0xa4cfa84c, a=...) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_plist.h:179 #6 0x80733158 in THD::mark_tmp_tables_as_free_for_reuse (this=0xa6f77dd0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/temporary_tables.cc:749 #7 0x80464f0c in close_thread_tables (thd=0xa6f77dd0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_base.cc:806 #8 0x80504e35 in mysql_execute_command (thd=0xa6f77dd0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:6160 #9 0x80509856 in mysql_parse (thd=0xa6f77dd0, rawbuf=0xb0f2e6fb "COMMIT", length=6, parser_state=0xa4cfafa0, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/sql_parse.cc:7870 #10 0x8095216d in Query_log_event::do_apply_event (this=0xb0f21788, rgi=0xb0f23bb0, query_arg=0xb0f2e6fb "COMMIT", q_len_arg=6) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.cc:5700 #11 0x80950e1e in Query_log_event::do_apply_event (this=0xb0f21788, rgi=0xb0f23bb0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.cc:5377 #12 0x80416c87 in Log_event::apply_event (this=0xb0f21788, rgi=0xb0f23bb0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/log_event.h:1492 #13 0x80409907 in apply_event_and_update_pos_apply (ev=0xb0f21788, thd=0xa6f77dd0, rgi=0xb0f23bb0, reason=0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:3789 #14 0x8040a0bd in apply_event_and_update_pos_for_parallel (ev=0xb0f21788, thd=0xa6f77dd0, rgi=0xb0f23bb0) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/slave.cc:3981 #15 0x806d2e38 in rpt_handle_event (qev=0xb0f24558, rpt=0xb0f0dd50) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:51 #16 0x806d633a in handle_rpl_parallel_thread (arg=0xb0f0dd50) at /home/buildbot/buildbot/build/mariadb-10.3.32/sql/rpl_parallel.cc:1335 #17 0x80a49677 in pfs_spawn_thread (arg=0xb0f0e618) at /home/buildbot/buildbot/build/mariadb-10.3.32/storage/perfschema/pfs.cc:1869 #18 0xb72242b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #19 0xb714f16e in clone () from /lib/i386-linux-gnu/libc.so.6

Marko Mäkelä added a comment - 2021-12-03 14:26

Failure in a 10.7-based branch:

preview-10.7-MDEV-12933-provider-plugins d47ba02ecdf9babfde4a91b71b6b4a41
rpl.rpl_parallel_temptable 'innodb,stmt' w4 [ fail ]
Test ended at 2021-10-27 00:00:57

CURRENT_TEST: rpl.rpl_parallel_temptable
mysqltest: In included file "./include/stop_slave.inc":
included from /home/buildbot/buildbot/build/mariadb-10.7.1/mysql-test/suite/rpl/t/rpl_parallel_temptable.test at line 257:
At line 82: query 'STOP SLAVE' failed: <Unknown> (2013): Lost connection to server during query
…
2021-10-27 0:00:52 24 [Note] Error reading relay log event: slave SQL thread was killed
2021-10-27 0:00:52 24 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000004' at position 1367; GTID position '0-1-9,1-1-31,2-1-17', master: 127.0.0.1:16000
2021-10-27 0:00:52 23 [Note] Slave I/O thread exiting, read up to log 'master-bin.000004', position 1367; GTID position 1-1-32,0-1-9,2-1-17, master 127.0.0.1:16000
2021-10-27 0:00:52 30 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16000' in log 'master-bin.000004' at position 1367
2021-10-27 0:00:52 30 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16000',replication starts at GTID position '1-1-31,0-1-9,2-1-17'
2021-10-27 0:00:52 31 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000004' at position 1367, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-9,1-1-31,2-1-17'
=================================================================
==63203==ERROR: AddressSanitizer: heap-use-after-free on address 0x6180000b7f98 at pc 0x56167d21c8c0 bp 0x7ff29435f0a0 sp 0x7ff29435f090
WRITE of size 8 at 0x6180000b7f98 thread T43
#0 0x56167d21c8bf in std::__atomic_base<long>::store(long, std::memory_order) /usr/include/c++/9/bits/atomic_base.h:397
#1 0x56167d21c8bf in Atomic_relaxed<long>::store(long, std::memory_order) /home/buildbot/buildbot/build/mariadb-10.7.1/include/my_atomic_wrapper.h:47
#2 0x56167d218c45 in Atomic_relaxed<long>::operator=(long) /home/buildbot/buildbot/build/mariadb-10.7.1/include/my_atomic_wrapper.h:49
#3 0x56167d6d24a8 in trx_t::commit_tables() /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/trx/trx0trx.cc:1185
#4 0x56167d6ca0c6 in trx_t::commit_in_memory(mtr_t const*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/trx/trx0trx.cc:1287
#5 0x56167d6ca0c6 in trx_t::commit_low(mtr_t*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/trx/trx0trx.cc:1448
#6 0x56167d6cae7a in trx_t::commit_persist() /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/trx/trx0trx.cc:1462
2021-10-27 0:00:54 31 [Note] Error reading relay log event: slave SQL thread was killed
#7 0x56167d6cafd3 in trx_t::commit() /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/trx/trx0trx.cc:1468
#8 0x56167d6cb7e5 in trx_commit_for_mysql(trx_t*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/trx/trx0trx.cc:1584
#9 0x56167d19822e in innobase_commit_low(trx_t*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/handler/ha_innodb.cc:4308
#10 0x56167d19896f in innobase_commit_ordered_2 /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/handler/ha_innodb.cc:4414
#11 0x56167d199439 in innobase_commit /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/handler/ha_innodb.cc:4518
#12 0x56167c714d1a in commit_one_phase_2 /home/buildbot/buildbot/build/mariadb-10.7.1/sql/handler.cc:2052
#13 0x56167c71489a in ha_commit_one_phase(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/handler.cc:2005
#14 0x56167c712bda in ha_commit_trans(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/handler.cc:1799
#15 0x56167c341744 in trans_commit(THD*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/transaction.cc:266
#16 0x56167be7a513 in mysql_execute_command(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:5617
#17 0x56167be8a592 in mysql_parse(THD, char, unsigned int, Parser_state*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:8028
#18 0x56167cae4896 in Query_log_event::do_apply_event(rpl_group_info, char const, unsigned int) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event_server.cc:1916
#19 0x56167cae1ac0 in Query_log_event::do_apply_event(rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event_server.cc:1591
#20 0x56167bbeeae6 in Log_event::apply_event(rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event.h:1516
#21 0x56167bbd06d7 in apply_event_and_update_pos_apply /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:3881
#22 0x56167bbd1463 in apply_event_and_update_pos_for_parallel(Log_event, THD, rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:4067
#23 0x56167c470788 in rpt_handle_event /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:62
#24 0x56167c478400 in handle_rpl_parallel_thread /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1376
#25 0x56167cfa2de0 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201
#26 0x7ff2a43d7608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)
#27 0x7ff2a3fad292 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x122292)

0x6180000b7f98 is located 792 bytes inside of 824-byte region [0x6180000b7c80,0x6180000b7fb8)
freed by thread T40 here:
#0 0x7ff2a46417cf in __interceptor_free (/lib/x86_64-linux-gnu/libasan.so.5+0x10d7cf)
#1 0x56167d20b155 in ut_allocator<unsigned char, true>::deallocate(unsigned char*, unsigned long) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/ut0new.h:424
#2 0x56167d3f4ed5 in mem_heap_block_free(mem_block_info_t, mem_block_info_t) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/mem/mem0mem.cc:416
#3 0x56167d8808fe in mem_heap_free /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/mem0mem.ic:419
#4 0x56167d883e6f in dict_mem_table_free(dict_table_t*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0mem.cc:234
#5 0x56167d1ce11a in ha_innobase::delete_table(char const*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/handler/ha_innodb.cc:13414
#6 0x56167c70cee4 in hton_drop_table /home/buildbot/buildbot/build/mariadb-10.7.1/sql/handler.cc:576
#7 0x56167c553395 in THD::rm_temporary_table(handlerton, char const) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/temporary_tables.cc:703
#8 0x56167c558433 in THD::free_tmp_table_share(TMP_TABLE_SHARE*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/temporary_tables.cc:1462
#9 0x56167c552f86 in THD::drop_temporary_table(TABLE, bool, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/temporary_tables.cc:669
#10 0x56167c11adbf in mysql_rm_table_no_locks(THD, TABLE_LIST, st_mysql_const_lex_string const, st_ddl_log_state, bool, bool, bool, bool, bool, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_table.cc:1401
#11 0x56167c1197ea in mysql_rm_table(THD, TABLE_LIST, bool, bool, bool, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_table.cc:1148
#12 0x56167be7582d in mysql_execute_command(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:4947
#13 0x56167be8a592 in mysql_parse(THD, char, unsigned int, Parser_state*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:8028
#14 0x56167cae4896 in Query_log_event::do_apply_event(rpl_group_info, char const, unsigned int) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event_server.cc:1916
#15 0x56167cae1ac0 in Query_log_event::do_apply_event(rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event_server.cc:1591
#16 0x56167bbeeae6 in Log_event::apply_event(rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event.h:1516
#17 0x56167bbd06d7 in apply_event_and_update_pos_apply /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:3881
#18 0x56167bbd1463 in apply_event_and_update_pos_for_parallel(Log_event, THD, rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:4067
#19 0x56167c470788 in rpt_handle_event /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:62
#20 0x56167c478400 in handle_rpl_parallel_thread /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1376
#21 0x56167cfa2de0 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201
#22 0x7ff2a43d7608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)

previously allocated by thread T44 here:
#0 0x7ff2a4641bc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
#1 0x56167d20ac40 in ut_allocator<unsigned char, true>::allocate(unsigned long, unsigned char const*, unsigned int, bool, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/ut0new.h:375
#2 0x56167d3f4179 in mem_heap_create_block_func(mem_block_info_t, unsigned long, char const, unsigned int, unsigned long) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/mem/mem0mem.cc:277
#3 0x56167d3f4ac9 in mem_heap_add_block(mem_block_info_t*, unsigned long) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/mem/mem0mem.cc:378
#4 0x56167d88058b in mem_heap_alloc /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/mem0mem.ic:193
#5 0x56167d88039f in mem_heap_zalloc /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/mem0mem.ic:162
#6 0x56167d882b65 in dict_table_t::create(st_::span<char const> const&, fil_space_t*, unsigned long, unsigned long, unsigned long, unsigned long) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0mem.cc:148
#7 0x56167d2031af in create_table_info_t::create_table_def() (/home/buildbot/buildbot/build/mariadb-10.7.1/sql/mariadbd+0x2f5c1af)
#8 0x56167d1c95ba in create_table_info_t::create_table(bool) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/handler/ha_innodb.cc:12626
#9 0x56167d208cfc in ha_innobase::create(char const, TABLE, HA_CREATE_INFO, bool, trx_t) (/home/buildbot/buildbot/build/mariadb-10.7.1/sql/mariadbd+0x2f61cfc)
#10 0x56167d1cc06d in ha_innobase::create(char const, TABLE, HA_CREATE_INFO*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/handler/ha_innodb.cc:13222
#11 0x56167c73393f in handler::ha_create(char const, TABLE, HA_CREATE_INFO*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/handler.cc:5419
#12 0x56167c73804d in ha_create_table(THD, char const, char const, char const, HA_CREATE_INFO, st_mysql_const_unsigned_lex_string, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/handler.cc:5884
#13 0x56167c13368f in create_table_impl(THD, st_ddl_log_state, st_ddl_log_state, st_mysql_const_lex_string const&, st_mysql_const_lex_string const&, st_mysql_const_lex_string const&, st_mysql_const_lex_string const&, st_mysql_const_lex_string const&, DDL_options_st, HA_CREATE_INFO, Alter_info, int, bool, st_key*, unsigned int, st_mysql_const_unsigned_lex_string*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_table.cc:4547
#14 0x56167c13416e in mysql_create_table_no_lock(THD, st_ddl_log_state, st_ddl_log_state, st_mysql_const_lex_string const, st_mysql_const_lex_string const, Table_specification_st, Alter_info, bool, int, TABLE_LIST*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_table.cc:4646
#15 0x56167c134e9b in mysql_create_table(THD, TABLE_LIST, Table_specification_st, Alter_info) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_table.cc:4758
#16 0x56167c169f95 in Sql_cmd_create_table_like::execute(THD*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_table.cc:11871
#17 0x56167be7cd6d in mysql_execute_command(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:5989
#18 0x56167be8a592 in mysql_parse(THD, char, unsigned int, Parser_state*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:8028
#19 0x56167cae4896 in Query_log_event::do_apply_event(rpl_group_info, char const, unsigned int) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event_server.cc:1916
#20 0x56167cae1ac0 in Query_log_event::do_apply_event(rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event_server.cc:1591
#21 0x56167bbeeae6 in Log_event::apply_event(rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event.h:1516
#22 0x56167bbd06d7 in apply_event_and_update_pos_apply /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:3881
#23 0x56167bbd1463 in apply_event_and_update_pos_for_parallel(Log_event, THD, rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:4067
#24 0x56167c470788 in rpt_handle_event /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:62
#25 0x56167c478400 in handle_rpl_parallel_thread /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1376
#26 0x56167cfa2de0 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201
#27 0x7ff2a43d7608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)

Thread T43 created by T39 here:
#0 0x7ff2a456e805 in pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805)
#1 0x56167cf9dd96 in my_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/my_thread.h:48
#2 0x56167cfa31d3 in pfs_spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2252
#3 0x56167c46f0e5 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/include/mysql/psi/mysql_thread.h:1139
#4 0x56167c479d5c in rpl_parallel_change_thread_count /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1620
#5 0x56167c47ae50 in rpl_parallel_activate_pool(rpl_parallel_thread_pool*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1761
#6 0x56167bbd891f in handle_slave_sql /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:5274
#7 0x56167cfa2de0 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201
#8 0x7ff2a43d7608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)

Thread T39 created by T13 here:
#0 0x7ff2a456e805 in pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805)
#1 0x56167cf9dd96 in my_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/my_thread.h:48
#2 0x56167cfa31d3 in pfs_spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2252
#3 0x56167bbb6b66 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/include/mysql/psi/mysql_thread.h:1139
#4 0x56167bbbee9b in start_slave_thread(unsigned int, void* ()(void), st_mysql_mutex, st_mysql_mutex, st_mysql_cond, unsigned int volatile, unsigned long volatile, Master_info) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:1150
#5 0x56167bbbf9e1 in start_slave_threads(THD, bool, bool, Master_info, char const, char const, int) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:1276
#6 0x56167bf48eb1 in start_slave(THD, Master_info, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_repl.cc:3274
#7 0x56167be6f6df in mysql_execute_command(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:4245
#8 0x56167be8a592 in mysql_parse(THD, char, unsigned int, Parser_state*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:8028
#9 0x56167be606a1 in dispatch_command(enum_server_command, THD, char, unsigned int, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:1894
#10 0x56167be5d3c5 in do_command(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:1402
#11 0x56167c2fd96b in do_handle_one_connection(CONNECT*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_connect.cc:1418
#12 0x56167c2fd1fc in handle_one_connection /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_connect.cc:1312
#13 0x56167cfa2de0 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201
#14 0x7ff2a43d7608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)

Thread T13 created by T0 here:
#0 0x7ff2a456e805 in pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805)
#1 0x56167cf9dd96 in my_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/my_thread.h:48
#2 0x56167cfa31d3 in pfs_spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2252
#3 0x56167baa6ce8 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/include/mysql/psi/mysql_thread.h:1139
#4 0x56167babeaad in create_thread_to_handle_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/mysqld.cc:5962
#5 0x56167babf129 in create_new_thread(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/mysqld.cc:6021
#6 0x56167babf496 in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/mysqld.cc:6083
#7 0x56167babfe7c in handle_connections_sockets() /home/buildbot/buildbot/build/mariadb-10.7.1/sql/mysqld.cc:6207
#8 0x56167babe2ae in mysqld_main(int, char**) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/mysqld.cc:5857
#9 0x56167baa600c in main /home/buildbot/buildbot/build/mariadb-10.7.1/sql/main.cc:34
#10 0x7ff2a3eb20b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)

Thread T40 created by T39 here:
#0 0x7ff2a456e805 in pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805)
#1 0x56167cf9dd96 in my_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/my_thread.h:48
#2 0x56167cfa31d3 in pfs_spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2252
#3 0x56167c46f0e5 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/include/mysql/psi/mysql_thread.h:1139
#4 0x56167c479d5c in rpl_parallel_change_thread_count /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1620
#5 0x56167c47ae50 in rpl_parallel_activate_pool(rpl_parallel_thread_pool*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1761
#6 0x56167bbd891f in handle_slave_sql /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:5274
#7 0x56167cfa2de0 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201
#8 0x7ff2a43d7608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)

Thread T44 created by T39 here:
#0 0x7ff2a456e805 in pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805)
#1 0x56167cf9dd96 in my_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/my_thread.h:48
#2 0x56167cfa31d3 in pfs_spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2252
#3 0x56167c46f0e5 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/include/mysql/psi/mysql_thread.h:1139
#4 0x56167c479d5c in rpl_parallel_change_thread_count /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1620
#5 0x56167c47ae50 in rpl_parallel_activate_pool(rpl_parallel_thread_pool*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1761
#6 0x56167bbd891f in handle_slave_sql /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:5274
#7 0x56167cfa2de0 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201
#8 0x7ff2a43d7608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)

Marko Mäkelä added a comment - 2021-12-03 14:26 Failure in a 10.7-based branch : preview-10.7-MDEV-12933-provider-plugins d47ba02ecdf9babfde4a91b71b6b4a41 rpl.rpl_parallel_temptable 'innodb,stmt' w4 [ fail ] Test ended at 2021-10-27 00:00:57 CURRENT_TEST: rpl.rpl_parallel_temptable mysqltest: In included file "./include/stop_slave.inc": included from /home/buildbot/buildbot/build/mariadb-10.7.1/mysql-test/suite/rpl/t/rpl_parallel_temptable.test at line 257: At line 82: query 'STOP SLAVE' failed: <Unknown> (2013): Lost connection to server during query … 2021-10-27 0:00:52 24 [Note] Error reading relay log event: slave SQL thread was killed 2021-10-27 0:00:52 24 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000004' at position 1367; GTID position '0-1-9,1-1-31,2-1-17', master: 127.0.0.1:16000 2021-10-27 0:00:52 23 [Note] Slave I/O thread exiting, read up to log 'master-bin.000004', position 1367; GTID position 1-1-32,0-1-9,2-1-17, master 127.0.0.1:16000 2021-10-27 0:00:52 30 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:16000' in log 'master-bin.000004' at position 1367 2021-10-27 0:00:52 30 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16000',replication starts at GTID position '1-1-31,0-1-9,2-1-17' 2021-10-27 0:00:52 31 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000004' at position 1367, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-9,1-1-31,2-1-17' ================================================================= ==63203==ERROR: AddressSanitizer: heap-use-after-free on address 0x6180000b7f98 at pc 0x56167d21c8c0 bp 0x7ff29435f0a0 sp 0x7ff29435f090 WRITE of size 8 at 0x6180000b7f98 thread T43 #0 0x56167d21c8bf in std::__atomic_base<long>::store(long, std::memory_order) /usr/include/c++/9/bits/atomic_base.h:397 #1 0x56167d21c8bf in Atomic_relaxed<long>::store(long, std::memory_order) /home/buildbot/buildbot/build/mariadb-10.7.1/include/my_atomic_wrapper.h:47 #2 0x56167d218c45 in Atomic_relaxed<long>::operator=(long) /home/buildbot/buildbot/build/mariadb-10.7.1/include/my_atomic_wrapper.h:49 #3 0x56167d6d24a8 in trx_t::commit_tables() /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/trx/trx0trx.cc:1185 #4 0x56167d6ca0c6 in trx_t::commit_in_memory(mtr_t const*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/trx/trx0trx.cc:1287 #5 0x56167d6ca0c6 in trx_t::commit_low(mtr_t*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/trx/trx0trx.cc:1448 #6 0x56167d6cae7a in trx_t::commit_persist() /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/trx/trx0trx.cc:1462 2021-10-27 0:00:54 31 [Note] Error reading relay log event: slave SQL thread was killed #7 0x56167d6cafd3 in trx_t::commit() /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/trx/trx0trx.cc:1468 #8 0x56167d6cb7e5 in trx_commit_for_mysql(trx_t*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/trx/trx0trx.cc:1584 #9 0x56167d19822e in innobase_commit_low(trx_t*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/handler/ha_innodb.cc:4308 #10 0x56167d19896f in innobase_commit_ordered_2 /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/handler/ha_innodb.cc:4414 #11 0x56167d199439 in innobase_commit /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/handler/ha_innodb.cc:4518 #12 0x56167c714d1a in commit_one_phase_2 /home/buildbot/buildbot/build/mariadb-10.7.1/sql/handler.cc:2052 #13 0x56167c71489a in ha_commit_one_phase(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/handler.cc:2005 #14 0x56167c712bda in ha_commit_trans(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/handler.cc:1799 #15 0x56167c341744 in trans_commit(THD*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/transaction.cc:266 #16 0x56167be7a513 in mysql_execute_command(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:5617 #17 0x56167be8a592 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:8028 #18 0x56167cae4896 in Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event_server.cc:1916 #19 0x56167cae1ac0 in Query_log_event::do_apply_event(rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event_server.cc:1591 #20 0x56167bbeeae6 in Log_event::apply_event(rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event.h:1516 #21 0x56167bbd06d7 in apply_event_and_update_pos_apply /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:3881 #22 0x56167bbd1463 in apply_event_and_update_pos_for_parallel(Log_event*, THD*, rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:4067 #23 0x56167c470788 in rpt_handle_event /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:62 #24 0x56167c478400 in handle_rpl_parallel_thread /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1376 #25 0x56167cfa2de0 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201 #26 0x7ff2a43d7608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608) #27 0x7ff2a3fad292 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x122292) 0x6180000b7f98 is located 792 bytes inside of 824-byte region [0x6180000b7c80,0x6180000b7fb8) freed by thread T40 here: #0 0x7ff2a46417cf in __interceptor_free (/lib/x86_64-linux-gnu/libasan.so.5+0x10d7cf) #1 0x56167d20b155 in ut_allocator<unsigned char, true>::deallocate(unsigned char*, unsigned long) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/ut0new.h:424 #2 0x56167d3f4ed5 in mem_heap_block_free(mem_block_info_t*, mem_block_info_t*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/mem/mem0mem.cc:416 #3 0x56167d8808fe in mem_heap_free /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/mem0mem.ic:419 #4 0x56167d883e6f in dict_mem_table_free(dict_table_t*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0mem.cc:234 #5 0x56167d1ce11a in ha_innobase::delete_table(char const*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/handler/ha_innodb.cc:13414 #6 0x56167c70cee4 in hton_drop_table /home/buildbot/buildbot/build/mariadb-10.7.1/sql/handler.cc:576 #7 0x56167c553395 in THD::rm_temporary_table(handlerton*, char const*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/temporary_tables.cc:703 #8 0x56167c558433 in THD::free_tmp_table_share(TMP_TABLE_SHARE*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/temporary_tables.cc:1462 #9 0x56167c552f86 in THD::drop_temporary_table(TABLE*, bool*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/temporary_tables.cc:669 #10 0x56167c11adbf in mysql_rm_table_no_locks(THD*, TABLE_LIST*, st_mysql_const_lex_string const*, st_ddl_log_state*, bool, bool, bool, bool, bool, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_table.cc:1401 #11 0x56167c1197ea in mysql_rm_table(THD*, TABLE_LIST*, bool, bool, bool, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_table.cc:1148 #12 0x56167be7582d in mysql_execute_command(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:4947 #13 0x56167be8a592 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:8028 #14 0x56167cae4896 in Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event_server.cc:1916 #15 0x56167cae1ac0 in Query_log_event::do_apply_event(rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event_server.cc:1591 #16 0x56167bbeeae6 in Log_event::apply_event(rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event.h:1516 #17 0x56167bbd06d7 in apply_event_and_update_pos_apply /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:3881 #18 0x56167bbd1463 in apply_event_and_update_pos_for_parallel(Log_event*, THD*, rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:4067 #19 0x56167c470788 in rpt_handle_event /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:62 #20 0x56167c478400 in handle_rpl_parallel_thread /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1376 #21 0x56167cfa2de0 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201 #22 0x7ff2a43d7608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608) previously allocated by thread T44 here: #0 0x7ff2a4641bc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8) #1 0x56167d20ac40 in ut_allocator<unsigned char, true>::allocate(unsigned long, unsigned char const*, unsigned int, bool, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/ut0new.h:375 #2 0x56167d3f4179 in mem_heap_create_block_func(mem_block_info_t*, unsigned long, char const*, unsigned int, unsigned long) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/mem/mem0mem.cc:277 #3 0x56167d3f4ac9 in mem_heap_add_block(mem_block_info_t*, unsigned long) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/mem/mem0mem.cc:378 #4 0x56167d88058b in mem_heap_alloc /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/mem0mem.ic:193 #5 0x56167d88039f in mem_heap_zalloc /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/mem0mem.ic:162 #6 0x56167d882b65 in dict_table_t::create(st_::span<char const> const&, fil_space_t*, unsigned long, unsigned long, unsigned long, unsigned long) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0mem.cc:148 #7 0x56167d2031af in create_table_info_t::create_table_def() (/home/buildbot/buildbot/build/mariadb-10.7.1/sql/mariadbd+0x2f5c1af) #8 0x56167d1c95ba in create_table_info_t::create_table(bool) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/handler/ha_innodb.cc:12626 #9 0x56167d208cfc in ha_innobase::create(char const*, TABLE*, HA_CREATE_INFO*, bool, trx_t*) (/home/buildbot/buildbot/build/mariadb-10.7.1/sql/mariadbd+0x2f61cfc) #10 0x56167d1cc06d in ha_innobase::create(char const*, TABLE*, HA_CREATE_INFO*) /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/handler/ha_innodb.cc:13222 #11 0x56167c73393f in handler::ha_create(char const*, TABLE*, HA_CREATE_INFO*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/handler.cc:5419 #12 0x56167c73804d in ha_create_table(THD*, char const*, char const*, char const*, HA_CREATE_INFO*, st_mysql_const_unsigned_lex_string*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/handler.cc:5884 #13 0x56167c13368f in create_table_impl(THD*, st_ddl_log_state*, st_ddl_log_state*, st_mysql_const_lex_string const&, st_mysql_const_lex_string const&, st_mysql_const_lex_string const&, st_mysql_const_lex_string const&, st_mysql_const_lex_string const&, DDL_options_st, HA_CREATE_INFO*, Alter_info*, int, bool*, st_key**, unsigned int*, st_mysql_const_unsigned_lex_string*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_table.cc:4547 #14 0x56167c13416e in mysql_create_table_no_lock(THD*, st_ddl_log_state*, st_ddl_log_state*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, Table_specification_st*, Alter_info*, bool*, int, TABLE_LIST*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_table.cc:4646 #15 0x56167c134e9b in mysql_create_table(THD*, TABLE_LIST*, Table_specification_st*, Alter_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_table.cc:4758 #16 0x56167c169f95 in Sql_cmd_create_table_like::execute(THD*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_table.cc:11871 #17 0x56167be7cd6d in mysql_execute_command(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:5989 #18 0x56167be8a592 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:8028 #19 0x56167cae4896 in Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event_server.cc:1916 #20 0x56167cae1ac0 in Query_log_event::do_apply_event(rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event_server.cc:1591 #21 0x56167bbeeae6 in Log_event::apply_event(rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/log_event.h:1516 #22 0x56167bbd06d7 in apply_event_and_update_pos_apply /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:3881 #23 0x56167bbd1463 in apply_event_and_update_pos_for_parallel(Log_event*, THD*, rpl_group_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:4067 #24 0x56167c470788 in rpt_handle_event /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:62 #25 0x56167c478400 in handle_rpl_parallel_thread /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1376 #26 0x56167cfa2de0 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201 #27 0x7ff2a43d7608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608) Thread T43 created by T39 here: #0 0x7ff2a456e805 in pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805) #1 0x56167cf9dd96 in my_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/my_thread.h:48 #2 0x56167cfa31d3 in pfs_spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2252 #3 0x56167c46f0e5 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/include/mysql/psi/mysql_thread.h:1139 #4 0x56167c479d5c in rpl_parallel_change_thread_count /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1620 #5 0x56167c47ae50 in rpl_parallel_activate_pool(rpl_parallel_thread_pool*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1761 #6 0x56167bbd891f in handle_slave_sql /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:5274 #7 0x56167cfa2de0 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201 #8 0x7ff2a43d7608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608) Thread T39 created by T13 here: #0 0x7ff2a456e805 in pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805) #1 0x56167cf9dd96 in my_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/my_thread.h:48 #2 0x56167cfa31d3 in pfs_spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2252 #3 0x56167bbb6b66 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/include/mysql/psi/mysql_thread.h:1139 #4 0x56167bbbee9b in start_slave_thread(unsigned int, void* (*)(void*), st_mysql_mutex*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, unsigned long volatile*, Master_info*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:1150 #5 0x56167bbbf9e1 in start_slave_threads(THD*, bool, bool, Master_info*, char const*, char const*, int) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:1276 #6 0x56167bf48eb1 in start_slave(THD*, Master_info*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_repl.cc:3274 #7 0x56167be6f6df in mysql_execute_command(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:4245 #8 0x56167be8a592 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:8028 #9 0x56167be606a1 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:1894 #10 0x56167be5d3c5 in do_command(THD*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:1402 #11 0x56167c2fd96b in do_handle_one_connection(CONNECT*, bool) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_connect.cc:1418 #12 0x56167c2fd1fc in handle_one_connection /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_connect.cc:1312 #13 0x56167cfa2de0 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201 #14 0x7ff2a43d7608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608) Thread T13 created by T0 here: #0 0x7ff2a456e805 in pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805) #1 0x56167cf9dd96 in my_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/my_thread.h:48 #2 0x56167cfa31d3 in pfs_spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2252 #3 0x56167baa6ce8 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/include/mysql/psi/mysql_thread.h:1139 #4 0x56167babeaad in create_thread_to_handle_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/mysqld.cc:5962 #5 0x56167babf129 in create_new_thread(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/mysqld.cc:6021 #6 0x56167babf496 in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/mysqld.cc:6083 #7 0x56167babfe7c in handle_connections_sockets() /home/buildbot/buildbot/build/mariadb-10.7.1/sql/mysqld.cc:6207 #8 0x56167babe2ae in mysqld_main(int, char**) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/mysqld.cc:5857 #9 0x56167baa600c in main /home/buildbot/buildbot/build/mariadb-10.7.1/sql/main.cc:34 #10 0x7ff2a3eb20b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) Thread T40 created by T39 here: #0 0x7ff2a456e805 in pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805) #1 0x56167cf9dd96 in my_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/my_thread.h:48 #2 0x56167cfa31d3 in pfs_spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2252 #3 0x56167c46f0e5 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/include/mysql/psi/mysql_thread.h:1139 #4 0x56167c479d5c in rpl_parallel_change_thread_count /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1620 #5 0x56167c47ae50 in rpl_parallel_activate_pool(rpl_parallel_thread_pool*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1761 #6 0x56167bbd891f in handle_slave_sql /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:5274 #7 0x56167cfa2de0 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201 #8 0x7ff2a43d7608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608) Thread T44 created by T39 here: #0 0x7ff2a456e805 in pthread_create (/lib/x86_64-linux-gnu/libasan.so.5+0x3a805) #1 0x56167cf9dd96 in my_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/my_thread.h:48 #2 0x56167cfa31d3 in pfs_spawn_thread_v1 /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2252 #3 0x56167c46f0e5 in inline_mysql_thread_create /home/buildbot/buildbot/build/mariadb-10.7.1/include/mysql/psi/mysql_thread.h:1139 #4 0x56167c479d5c in rpl_parallel_change_thread_count /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1620 #5 0x56167c47ae50 in rpl_parallel_activate_pool(rpl_parallel_thread_pool*) /home/buildbot/buildbot/build/mariadb-10.7.1/sql/rpl_parallel.cc:1761 #6 0x56167bbd891f in handle_slave_sql /home/buildbot/buildbot/build/mariadb-10.7.1/sql/slave.cc:5274 #7 0x56167cfa2de0 in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201 #8 0x7ff2a43d7608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)

Marko Mäkelä made changes - 2021-12-03 14:26

Fix Version/s		10.6 [ 24028 ]
Fix Version/s		10.7 [ 24805 ]
Affects Version/s		10.7 [ 24805 ]

Sergei Golubchik made changes - 2021-12-06 21:32

Workflow

MariaDB v3 [ 76383 ]

MariaDB v4 [ 140016 ]

Marko Mäkelä made changes - 2022-02-08 16:34

Link

This issue relates to ~~MDEV-12459~~ [ ~~MDEV-12459~~ ]

Ralf Gebhardt made changes - 2022-08-04 09:20

Fix Version/s

10.2 [ 14601 ]

Julien Fritsch made changes - 2023-03-03 17:28

Fix Version/s

10.7 [ 24805 ]

Julien Fritsch made changes - 2023-04-27 14:24

Fix Version/s

10.3 [ 22126 ]

Andrei Elkin made changes - 2023-08-03 17:00

Priority

Major [ 3 ]

Critical [ 2 ]

Andrei Elkin added a comment - 2023-09-06 11:44 - edited

julien.fritsch, the same comments apply here. The ticket can't relate to the support issue.
Instead of to set to NEED_FEEDBACK (as it was going to be done to ~~MDEV-25064~~ and which looks as a duplicate but I can't confirm) I just removed the CS reference.

Andrei Elkin added a comment - 2023-09-06 11:44 - edited julien.fritsch , the same comments apply here. The ticket can't relate to the support issue. Instead of to set to NEED_FEEDBACK (as it was going to be done to MDEV-25064 and which looks as a duplicate but I can't confirm) I just removed the CS reference.

Michael Widenius added a comment - 2023-09-06 12:14 - edited

Andrei, please have someone create a test case to prove/disapprove Marko statement and also if it could be the cause of the problem.

"Starting with MariaDB Server 10.2.2, InnoDB assumes that temporary tables are private to the connection that created the table. In ~~MDEV-25064~~, we have direct evidence that parallel replication is executing CREATE TEMPORARY TABLE in one thread, executing a transaction on the temporary table in another thread, and executing DROP TEMPORARY TABLE in yet another thread (while the transaction that is using the temporary table is still active)."

Execution of temporary tables transaction on the slave cannot be done in parallel.
In theory it should work that CRETATE/DROP is done by different threads as long as the DROP is scheduled to be
done after any transaction.
Note that there are no MDL locks on temporary tables, so replication needs to ensure that there are no concurrent usage of the temporary table.
As a transaction can in theory use multiple temporary tables, possible created by different threads, it is not possible to schedule the query to the original thread. This means that any solution has to depend on that any queries using temporary table queries are executed strictly in order.

Michael Widenius added a comment - 2023-09-06 12:14 - edited Andrei, please have someone create a test case to prove/disapprove Marko statement and also if it could be the cause of the problem. "Starting with MariaDB Server 10.2.2, InnoDB assumes that temporary tables are private to the connection that created the table. In MDEV-25064 , we have direct evidence that parallel replication is executing CREATE TEMPORARY TABLE in one thread, executing a transaction on the temporary table in another thread, and executing DROP TEMPORARY TABLE in yet another thread (while the transaction that is using the temporary table is still active)." Execution of temporary tables transaction on the slave cannot be done in parallel. In theory it should work that CRETATE/DROP is done by different threads as long as the DROP is scheduled to be done after any transaction. Note that there are no MDL locks on temporary tables, so replication needs to ensure that there are no concurrent usage of the temporary table. As a transaction can in theory use multiple temporary tables, possible created by different threads, it is not possible to schedule the query to the original thread. This means that any solution has to depend on that any queries using temporary table queries are executed strictly in order.

Kristian Nielsen added a comment - 2023-09-06 15:30

I think I understand the problem here, which seems to be a real bug in parallel replication and statement-based replication with temporary tables.

Temporary tables are binlogged as non-transactional. Parallel replication will wait for all prior transactions to commit before starting such event groups. This means that each temporary table query will complete before the next one starts. This is intended to make temporary tables safe for parallel replication.

But in the stack traces for the crashes during STOP SLAVE, I see a problem. There is a non-transactional INSERT SELECT followed by a DDL DROP TEMPORARY TABLE. As an optimization, the DDL is allowed to start as soon as the the INSERT...SELECT starts to commit. That means that innobase_commit() can run in parallel with DROP TABLE. The problem is the call of mark_start_commit() in handle_rpl_parallel_thread().

There may be a similar problem that the COMMIT of one temptable transaction can run in parallel with the following (non-DDL) query, due to the call of thd_wakeup_subsequent_commits() in innobase_commit(). I will need to analyze further to know for sure.

I was not so far able to reproduce the crash, I will need more time to do so. But it seems clear that the problem described above is real, and very likely that it is the cause of these test failures.

I'm not yet sure of the correct way to fix this, will need to think on it. The optimisations to overlap part of the commit of one transaction with the following transaction or DDL is very important for performance, while statement-based parallel replication of temporary tables is not an important use case. So we need to find a solution that solves the problem and makes temptables safe for statement-based without regressing the performance of other workloads. (In mixed mode binlogging, this problem does not occur).

Kristian Nielsen added a comment - 2023-09-06 15:30 I think I understand the problem here, which seems to be a real bug in parallel replication and statement-based replication with temporary tables. Temporary tables are binlogged as non-transactional. Parallel replication will wait for all prior transactions to commit before starting such event groups. This means that each temporary table query will complete before the next one starts. This is intended to make temporary tables safe for parallel replication. But in the stack traces for the crashes during STOP SLAVE, I see a problem. There is a non-transactional INSERT SELECT followed by a DDL DROP TEMPORARY TABLE. As an optimization, the DDL is allowed to start as soon as the the INSERT...SELECT starts to commit. That means that innobase_commit() can run in parallel with DROP TABLE. The problem is the call of mark_start_commit() in handle_rpl_parallel_thread(). There may be a similar problem that the COMMIT of one temptable transaction can run in parallel with the following (non-DDL) query, due to the call of thd_wakeup_subsequent_commits() in innobase_commit(). I will need to analyze further to know for sure. I was not so far able to reproduce the crash, I will need more time to do so. But it seems clear that the problem described above is real, and very likely that it is the cause of these test failures. I'm not yet sure of the correct way to fix this, will need to think on it. The optimisations to overlap part of the commit of one transaction with the following transaction or DDL is very important for performance, while statement-based parallel replication of temporary tables is not an important use case. So we need to find a solution that solves the problem and makes temptables safe for statement-based without regressing the performance of other workloads. (In mixed mode binlogging, this problem does not occur).

Julien Fritsch made changes - 2023-09-06 17:10

Link

This issue blocks MDEV-17805 [ MDEV-17805 ]

Kristian Nielsen made changes - 2023-09-07 07:26

Assignee

Andrei Elkin [ elkin ]

Kristian Nielsen [ knielsen ]

Kristian Nielsen made changes - 2023-09-07 11:20

Summary

rpl.rpl_parallel_temptable fails in buildbot: crashed in close_thread_tables

rpl.rpl_parallel_temptable failure due to incorrect commit optimization of temptables

Kristian Nielsen made changes - 2023-09-07 11:26

Description

http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/5564/steps/test_6/logs/stdio
{noformat}
rpl.rpl_parallel_temptable 'innodb_plugin,stmt' w1 [ fail ]
Test ended at 2016-07-09 07:04:06

CURRENT_TEST: rpl.rpl_parallel_temptable
mysqltest: At line 132: failed in 'select master_pos_wait('master-bin.000002', 558, 300, '')': 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:
< snip >
INSERT INTO t2 VALUES (13);
INSERT INTO t3 SELECT a+32 FROM t4;
INSERT INTO t2 VALUES (14);
INSERT INTO t2 VALUES (15);
INSERT INTO t2 VALUES (16);
INSERT INTO t4 SELECT a+64 FROM t3;
INSERT INTO t2 VALUES (17);
INSERT INTO t2 VALUES (18);
INSERT INTO t2 VALUES (19);
INSERT INTO t3 SELECT a+128 FROM t4;
INSERT INTO t2 VALUES (20);
INSERT INTO t1 SELECT a, a MOD 7 FROM t3;
INSERT INTO t1 SELECT a, a MOD 7 FROM t4;
INSERT INTO t1 SELECT a, COUNT(*) FROM t2 GROUP BY a;
FLUSH TABLES;
SET SESSION debug_dbug="+d,crash_dispatch_command_before";
SELECT 1;
Got one of the listed errors
INSERT INTO t1 VALUES (0, 1);
include/start_slave.inc

More results from queries before failure can be found in /mnt/buildbot/build/mariadb-10.1.16/mysql-test/var/1/log/rpl_parallel_temptable.log

Server [mysqld.2 - pid: 26230, winpid: 26230, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
2016-07-09 7:03:47 3077719808 [Note] /mnt/buildbot/build/mariadb-10.1.16/sql/mysqld (mysqld 10.1.16-MariaDB-debug) starting as process 26231 ...
2016-07-09 7:03:47 3077719808 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4162)
2016-07-09 7:03:47 3077719808 [Note] Plugin 'file_key_management' is disabled.
2016-07-09 7:03:47 3077719808 [Note] Plugin 'BLACKHOLE' is disabled.
2016-07-09 7:03:47 3077719808 [Note] Plugin 'partition' is disabled.
2016-07-09 7:03:47 3077719808 [Note] Plugin 'ARCHIVE' is disabled.
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2016-07-09 7:03:48 3077719808 [Note] InnoDB: InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2016-07-09 7:03:48 3077719808 [Note] InnoDB: InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!
2016-07-09 7:03:48 3077719808 [Note] InnoDB: The InnoDB memory heap is disabled
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Memory barrier is not used
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Compressed tables use zlib 1.2.3.4
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Using Linux native AIO
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Using generic crc32 instructions
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Initializing buffer pool, size = 8.0M
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Completed initialization of buffer pool
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Highest supported file format is Barracuda.
2016-07-09 7:03:49 3077719808 [Note] InnoDB: 128 rollback segment(s) are active.
2016-07-09 7:03:49 3077719808 [Note] InnoDB: Waiting for purge to start
2016-07-09 7:03:49 3077719808 [Note] InnoDB: 5.6.31 started; log sequence number 1629517
2016-07-09 7:03:49 2858416960 [Note] InnoDB: Dumping buffer pool(s) not yet started
2016-07-09 7:03:49 3077719808 [Note] Plugin 'TEST_SQL_DISCOVERY' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'CONNECT' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'SPHINX' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'SEQUENCE' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'FEDERATED' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'XTRADB_READ_VIEW' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'XTRADB_RSEG' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'CHANGED_PAGE_BITMAPS' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'QUERY_CACHE_INFO' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'XTRADB_INTERNAL_HASH_TABLES' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_MUTEXES' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'FEEDBACK' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'METADATA_LOCK_INFO' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_CMP' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'QUERY_RESPONSE_TIME' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'LOCALES' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'unix_socket' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'QUERY_RESPONSE_TIME_AUDIT' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'rpl_semi_sync_slave' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'rpl_semi_sync_master' is disabled.
2016-07-09 7:03:49 3077719808 [Warning] /mnt/buildbot/build/mariadb-10.1.16/sql/mysqld: unknown option '--loose-skip-plugin-innodb-changed-pages'
2016-07-09 7:03:49 3077719808 [Note] Server socket created on IP: '127.0.0.1'.
2016-07-09 7:03:49 3077719808 [Note] /mnt/buildbot/build/mariadb-10.1.16/sql/mysqld: ready for connections.
Version: '10.1.16-MariaDB-debug' socket: '/mnt/buildbot/build/mariadb-10.1.16/mysql-test/var/tmp/1/mysqld.2.sock' port: 16041 Source distribution
2016-07-09 7:03:50 2918969152 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16040', master_log_file='master-bin.000001', master_log_pos='4'.
2016-07-09 7:03:50 2872782656 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16040',replication started in log 'master-bin.000001' at position 4
2016-07-09 7:03:50 2872479552 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 4, relay log './slave-relay-bin.000001' position: 4
2016-07-09 7:03:51 2872479552 [Note] Error reading relay log event: slave SQL thread was killed
2016-07-09 7:03:51 2872782656 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)
2016-07-09 7:03:51 2872782656 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 314
2016-07-09 7:03:51 2918969152 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port='16040', master_log_file='master-bin.000001', master_log_pos='314'. New state master_host='127.0.0.1', master_port='16040', master_log_file='master-bin.000001', master_log_pos='314'.
2016-07-09 7:03:51 2918969152 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos
2016-07-09 7:03:51 2872479552 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16040',replication starts at GTID position ''
2016-07-09 7:03:51 2824539968 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 314, relay log './slave-relay-bin.000001' position: 4; GTID position ''
2016-07-09 7:03:51 2824539968 [Note] Error reading relay log event: slave SQL thread was killed
2016-07-09 7:03:51 2872479552 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)
2016-07-09 7:03:51 2872479552 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 485; GTID position 0-1-1
2016-07-09 7:03:56 2824539968 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16040',replication starts at GTID position '0-1-1'
2016-07-09 7:03:56 2862300992 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 485, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-1'
160709 7:03:56 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.16-MariaDB-debug
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=2
max_threads=153
thread_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61813 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0xaa800670
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xaa9696b8 thread_stack 0x48400
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(my_print_stacktrace+0x29)[0x8b0fe39]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(handle_fatal_signal+0x53c)[0x8438b4c]
[0xe9b400]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_Z19close_thread_tablesP3THD+0x48f)[0x81f7c3f]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_Z21mysql_execute_commandP3THD+0x45b)[0x825582b]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x36c)[0x8261b2c]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0xcef)[0x8540d3f]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_info+0x31)[0x8542031]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x63d)[0x81c41ad]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld[0x83ad240]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(handle_rpl_parallel_thread+0x99b)[0x83b1e4b]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld[0x8619a8a]
/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0x1a3d4c]
/lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xa38ace]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0xb2d12cd3): COMMIT
Connection ID (thread ID): 18
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
{noformat}

It happens very rarely in buildbot, a handful of occasions over 2 years.
It's possible that the failure is caused by the same problem as was recently fixed in ~~MDEV-10219~~, but then the fix needs to be backported into 10.1 and possibly 10.0.

Temporary tables require strict single-threaded operation, there is no locking protecting concurrent access. For this reason, they are considered non-transactional for the purpose of parallel replication.

However, even non-transactional event groups have an optimisation that allow part of the commit of such event groups to run in parallel with following event groups. Thus, it was still possible for part of the commit codepath to run in parallel with another operation on the temporary table such as a SELECT or DML or even a DROP TEMPORARY TABLE. This can cause access of freed memory and probably other nasty effects.

The problem is the mark_start_commit() in handle_rpl_parallel_thread() just before the COMMIT event is applied, as well as calls of wakeup_subsequent_commit() in the binlog code and in innobase_commit(). These are too early at these points when temporary tables are replicated.

Since temporary table operations are not binlogged in MIXED or ROW mode, this problem affects only statement-based replication.

This is seen as occasional test failures of rpl.rpl_parallel_temptable as reported by Elena and others:

http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/5564/steps/test_6/logs/stdio
{noformat}
rpl.rpl_parallel_temptable 'innodb_plugin,stmt' w1 [ fail ]
Test ended at 2016-07-09 07:04:06

CURRENT_TEST: rpl.rpl_parallel_temptable
mysqltest: At line 132: failed in 'select master_pos_wait('master-bin.000002', 558, 300, '')': 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:
< snip >
INSERT INTO t2 VALUES (13);
INSERT INTO t3 SELECT a+32 FROM t4;
INSERT INTO t2 VALUES (14);
INSERT INTO t2 VALUES (15);
INSERT INTO t2 VALUES (16);
INSERT INTO t4 SELECT a+64 FROM t3;
INSERT INTO t2 VALUES (17);
INSERT INTO t2 VALUES (18);
INSERT INTO t2 VALUES (19);
INSERT INTO t3 SELECT a+128 FROM t4;
INSERT INTO t2 VALUES (20);
INSERT INTO t1 SELECT a, a MOD 7 FROM t3;
INSERT INTO t1 SELECT a, a MOD 7 FROM t4;
INSERT INTO t1 SELECT a, COUNT(*) FROM t2 GROUP BY a;
FLUSH TABLES;
SET SESSION debug_dbug="+d,crash_dispatch_command_before";
SELECT 1;
Got one of the listed errors
INSERT INTO t1 VALUES (0, 1);
include/start_slave.inc

More results from queries before failure can be found in /mnt/buildbot/build/mariadb-10.1.16/mysql-test/var/1/log/rpl_parallel_temptable.log

Server [mysqld.2 - pid: 26230, winpid: 26230, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
2016-07-09 7:03:47 3077719808 [Note] /mnt/buildbot/build/mariadb-10.1.16/sql/mysqld (mysqld 10.1.16-MariaDB-debug) starting as process 26231 ...
2016-07-09 7:03:47 3077719808 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4162)
2016-07-09 7:03:47 3077719808 [Note] Plugin 'file_key_management' is disabled.
2016-07-09 7:03:47 3077719808 [Note] Plugin 'BLACKHOLE' is disabled.
2016-07-09 7:03:47 3077719808 [Note] Plugin 'partition' is disabled.
2016-07-09 7:03:47 3077719808 [Note] Plugin 'ARCHIVE' is disabled.
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2016-07-09 7:03:48 3077719808 [Note] InnoDB: InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2016-07-09 7:03:48 3077719808 [Note] InnoDB: InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!
2016-07-09 7:03:48 3077719808 [Note] InnoDB: The InnoDB memory heap is disabled
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Memory barrier is not used
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Compressed tables use zlib 1.2.3.4
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Using Linux native AIO
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Using generic crc32 instructions
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Initializing buffer pool, size = 8.0M
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Completed initialization of buffer pool
2016-07-09 7:03:48 3077719808 [Note] InnoDB: Highest supported file format is Barracuda.
2016-07-09 7:03:49 3077719808 [Note] InnoDB: 128 rollback segment(s) are active.
2016-07-09 7:03:49 3077719808 [Note] InnoDB: Waiting for purge to start
2016-07-09 7:03:49 3077719808 [Note] InnoDB: 5.6.31 started; log sequence number 1629517
2016-07-09 7:03:49 2858416960 [Note] InnoDB: Dumping buffer pool(s) not yet started
2016-07-09 7:03:49 3077719808 [Note] Plugin 'TEST_SQL_DISCOVERY' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'CONNECT' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'SPHINX' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'SEQUENCE' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'FEDERATED' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'XTRADB_READ_VIEW' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'XTRADB_RSEG' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'CHANGED_PAGE_BITMAPS' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'QUERY_CACHE_INFO' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'XTRADB_INTERNAL_HASH_TABLES' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_MUTEXES' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'FEEDBACK' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'METADATA_LOCK_INFO' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_CMP' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'QUERY_RESPONSE_TIME' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'LOCALES' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'unix_socket' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'QUERY_RESPONSE_TIME_AUDIT' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'rpl_semi_sync_slave' is disabled.
2016-07-09 7:03:49 3077719808 [Note] Plugin 'rpl_semi_sync_master' is disabled.
2016-07-09 7:03:49 3077719808 [Warning] /mnt/buildbot/build/mariadb-10.1.16/sql/mysqld: unknown option '--loose-skip-plugin-innodb-changed-pages'
2016-07-09 7:03:49 3077719808 [Note] Server socket created on IP: '127.0.0.1'.
2016-07-09 7:03:49 3077719808 [Note] /mnt/buildbot/build/mariadb-10.1.16/sql/mysqld: ready for connections.
Version: '10.1.16-MariaDB-debug' socket: '/mnt/buildbot/build/mariadb-10.1.16/mysql-test/var/tmp/1/mysqld.2.sock' port: 16041 Source distribution
2016-07-09 7:03:50 2918969152 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16040', master_log_file='master-bin.000001', master_log_pos='4'.
2016-07-09 7:03:50 2872782656 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16040',replication started in log 'master-bin.000001' at position 4
2016-07-09 7:03:50 2872479552 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 4, relay log './slave-relay-bin.000001' position: 4
2016-07-09 7:03:51 2872479552 [Note] Error reading relay log event: slave SQL thread was killed
2016-07-09 7:03:51 2872782656 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)
2016-07-09 7:03:51 2872782656 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 314
2016-07-09 7:03:51 2918969152 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port='16040', master_log_file='master-bin.000001', master_log_pos='314'. New state master_host='127.0.0.1', master_port='16040', master_log_file='master-bin.000001', master_log_pos='314'.
2016-07-09 7:03:51 2918969152 [Note] Previous Using_Gtid=No. New Using_Gtid=Current_Pos
2016-07-09 7:03:51 2872479552 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16040',replication starts at GTID position ''
2016-07-09 7:03:51 2824539968 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 314, relay log './slave-relay-bin.000001' position: 4; GTID position ''
2016-07-09 7:03:51 2824539968 [Note] Error reading relay log event: slave SQL thread was killed
2016-07-09 7:03:51 2872479552 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)
2016-07-09 7:03:51 2872479552 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 485; GTID position 0-1-1
2016-07-09 7:03:56 2824539968 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16040',replication starts at GTID position '0-1-1'
2016-07-09 7:03:56 2862300992 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 485, relay log './slave-relay-bin.000001' position: 4; GTID position '0-1-1'
160709 7:03:56 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.16-MariaDB-debug
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=2
max_threads=153
thread_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61813 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0xaa800670
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xaa9696b8 thread_stack 0x48400
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(my_print_stacktrace+0x29)[0x8b0fe39]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(handle_fatal_signal+0x53c)[0x8438b4c]
[0xe9b400]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_Z19close_thread_tablesP3THD+0x48f)[0x81f7c3f]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_Z21mysql_execute_commandP3THD+0x45b)[0x825582b]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x36c)[0x8261b2c]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0xcef)[0x8540d3f]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_info+0x31)[0x8542031]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x63d)[0x81c41ad]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld[0x83ad240]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld(handle_rpl_parallel_thread+0x99b)[0x83b1e4b]
/mnt/buildbot/build/mariadb-10.1.16/sql/mysqld[0x8619a8a]
/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0x1a3d4c]
/lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xa38ace]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0xb2d12cd3): COMMIT
Connection ID (thread ID): 18
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
{noformat}

It happens very rarely in buildbot, a handful of occasions over 2 years.
It's possible that the failure is caused by the same problem as was recently fixed in ~~MDEV-10219~~, but then the fix needs to be backported into 10.1 and possibly 10.0.

Kristian Nielsen made changes - 2023-09-07 11:26

Link

This issue is duplicated by ~~MDEV-25064~~ [ ~~MDEV-25064~~ ]

Kristian Nielsen made changes - 2023-09-07 11:27

Status

Open [ 1 ]

In Progress [ 3 ]

Marko Mäkelä added a comment - 2023-09-07 12:36

knielsen, I tested your 10.6 patch on top of my current local 10.6 branch (some changes on top of what I tested in ~~MDEV-25064~~ yesterday). I can reproduce the test failure without the patch, and can’t with the patch. Great work!

Marko Mäkelä added a comment - 2023-09-07 12:36 knielsen , I tested your 10.6 patch on top of my current local 10.6 branch (some changes on top of what I tested in MDEV-25064 yesterday). I can reproduce the test failure without the patch, and can’t with the patch. Great work!

Kristian Nielsen added a comment - 2023-09-07 13:09

Pushed to 10.4.
Thanks Monty for review, and Marko for testing.

Kristian Nielsen added a comment - 2023-09-07 13:09 Pushed to 10.4. Thanks Monty for review, and Marko for testing.

Kristian Nielsen made changes - 2023-09-07 13:09

Fix Version/s		10.4.32 [ 29300 ]
Fix Version/s	10.4 [ 22408 ]
Fix Version/s	10.5 [ 23123 ]
Fix Version/s	10.6 [ 24028 ]
Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Closed [ 6 ]

Julien Fritsch made changes - 2023-09-14 07:19

Fix Version/s		10.5.23 [ 29012 ]
Fix Version/s		10.6.16 [ 29014 ]
Fix Version/s		10.10.7 [ 29018 ]
Fix Version/s		10.11.6 [ 29020 ]
Fix Version/s		11.0.4 [ 29021 ]
Fix Version/s		11.1.3 [ 29023 ]

People

Assignee:: Kristian Nielsen

Reporter:: Elena Stepanova

Votes:: 0 Vote for this issue

Watchers:: 7 Start watching this issue

Dates

Created:: 2016-07-10 23:01

Updated:: 2023-09-14 07:19

Resolved:: 2023-09-07 13:09

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.