binlog_encryption.encrypted_master failed with InnoDB: my_atomic_load32_explicit(&lock->lock_word, MY_MEMORY_ORDER_RELAXED) == X_LOCK_DECR

binlog_encryption.encrypted_master 'cbc,innodb,mix' w3 [ fail ]

        Test ended at 2020-08-31 13:37:23

CURRENT_TEST: binlog_encryption.encrypted_master

mysqltest: In included file "/usr/share/mysql/mysql-test/suite/binlog_encryption/testdata.inc": 

included from /usr/share/mysql/mysql-test/suite/binlog_encryption/encrypted_master.test at line 103:

At line 144: query 'BEGIN NOT ATOMIC

DECLARE counter_name_to_encrypt INT DEFAULT 0;

START TRANSACTION;

WHILE counter_name_to_encrypt<12 DO

SELECT COUNT(*) INTO @cnt FROM innodb_table_name_to_encrypt;

INSERT INTO innodb_table_name_to_encrypt 

SELECT int_column_name_to_encrypt+@cnt, NOW(6), blob_column_name_to_encrypt, NULL, NULL

FROM innodb_table_name_to_encrypt

ORDER BY int_column_name_to_encrypt;

SET counter_name_to_encrypt = counter_name_to_encrypt+1;

END WHILE;

COMMIT;

END

' failed: 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:

< snip >

set @user_var1_to_encrypt= 'dyncol1_val_to_encrypt';

set @user_var2_to_encrypt= 'dyncol2_name_to_encrypt';

INSERT INTO view_name_to_encrypt VALUES

(1, NOW(6), COLUMN_CREATE('dyncol1_name_to_encrypt',@user_var1_to_encrypt), NULL, NULL),

(2, NOW(6), COLUMN_CREATE(@user_var2_to_encrypt,'dyncol2_val_to_encrypt'), NULL, NULL)

;

BEGIN NOT ATOMIC

DECLARE counter_name_to_encrypt INT DEFAULT 0;

START TRANSACTION;

WHILE counter_name_to_encrypt<12 DO

SELECT COUNT(*) INTO @cnt FROM innodb_table_name_to_encrypt;

INSERT INTO innodb_table_name_to_encrypt 

SELECT int_column_name_to_encrypt+@cnt, NOW(6), blob_column_name_to_encrypt, NULL, NULL

FROM innodb_table_name_to_encrypt

ORDER BY int_column_name_to_encrypt;

SET counter_name_to_encrypt = counter_name_to_encrypt+1;

END WHILE;

COMMIT;

END

$$

More results from queries before failure can be found in /dev/shm/var/3/log/encrypted_master.log

Server [mysqld.1 - pid: 3818, winpid: 3818, exit: 256] failed during test run

Server log from this test:

----------SERVER LOG START-----------

2020-08-31 13:37:19 0 [Note] /usr/sbin/mysqld (mysqld 10.3.25-MariaDB-1:10.3.25+maria~jessie-log) starting as process 3819 ...

2020-08-31 13:37:19 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32186)

2020-08-31 13:37:19 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)

2020-08-31 13:37:20 0 [Note] InnoDB: Using Linux native AIO

2020-08-31 13:37:20 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2020-08-31 13:37:20 0 [Note] InnoDB: Uses event mutexes

2020-08-31 13:37:20 0 [Note] InnoDB: Compressed tables use zlib 1.2.8

2020-08-31 13:37:20 0 [Note] InnoDB: Number of pools: 1

2020-08-31 13:37:20 0 [Note] InnoDB: Using generic crc32 instructions

2020-08-31 13:37:20 0 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M

2020-08-31 13:37:20 0 [Note] InnoDB: Completed initialization of buffer pool

2020-08-31 13:37:20 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

2020-08-31 13:37:20 0 [Note] InnoDB: 128 out of 128 rollback segments are active.

2020-08-31 13:37:20 0 [Note] InnoDB: Creating shared tablespace for temporary tables

2020-08-31 13:37:20 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

2020-08-31 13:37:20 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.

2020-08-31 13:37:20 0 [Note] InnoDB: Waiting for purge to start

2020-08-31 13:37:20 0 [Note] InnoDB: 10.3.25 started; log sequence number 1635872; transaction id 34

2020-08-31 13:37:20 0 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var/3/mysqld.1/data/ib_buffer_pool

2020-08-31 13:37:20 0 [Note] Plugin 'SEQUENCE' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.

2020-08-31 13:37:20 0 [Note] InnoDB: Buffer pool(s) load completed at 200831 13:37:20

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_CMP' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'FEEDBACK' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'user_variables' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_MUTEXES' is disabled.

2020-08-31 13:37:20 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.

2020-08-31 13:37:20 0 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-startup-interval=20'

2020-08-31 13:37:20 0 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-first-interval=60'

2020-08-31 13:37:20 0 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-interval=60'

2020-08-31 13:37:20 0 [Warning] /usr/sbin/mysqld: unknown option '--loose-pam-debug'

2020-08-31 13:37:20 0 [Warning] /usr/sbin/mysqld: unknown variable 'loose-debug-sync-timeout=300'

2020-08-31 13:37:20 0 [Note] Server socket created on IP: '127.0.0.1'.

2020-08-31 13:37:20 0 [Note] Reading of all Master_info entries succeeded

2020-08-31 13:37:20 0 [Note] Added new Master_info '' to hash table

2020-08-31 13:37:20 0 [Note] /usr/sbin/mysqld: ready for connections.

Version: '10.3.25-MariaDB-1:10.3.25+maria~jessie-log'  socket: '/dev/shm/var/tmp/3/mysqld.1.sock'  port: 16040  mariadb.org binary distribution

2020-08-31 13:37:21 11 [Note] Deleted Master_info file '/dev/shm/var/3/mysqld.1/data/master.info'.

2020-08-31 13:37:21 11 [Note] Deleted Master_info file '/dev/shm/var/3/mysqld.1/data/relay-log.info'.

2020-08-31 13:37:21 13 [Note] Start binlog_dump to slave_server(2), pos(master-bin.000001, 4), using_gtid(0), gtid('')

2020-08-31 13:37:22 0xa7effb40  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.3.25/storage/innobase/sync/sync0rw.cc line 258

InnoDB: Failing assertion: my_atomic_load32_explicit(&lock->lock_word, MY_MEMORY_ORDER_RELAXED) == X_LOCK_DECR

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/

InnoDB: about forcing recovery.

200831 13:37:22 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed, 

something is definitely wrong and this may fail.

Server version: 10.3.25-MariaDB-1:10.3.25+maria~jessie-log

key_buffer_size=1048576

read_buffer_size=131072

max_used_connections=5

max_threads=153

thread_count=12

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61956 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0xa89005a0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0xa7eff21c thread_stack 0x49000

/usr/sbin/mysqld(my_print_stacktrace+0x28)[0xb7137a98]

/usr/sbin/mysqld(handle_fatal_signal+0x2f9)[0xb6b9cb19]

linux-gate.so.1(__kernel_sigreturn+0x0)[0xb65fcd18]

linux-gate.so.1(__kernel_vsyscall+0x10)[0xb65fcd3c]

/lib/i386-linux-gnu/i686/cmov/libc.so.6(gsignal+0x47)[0xb6076367]

/lib/i386-linux-gnu/i686/cmov/libc.so.6(abort+0x143)[0xb6077a23]

/usr/sbin/mysqld(+0x284f4a)[0xb68a5f4a]

/usr/sbin/mysqld(+0x81fffe)[0xb6e40ffe]

/usr/sbin/mysqld(+0x8d5ee3)[0xb6ef6ee3]

/usr/sbin/mysqld(+0x8f7dfe)[0xb6f18dfe]

/usr/sbin/mysqld(+0x285ff7)[0xb68a6ff7]

/usr/sbin/mysqld(+0x88fa07)[0xb6eb0a07]

/usr/sbin/mysqld(+0x8c446e)[0xb6ee546e]

/usr/sbin/mysqld(+0x8c625b)[0xb6ee725b]

/usr/sbin/mysqld(+0x8c6e1e)[0xb6ee7e1e]

/usr/sbin/mysqld(+0x89e98e)[0xb6ebf98e]

/usr/sbin/mysqld(+0x8ca063)[0xb6eeb063]

/usr/sbin/mysqld(+0x89f591)[0xb6ec0591]

/usr/sbin/mysqld(+0x82df5d)[0xb6e4ef5d]

/usr/sbin/mysqld(+0x833bde)[0xb6e54bde]

/usr/sbin/mysqld(+0x815dd3)[0xb6e36dd3]

/lib/i386-linux-gnu/i686/cmov/libpthread.so.0(+0x6ecb)[0xb65ddecb]

/lib/i386-linux-gnu/i686/cmov/libc.so.6(clone+0x5e)[0xb6133d0e]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x0): (null)

Connection ID (thread ID): 2

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains

information that should help you find out what is causing the crash.

We think the query pointer is invalid, but we will try to print it anyway. 

Query: 

Writing a core file...

Working directory at /dev/shm/var/3/mysqld.1/data

Resource Limits:

Limit                     Soft Limit           Hard Limit           Units     

Max cpu time              unlimited            unlimited            seconds   

Max file size             unlimited            unlimited            bytes     

Max data size             unlimited            unlimited            bytes     

Max stack size            8388608              unlimited            bytes     

Max core file size        unlimited            unlimited            bytes     

Max resident set          unlimited            unlimited            bytes     

Max processes             48521                48521                processes 

Max open files            1024                 1024                 files     

Max locked memory         65536                65536                bytes     

Max address space         unlimited            unlimited            bytes     

Max file locks            unlimited            unlimited            locks     

Max pending signals       48521                48521                signals   

Max msgqueue size         819200               819200               bytes     

Max nice priority         0                    0                    

Max realtime priority     0                    0                    

Max realtime timeout      unlimited            unlimited            us        

Core pattern: core

----------SERVER LOG END-------------

 - found 'core' (0/0)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace from coredump /dev/shm/var/3/log/binlog_encryption.encrypted_master-cbc,innodb,mix/mysqld.1/data/core

Core generated by '/usr/sbin/mysqld'

Output from gdb follows. The first stack trace is from the failing thread.

The following stack traces are from all threads (so the failing one is

duplicated).

--------------------------

[New LWP 3849]

[New LWP 3965]

[New LWP 3850]

[New LWP 4047]

[New LWP 3851]

[New LWP 3871]

[New LWP 3847]

[New LWP 3848]

[New LWP 3843]

[New LWP 3846]

[New LWP 3852]

[New LWP 3840]

[New LWP 3845]

[New LWP 3872]

[New LWP 3838]

[New LWP 3839]

[New LWP 3889]

[New LWP 3836]

[New LWP 3837]

[New LWP 3819]

[New LWP 3834]

[New LWP 3835]

[New LWP 3844]

[New LWP 3828]

[New LWP 3831]

[New LWP 3989]

[New LWP 3875]

[New LWP 3876]

[New LWP 3966]

[New LWP 3877]

[New LWP 3874]

[New LWP 3873]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".

Core was generated by `/usr/sbin/mysqld --defaults-group-suffix=.1 --defaults-file=/dev/shm/var/3/my.c'.

Program terminated with signal SIGABRT, Aborted.

#0  0xb65fcd3c in __kernel_vsyscall ()

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e2d03 in __pthread_kill (threadid=2817522496, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61

#2  0xb71378ec in my_write_core ()

#3  0xb6b9cb96 in handle_fatal_signal ()

#4  <signal handler called>

#5  0xb65fcd3c in __kernel_vsyscall ()

#6  0xb6076367 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56

#7  0xb6077a23 in __GI_abort () at abort.c:89

#8  0xb68a5f4a in ?? ()

#9  0xb6e40ffe in ?? ()

#10 0xb6ef6ee3 in ?? ()

#11 0xb6f18dfe in ?? ()

#12 0xb68a6ff7 in ?? ()

#13 0xb6eb0a07 in ?? ()

#14 0xb6ee546e in ?? ()

#15 0xb6ee725b in ?? ()

#16 0xb6ee7e1e in ?? ()

#17 0xb6ebf98e in ?? ()

#18 0xb6eeb063 in ?? ()

#19 0xb6ec0591 in ?? ()

#20 0xb6e4ef5d in ?? ()

#21 0xb6e54bde in ?? ()

#22 0xb6e36dd3 in ?? ()

#23 0xb65ddecb in start_thread (arg=0xa7effb40) at pthread_create.c:309

#24 0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 32 (Thread 0xa4cfdb40 (LWP 3873)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1c1b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187

#2  0xb6da27c9 in ?? ()

#3  0xb6ec9074 in ?? ()

#4  0xb65ddecb in start_thread (arg=0xa4cfdb40) at pthread_create.c:309

#5  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 31 (Thread 0xa44fcb40 (LWP 3874)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1c1b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187

#2  0xb6d1b89c in ?? ()

#3  0xb70d7ceb in ?? ()

#4  0xb65ddecb in start_thread (arg=0xa44fcb40) at pthread_create.c:309

#5  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 30 (Thread 0xaf098b40 (LWP 3877)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1c1b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187

#2  0xb68e283e in handle_slave_background ()

#3  0xb70d7ceb in ?? ()

#4  0xb65ddecb in start_thread (arg=0xaf098b40) at pthread_create.c:309

#5  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 29 (Thread 0xab691b40 (LWP 3966)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb61294cb in poll () at ../sysdeps/unix/syscall-template.S:81

#2  0xb7187d8d in vio_io_wait ()

#3  0xb7187eec in vio_socket_io_wait ()

#4  0xb7187fc2 in vio_read ()

#5  0xb68ceea1 in ?? ()

#6  0xb68d00e6 in my_net_read_packet_reallen ()

#7  0xb68d021a in my_net_read_packet ()

#8  0xb6993fb3 in do_command(THD*) ()

#9  0xb6a7c4c9 in do_handle_one_connection(CONNECT*) ()

#10 0xb6a7c682 in handle_one_connection ()

#11 0xb70d7ceb in ?? ()

#12 0xb65ddecb in start_thread (arg=0xab691b40) at pthread_create.c:309

#13 0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 28 (Thread 0xaf0e2b40 (LWP 3876)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e59ec in do_sigwait (set=0xb77d30b4, set@entry=0xaf0e2224, sig=sig@entry=0xaf0e21f4) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:60

#2  0xb65e5a69 in __sigwait (set=0xaf0e2224, sig=0xaf0e21f4) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:97

#3  0xb68c220a in signal_hand ()

#4  0xb70d7ceb in ?? ()

#5  0xb65ddecb in start_thread (arg=0xaf0e2b40) at pthread_create.c:309

#6  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 27 (Thread 0xb005cb40 (LWP 3875)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1c1b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187

#2  0xb6c7b58c in binlog_background_thread ()

#3  0xb70d7ceb in ?? ()

#4  0xb65ddecb in start_thread (arg=0xb005cb40) at pthread_create.c:309

#5  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 26 (Thread 0xa8af5b40 (LWP 3989)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1fe5 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb6c76d4e in MYSQL_BIN_LOG::wait_for_update_binlog_end_pos(THD*, timespec*) ()

#3  0xb69aad01 in mysql_binlog_send(THD*, char*, unsigned long long, unsigned short) ()

#4  0xb69937c7 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) ()

#5  0xb699404a in do_command(THD*) ()

#6  0xb6a7c4c9 in do_handle_one_connection(CONNECT*) ()

#7  0xb6a7c682 in handle_one_connection ()

#8  0xb70d7ceb in ?? ()

#9  0xb65ddecb in start_thread (arg=0xa8af5b40) at pthread_create.c:309

#10 0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 25 (Thread 0xb085db40 (LWP 3831)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1fe5 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb6ff0451 in ?? ()

#3  0xb6fe6c7e in ?? ()

#4  0xb70d7ceb in ?? ()

#5  0xb65ddecb in start_thread (arg=0xb085db40) at pthread_create.c:309

#6  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 24 (Thread 0xb1cd2b40 (LWP 3828)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1fe5 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb713c1cd in ?? ()

#3  0xb70d7ceb in ?? ()

#4  0xb65ddecb in start_thread (arg=0xb1cd2b40) at pthread_create.c:309

#5  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 23 (Thread 0xaa2f8b40 (LWP 3844)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1fe5 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb6da21cb in ?? ()

#3  0xb6e347e7 in ?? ()

#4  0xb65ddecb in start_thread (arg=0xaa2f8b40) at pthread_create.c:309

#5  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 22 (Thread 0xae6e1b40 (LWP 3835)):

#0  0xb65d45cd in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb65d4646 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0xb6d976e9 in ?? ()

#3  0xb6d9d299 in ?? ()

#4  0xb6da0c1b in ?? ()

#5  0xb6f33f84 in ?? ()

#6  0xb6e38549 in ?? ()

#7  0xb65ddecb in start_thread (arg=0xae6e1b40) at pthread_create.c:309

#8  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 21 (Thread 0xaeee2b40 (LWP 3834)):

#0  0xb65d45cd in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb65d4646 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0xb6d976e9 in ?? ()

#3  0xb6d9d299 in ?? ()

#4  0xb6da0c1b in ?? ()

#5  0xb6f33f84 in ?? ()

#6  0xb6e38549 in ?? ()

#7  0xb65ddecb in start_thread (arg=0xaeee2b40) at pthread_create.c:309

#8  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 20 (Thread 0xb5f18740 (LWP 3819)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb61294cb in poll () at ../sysdeps/unix/syscall-template.S:81

#2  0xb68cc05f in handle_connections_sockets() ()

#3  0xb68cda53 in mysqld_main(int, char**) ()

#4  0xb68abb57 in main ()

Thread 19 (Thread 0xad6dfb40 (LWP 3837)):

#0  0xb65d45cd in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb65d4646 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0xb6d976e9 in ?? ()

#3  0xb6d9d299 in ?? ()

#4  0xb6da0c1b in ?? ()

#5  0xb6f33f84 in ?? ()

#6  0xb6e38549 in ?? ()

#7  0xb65ddecb in start_thread (arg=0xad6dfb40) at pthread_create.c:309

#8  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 18 (Thread 0xadee0b40 (LWP 3836)):

#0  0xb65d45cd in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb65d4646 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0xb6d976e9 in ?? ()

#3  0xb6d9d299 in ?? ()

#4  0xb6da0c1b in ?? ()

#5  0xb6f33f84 in ?? ()

#6  0xb6e38549 in ?? ()

#7  0xb65ddecb in start_thread (arg=0xadee0b40) at pthread_create.c:309

#8  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 17 (Thread 0xaf04eb40 (LWP 3889)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb61294cb in poll () at ../sysdeps/unix/syscall-template.S:81

#2  0xb7187d8d in vio_io_wait ()

#3  0xb7187eec in vio_socket_io_wait ()

#4  0xb7187fc2 in vio_read ()

#5  0xb68ceea1 in ?? ()

#6  0xb68d00e6 in my_net_read_packet_reallen ()

#7  0xb68d021a in my_net_read_packet ()

#8  0xb6993fb3 in do_command(THD*) ()

#9  0xb6a7c4c9 in do_handle_one_connection(CONNECT*) ()

#10 0xb6a7c682 in handle_one_connection ()

#11 0xb70d7ceb in ?? ()

#12 0xb65ddecb in start_thread (arg=0xaf04eb40) at pthread_create.c:309

#13 0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 16 (Thread 0xac6ddb40 (LWP 3839)):

#0  0xb65d45cd in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb65d4646 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0xb6d976e9 in ?? ()

#3  0xb6d9d299 in ?? ()

#4  0xb6da0c1b in ?? ()

#5  0xb6f33f84 in ?? ()

#6  0xb6e38549 in ?? ()

#7  0xb65ddecb in start_thread (arg=0xac6ddb40) at pthread_create.c:309

#8  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 15 (Thread 0xacedeb40 (LWP 3838)):

#0  0xb65d45cd in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb65d4646 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0xb6d976e9 in ?? ()

#3  0xb6d9d299 in ?? ()

#4  0xb6da0c1b in ?? ()

#5  0xb6f33f84 in ?? ()

#6  0xb6e38549 in ?? ()

#7  0xb65ddecb in start_thread (arg=0xacedeb40) at pthread_create.c:309

#8  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 14 (Thread 0xa54feb40 (LWP 3872)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e5166 in nanosleep () at ../sysdeps/unix/syscall-template.S:81

#2  0xb6da2b40 in ?? ()

#3  0xb6eb9ded in ?? ()

#4  0xb65ddecb in start_thread (arg=0xa54feb40) at pthread_create.c:309

#5  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 13 (Thread 0xa9af7b40 (LWP 3845)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1fe5 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb6da21cb in ?? ()

#3  0xb6e37d1c in ?? ()

#4  0xb65ddecb in start_thread (arg=0xa9af7b40) at pthread_create.c:309

#5  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 12 (Thread 0xabedcb40 (LWP 3840)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1fe5 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb6da21cb in ?? ()

#3  0xb6edcc7e in ?? ()

#4  0xb65ddecb in start_thread (arg=0xabedcb40) at pthread_create.c:309

#5  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 11 (Thread 0xa66fcb40 (LWP 3852)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1c1b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187

#2  0xb6da27c9 in ?? ()

#3  0xb6e3057a in ?? ()

#4  0xb6e361d1 in ?? ()

#5  0xb65ddecb in start_thread (arg=0xa66fcb40) at pthread_create.c:309

#6  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 10 (Thread 0xa92f6b40 (LWP 3846)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1fe5 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb6da21cb in ?? ()

#3  0xb6f25d4c in ?? ()

#4  0xb65ddecb in start_thread (arg=0xa92f6b40) at pthread_create.c:309

#5  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 9 (Thread 0xaaaf9b40 (LWP 3843)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1fe5 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb6da21cb in ?? ()

#3  0xb6d753ad in ?? ()

#4  0xb65ddecb in start_thread (arg=0xaaaf9b40) at pthread_create.c:309

#5  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 8 (Thread 0xab2fab40 (LWP 3848)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e5166 in nanosleep () at ../sysdeps/unix/syscall-template.S:81

#2  0xb6da2b40 in ?? ()

#3  0xb6e356d7 in ?? ()

#4  0xb65ddecb in start_thread (arg=0xab2fab40) at pthread_create.c:309

#5  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 7 (Thread 0xa88f5b40 (LWP 3847)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1fe5 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb6da21cb in ?? ()

#3  0xb6e83388 in ?? ()

#4  0xb6f716a8 in ?? ()

#5  0xb65ddecb in start_thread (arg=0xa88f5b40) at pthread_create.c:309

#6  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 6 (Thread 0xa5cffb40 (LWP 3871)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1c1b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187

#2  0xb6da27c9 in ?? ()

#3  0xb6ed0462 in ?? ()

#4  0xb65ddecb in start_thread (arg=0xa5cffb40) at pthread_create.c:309

#5  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 5 (Thread 0xa6efdb40 (LWP 3851)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1c1b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187

#2  0xb6da27c9 in ?? ()

#3  0xb6e3057a in ?? ()

#4  0xb6e361d1 in ?? ()

#5  0xb65ddecb in start_thread (arg=0xa6efdb40) at pthread_create.c:309

#6  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 4 (Thread 0xa8aabb40 (LWP 4047)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1fe5 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb68c3812 in one_thread_per_connection_end(THD*, bool) ()

#3  0xb6a7c3e2 in do_handle_one_connection(CONNECT*) ()

#4  0xb6a7c682 in handle_one_connection ()

#5  0xb70d7ceb in ?? ()

#6  0xb65ddecb in start_thread (arg=0xa8aabb40) at pthread_create.c:309

#7  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 3 (Thread 0xa76feb40 (LWP 3850)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1c1b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187

#2  0xb6da27c9 in ?? ()

#3  0xb6e3057a in ?? ()

#4  0xb6e361d1 in ?? ()

#5  0xb65ddecb in start_thread (arg=0xa76feb40) at pthread_create.c:309

#6  0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 2 (Thread 0xab6dbb40 (LWP 3965)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e1c1b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187

#2  0xb6da27c9 in ?? ()

#3  0xb6e3f5c7 in ?? ()

#4  0xb6e41fa5 in ?? ()

#5  0xb6ead386 in ?? ()

#6  0xb6eb2869 in ?? ()

#7  0xb6ea610b in ?? ()

#8  0xb6e12449 in ?? ()

#9  0xb6e155bd in ?? ()

#10 0xb6d2a359 in ?? ()

#11 0xb6d2a5f5 in ?? ()

#12 0xb6ba2d4f in handler::ha_index_next(unsigned char*) ()

#13 0xb710964f in ?? ()

#14 0xb7109b3b in ?? ()

#15 0xb6ba2be2 in handler::ha_index_next(unsigned char*) ()

#16 0xb69d2883 in ?? ()

#17 0xb69c5bed in sub_select(JOIN*, st_join_table*, bool) ()

#18 0xb69ec0a5 in JOIN::exec_inner() ()

#19 0xb69ec35a in JOIN::exec() ()

#20 0xb69ec4cc in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) ()

#21 0xb69ecef2 in handle_select(THD*, LEX*, select_result*, unsigned long) ()

#22 0xb6893c00 in ?? ()

#23 0xb698acec in mysql_execute_command(THD*) ()

#24 0xb68f244a in sp_instr_stmt::exec_core(THD*, unsigned int*) ()

#25 0xb68fa344 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) ()

#26 0xb68fad7b in sp_instr_stmt::execute(THD*, unsigned int*) ()

#27 0xb68f615a in sp_head::execute(THD*, bool) ()

#28 0xb68f77f3 in sp_head::execute_procedure(THD*, List<Item>*) ()

#29 0xb6981883 in ?? ()

#30 0xb6989bb9 in mysql_execute_command(THD*) ()

#31 0xb6990683 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) ()

#32 0xb69924cf in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) ()

#33 0xb699404a in do_command(THD*) ()

#34 0xb6a7c4c9 in do_handle_one_connection(CONNECT*) ()

#35 0xb6a7c682 in handle_one_connection ()

#36 0xb70d7ceb in ?? ()

#37 0xb65ddecb in start_thread (arg=0xab6dbb40) at pthread_create.c:309

#38 0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 1 (Thread 0xa7effb40 (LWP 3849)):

#0  0xb65fcd3c in __kernel_vsyscall ()

#1  0xb65e2d03 in __pthread_kill (threadid=2817522496, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61

#2  0xb71378ec in my_write_core ()

#3  0xb6b9cb96 in handle_fatal_signal ()

#4  <signal handler called>

#5  0xb65fcd3c in __kernel_vsyscall ()

#6  0xb6076367 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56

#7  0xb6077a23 in __GI_abort () at abort.c:89

#8  0xb68a5f4a in ?? ()

#9  0xb6e40ffe in ?? ()

#10 0xb6ef6ee3 in ?? ()

#11 0xb6f18dfe in ?? ()

#12 0xb68a6ff7 in ?? ()

#13 0xb6eb0a07 in ?? ()

#14 0xb6ee546e in ?? ()

#15 0xb6ee725b in ?? ()

#16 0xb6ee7e1e in ?? ()

#17 0xb6ebf98e in ?? ()

#18 0xb6eeb063 in ?? ()

#19 0xb6ec0591 in ?? ()

#20 0xb6e4ef5d in ?? ()

#21 0xb6e54bde in ?? ()

#22 0xb6e36dd3 in ?? ()

#23 0xb65ddecb in start_thread (arg=0xa7effb40) at pthread_create.c:309

#24 0xb6133d0e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129