[MDEV-8536] Crash in os_event_free on shutdown Created: 2015-07-23  Updated: 2017-06-27  Resolved: 2017-06-27

Status: Closed
Project: MariaDB Server
Component/s: Platform Debian, Platform Power
Affects Version/s: 10.0.20
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Sergey Vojtovich Assignee: Otto Kekäläinen
Resolution: Fixed Votes: 0
Labels: None

Sprint: 10.0.22, 10.1.9-1, 10.1.9-2

 Description   

This crash comes from debian build system: https://buildd.debian.org/status/package.php?p=mariadb-10.0

Server crashes during shutdown of mtr bootstrap instance. Looks like either double free or memory corruption.

(gdb) --no-defaults --bootstrap --basedir=/home/otto/mariadb-10.0-10.0.20 --datadir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/install.db --default-storage-engine=myisam --skip-plugin-archive --skip-plugin-aria --skip-plugin-blackhole --skip-plugin-federated --skip-plugin-feedback --skip-plugin-innodb --skip-plugin-innodb-buffer-page --skip-plugin-innodb-buffer-page-lru --skip-plugin-innodb-buffer-pool-stats --skip-plugin-innodb-changed-pages --skip-plugin-innodb-cmp --skip-plugin-innodb-cmp-per-index --skip-plugin-innodb-cmp-per-index-reset --skip-plugin-innodb-cmp-reset --skip-plugin-innodb-cmpmem --skip-plugin-innodb-cmpmem-reset --skip-plugin-innodb-ft-being-deleted --skip-plugin-innodb-ft-config --skip-plugin-innodb-ft-default-stopword --skip-plugin-innodb-ft-deleted --skip-plugin-innodb-ft-index-cache --skip-plugin-innodb-ft-index-table --skip-plugin-innodb-lock-waits --skip-plugin-innodb-locks --skip-plugin-innodb-metrics --skip-plugin-innodb-sys-columns --skip-plugin-innodb-sys-datafiles --skip-plugin-innodb-sys-fields --skip-plugin-innodb-sys-foreign --skip-plugin-innodb-sys-foreign-cols --skip-plugin-innodb-sys-indexes --skip-plugin-innodb-sys-tables --skip-plugin-innodb-sys-tablespaces --skip-plugin-innodb-sys-tablestats --skip-plugin-innodb-trx --skip-plugin-partition --skip-plugin-xtradb-internal-hash-tables --skip-plugin-xtradb-read-view --skip-plugin-xtradb-rseg --loose-innodb --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/tmp/ --core-file --lc-messages-dir=/home/otto/mariadb-10.0-10.0.20/builddir/sql/share/ --character-sets-dir=/home/otto/mariadb-10.0-10.0.20/sql/share/charsets --debug --stack-trace
 
Undefined command: "--no-defaults".  Try "help".
 
(gdb) run --no-defaults --bootstrap --basedir=/home/otto/mariadb-10.0-10.0.20 --datadir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/install.db --default-storage-engine=myisam --skip-plugin-archive --skip-plugin-aria --skip-plugin-blackhole --skip-plugin-federated --skip-plugin-feedback --skip-plugin-innodb --skip-plugin-innodb-buffer-page --skip-plugin-innodb-buffer-page-lru --skip-plugin-innodb-buffer-pool-stats --skip-plugin-innodb-changed-pages --skip-plugin-innodb-cmp --skip-plugin-innodb-cmp-per-index --skip-plugin-innodb-cmp-per-index-reset --skip-plugin-innodb-cmp-reset --skip-plugin-innodb-cmpmem --skip-plugin-innodb-cmpmem-reset --skip-plugin-innodb-ft-being-deleted --skip-plugin-innodb-ft-config --skip-plugin-innodb-ft-default-stopword --skip-plugin-innodb-ft-deleted --skip-plugin-innodb-ft-index-cache --skip-plugin-innodb-ft-index-table --skip-plugin-innodb-lock-waits --skip-plugin-innodb-locks --skip-plugin-innodb-metrics --skip-plugin-innodb-sys-columns --skip-plugin-innodb-sys-datafiles --skip-plugin-innodb-sys-fields --skip-plugin-innodb-sys-foreign --skip-plugin-innodb-sys-foreign-cols --skip-plugin-innodb-sys-indexes --skip-plugin-innodb-sys-tables --skip-plugin-innodb-sys-tablespaces --skip-plugin-innodb-sys-tablestats --skip-plugin-innodb-trx --skip-plugin-partition --skip-plugin-xtradb-internal-hash-tables --skip-plugin-xtradb-read-view --skip-plugin-xtradb-rseg --loose-innodb --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/tmp/ --core-file --lc-messages-dir=/home/otto/mariadb-10.0-10.0.20/builddir/sql/share/ --character-sets-dir=/home/otto/mariadb-10.0-10.0.20/sql/share/charsets --debug --stack-trace
 
Starting program: /home/otto/builddir-debug/sql/mysqld --no-defaults --bootstrap --basedir=/home/otto/mariadb-10.0-10.0.20 --datadir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/install.db --default-storage-engine=myisam --skip-plugin-archive --skip-plugin-aria --skip-plugin-blackhole --skip-plugin-federated --skip-plugin-feedback --skip-plugin-innodb --skip-plugin-innodb-buffer-page --skip-plugin-innodb-buffer-page-lru --skip-plugin-innodb-buffer-pool-stats --skip-plugin-innodb-changed-pages --skip-plugin-innodb-cmp --skip-plugin-innodb-cmp-per-index --skip-plugin-innodb-cmp-per-index-reset --skip-plugin-innodb-cmp-reset --skip-plugin-innodb-cmpmem --skip-plugin-innodb-cmpmem-reset --skip-plugin-innodb-ft-being-deleted --skip-plugin-innodb-ft-config --skip-plugin-innodb-ft-default-stopword --skip-plugin-innodb-ft-deleted --skip-plugin-innodb-ft-index-cache --skip-plugin-innodb-ft-index-table --skip-plugin-innodb-lock-waits --skip-plugin-innodb-locks --skip-plugin-innodb-metrics --skip-plugin-innodb-sys-columns --skip-plugin-innodb-sys-datafiles --skip-plugin-innodb-sys-fields --skip-plugin-innodb-sys-foreign --skip-plugin-innodb-sys-foreign-cols --skip-plugin-innodb-sys-indexes --skip-plugin-innodb-sys-tables --skip-plugin-innodb-sys-tablespaces --skip-plugin-innodb-sys-tablestats --skip-plugin-innodb-trx --skip-plugin-partition --skip-plugin-xtradb-internal-hash-tables --skip-plugin-xtradb-read-view --skip-plugin-xtradb-rseg --loose-innodb --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/tmp/ --core-file --lc-messages-dir=/home/otto/mariadb-10.0-10.0.20/builddir/sql/share/ --character-sets-dir=/home/otto/mariadb-10.0-10.0.20/sql/share/charsets --debug --stack-trace
 
warning: Could not load shared library symbols for linux-vdso32.so.1.
 
Do you need "set solib-search-path" or "set sysroot"?
 
[Thread debugging using libthread_db enabled]
 
Using host libthread_db library "/lib/powerpc-linux-gnu/libthread_db.so.1".
 
150723 18:59:05 [Note] /home/otto/builddir-debug/sql/mysqld (mysqld 10.0.20-MariaDB-0+deb8u1-debug) starting as process 13168 ...
 
150723 18:59:05 [Note] Plugin 'ARCHIVE' is disabled.
 
150723 18:59:05 [Note] Plugin 'Aria' is disabled.
 
150723 18:59:05 [Note] Plugin 'BLACKHOLE' is disabled.
 
150723 18:59:05 [Note] InnoDB: Using mutexes to ref count buffer pool pages
 
2015-07-23 18:59:05 f7fe78d0 InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
 
2015-07-23 18:59:05 f7fe78d0 InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!
 
150723 18:59:05 [Note] InnoDB: The InnoDB memory heap is disabled
 
150723 18:59:05 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
150723 18:59:05 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
 
150723 18:59:05 [Note] InnoDB: Compressed tables use zlib 1.2.8
 
150723 18:59:05 [Note] InnoDB: Using Linux native AIO
 
150723 18:59:05 [Note] InnoDB: Not using CPU crc32 instructions
 
150723 18:59:05 [Note] InnoDB: Initializing buffer pool, size = 128.0M
 
150723 18:59:05 [Note] InnoDB: Completed initialization of buffer pool
 
[New Thread 0xe53ff430 (LWP 13173)]
 
[New Thread 0xe4bff430 (LWP 13174)]
 
[New Thread 0xe43ff430 (LWP 13175)]
 
[New Thread 0xe3bff430 (LWP 13176)]
 
[New Thread 0xe33ff430 (LWP 13177)]
 
[New Thread 0xe2bff430 (LWP 13178)]
 
[New Thread 0xe23ff430 (LWP 13179)]
 
[New Thread 0xe1bff430 (LWP 13180)]
 
[New Thread 0xe13ff430 (LWP 13181)]
 
[New Thread 0xe0bff430 (LWP 13182)]
 
150723 18:59:05 [Note] InnoDB: Highest supported file format is Barracuda.
 
[New Thread 0xdfbff430 (LWP 13183)]
 
150723 18:59:06 [Note] InnoDB: 128 rollback segment(s) are active.
 
[Thread 0xdfbff430 (LWP 13183) exited]
 
[New Thread 0xdfbff430 (LWP 13184)]
 
[New Thread 0xdefff430 (LWP 13185)]
 
[New Thread 0xde7ff430 (LWP 13186)]
 
[New Thread 0xddfff430 (LWP 13187)]
 
[New Thread 0xdd7ff430 (LWP 13188)]
 
150723 18:59:06 [Note] InnoDB: Waiting for purge to start
 
[New Thread 0xdc7ff430 (LWP 13189)]
 
[New Thread 0xdbfff430 (LWP 13190)]
 
150723 18:59:06 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.24-72.2 started; log sequence number 1629494
 
[New Thread 0xdb7ff430 (LWP 13191)]
 
[New Thread 0xdafff430 (LWP 13192)]
 
[New Thread 0xda7ff430 (LWP 13193)]
 
150723 18:59:06 [Note] Plugin 'XTRADB_READ_VIEW' is disabled.
 
150723 18:59:06 [Note] Plugin 'XTRADB_INTERNAL_HASH_TABLES' is disabled.
 
150723 18:59:06 [Note] Plugin 'XTRADB_RSEG' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_TRX' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_CMPMEM' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_METRICS' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
 
150723 18:59:06 [Note] Plugin 'INNODB_CHANGED_PAGES' is disabled.
 
150723 18:59:06 [Note] Plugin 'FEDERATED' is disabled.
 
150723 18:59:06 [Note] Plugin 'FEEDBACK' is disabled.
 
150723 18:59:06 [Note] Plugin 'partition' is disabled.
 
[New Thread 0xf7e87430 (LWP 13194)]
 
[New Thread 0xf7e27430 (LWP 13195)]
 
 
 
bt
 
exit;
 
ERROR: 1064  You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'bt
 
exit;' at line 1
 
150723 19:00:29 [ERROR] Aborting
 
 
 
[Thread 0xf7e27430 (LWP 13195) exited]
 
150723 19:00:29 [Note] InnoDB: FTS optimize thread exiting.
 
150723 19:00:29 [Note] InnoDB: Starting shutdown...
 
[Thread 0xda7ff430 (LWP 13193) exited]
 
[Thread 0xde7ff430 (LWP 13186) exited]
 
[Thread 0xdb7ff430 (LWP 13191) exited]
 
[Thread 0xdefff430 (LWP 13185) exited]
 
[Thread 0xdafff430 (LWP 13192) exited]
 
[Thread 0xdfbff430 (LWP 13184) exited]
 
[Thread 0xdd7ff430 (LWP 13188) exited]
 
[Thread 0xdc7ff430 (LWP 13189) exited]
 
[Thread 0xddfff430 (LWP 13187) exited]
 
[Thread 0xe3bff430 (LWP 13176) exited]
 
[Thread 0xe0bff430 (LWP 13182) exited]
 
[Thread 0xe53ff430 (LWP 13173) exited]
 
[Thread 0xe13ff430 (LWP 13181) exited]
 
[Thread 0xe23ff430 (LWP 13179) exited]
 
[Thread 0xe4bff430 (LWP 13174) exited]
 
[Thread 0xe43ff430 (LWP 13175) exited]
 
[Thread 0xe1bff430 (LWP 13180) exited]
 
[Thread 0xe33ff430 (LWP 13177) exited]
 
[Thread 0xe2bff430 (LWP 13178) exited]
 
[Thread 0xdbfff430 (LWP 13190) exited]
 
 
 
Program received signal SIGSEGV, Segmentation fault.
 
0x109e8b8c in arena_purge ()
 
(gdb) bt
 
#0  0x109e8b8c in arena_purge ()
 
#1  0x109e8f60 in arena_dalloc_bin_run ()
 
#2  0x109ea7d8 in je_arena_dalloc_bin_locked ()
 
#3  0x10a0566c in je_tcache_bin_flush_small ()
 
#4  0x109df5f8 in free ()
 
#5  0x10754780 in ut_free (ptr=ptr@entry=0xf77e58e0)
 
    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/ut/ut0mem.cc:223
 
#6  0x1065132c in os_event_free (event=0xf77e58e0)
 
    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/os/os0sync.cc:540
 
#7  0x106517f4 in os_sync_free ()
 
    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/os/os0sync.cc:321
 
#8  0x10713c04 in innobase_shutdown_for_mysql ()
 
    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/srv/srv0start.cc:3227
 
#9  0x105be570 in innobase_end (hton=<optimized out>, type=<optimized out>)
 
    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/handler/ha_innodb.cc:3883
 
#10 0x1034a458 in ha_finalize_handlerton (plugin=0xf745c368)
 
    at /home/otto/mariadb-10.0-10.0.20/sql/handler.cc:440
 
#11 0x101a8e80 in plugin_deinitialize (plugin=0xf745c368, 
    ref_check=ref_check@entry=true)
 
    at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1176
 
#12 0x101acf58 in reap_plugins ()
 
    at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1249
 
#13 0x101ad96c in plugin_shutdown ()
 
    at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1895
 
#14 0x100ef320 in clean_up (print_message=<optimized out>)
 
    at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:2029
 
#15 0x100f6c54 in unireg_abort (exit_code=1)
 
    at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:1954
 
#16 0x100fd178 in mysqld_main (argc=54, argv=0xf7409e30)
 
    at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:5510
 
#17 0x100ee8d0 in main (argc=<optimized out>, argv=<optimized out>)
 
    at /home/otto/mariadb-10.0-10.0.20/sql/main.cc:25
 
(gdb) x/10c 0xf77e58e0
 
0xf77e58e0:	0 '\000'	0 '\000'	0 '\000'	0 '\000'	0 '\000'	0 '\000'	0 '\000'	0 '\000'
 
0xf77e58e8:	0 '\000'	0 '\000'
 
(gdb) thr apply all bt full
 
 
 
Thread 23 (Thread 0xf7e87430 (LWP 13194)):
 
#0  0x0ffd25f0 in do_sigwait () from /lib/powerpc-linux-gnu/libpthread.so.0
 
No symbol table info available.
 
#1  0x0ffd26a8 in sigwait () from /lib/powerpc-linux-gnu/libpthread.so.0
 
No symbol table info available.
 
#2  0x100f35cc in signal_hand (arg=<optimized out>)
 
    at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:3346
 
        error = <optimized out>
 
        set = {__val = {548869, 0 <repeats 31 times>}}
 
        sig = 0
 
        _db_stack_frame_ = {func = 0x10b94f78 "?func", file = 0x10b94f80 "?file", 
          level = 2147483649, prev = 0x0}
 
#3  0x0ffc7c3c in start_thread () from /lib/powerpc-linux-gnu/libpthread.so.0
 
No symbol table info available.
 
#4  0x0fb56470 in clone () from /lib/powerpc-linux-gnu/libc.so.6
 
No symbol table info available.
 
 
 
Thread 1 (Thread 0xf7fe78d0 (LWP 13168)):
 
#0  0x109e8b8c in arena_purge ()
 
No symbol table info available.
 
#1  0x109e8f60 in arena_dalloc_bin_run ()
 
No symbol table info available.
 
#2  0x109ea7d8 in je_arena_dalloc_bin_locked ()
 
No symbol table info available.
 
#3  0x10a0566c in je_tcache_bin_flush_small ()
 
No symbol table info available.
 
#4  0x109df5f8 in free ()
 
No symbol table info available.
 
#5  0x10754780 in ut_free (ptr=ptr@entry=0xf77e58e0)
 
    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/ut/ut0mem.cc:223
 
        block = <optimized out>
 
#6  0x1065132c in os_event_free (event=0xf77e58e0)
 
    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/os/os0sync.cc:540
 
No locals.
 
#7  0x106517f4 in os_sync_free ()
 
    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/os/os0sync.cc:321
 
        event = <optimized out>
 
        mutex = <optimized out>
 
#8  0x10713c04 in innobase_shutdown_for_mysql ()
 
    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/srv/srv0start.cc:3227
 
        i = <optimized out>
 
#9  0x105be570 in innobase_end (hton=<optimized out>, type=<optimized out>)
 
    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/handler/ha_innodb.cc:3883
 
        err = 0
 
        _db_stack_frame_ = {func = 0x10ac98d8 "ha_finalize_handlerton", 
          file = 0x10ac9418 "/home/otto/mariadb-10.0-10.0.20/sql/handler.cc", 
          level = 2147483652, prev = 0xfffeea58}
 
        __PRETTY_FUNCTION__ = "int innobase_end(handlerton*, ha_panic_function)"
 
#10 0x1034a458 in ha_finalize_handlerton (plugin=0xf745c368)
 
    at /home/otto/mariadb-10.0-10.0.20/sql/handler.cc:440
 
        hton = 0xf7416738
 
        _db_stack_frame_ = {func = 0x10a33bec "plugin_shutdown", 
          file = 0x10a3347c "/home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc", 
          level = 2147483651, prev = 0xfffeebf8}
 
        __PRETTY_FUNCTION__ = "int ha_finalize_handlerton(st_plugin_int*)"
 
#11 0x101a8e80 in plugin_deinitialize (plugin=0xf745c368, 
    ref_check=ref_check@entry=true)
 
    at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1176
 
        __PRETTY_FUNCTION__ = "void plugin_deinitialize(st_plugin_int*, bool)"
 
#12 0x101acf58 in reap_plugins ()
 
    at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1249
 
        count = <optimized out>
 
        idx = <optimized out>
 
        reap = 0xfffeeb00
 
        list = 0xfffeeaec
 
        __PRETTY_FUNCTION__ = "void reap_plugins()"
 
        plugin = <optimized out>
 
#13 0x101ad96c in plugin_shutdown ()
 
    at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1895
 
        i = <optimized out>
 
        count = 47
 
        dl = <optimized out>
 
        plugins = <optimized out>
 
        plugin = <optimized out>
 
        _db_stack_frame_ = {func = 0x10a1126c "unireg_abort", 
          file = 0x10a107c0 "/home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc", 
          level = 2147483650, prev = 0xfffeec88}
 
#14 0x100ef320 in clean_up (print_message=<optimized out>)
 
    at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:2029
 
No locals.
 
#15 0x100f6c54 in unireg_abort (exit_code=1)
 
    at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:1954
 
        _db_stack_frame_ = {func = 0x10b94f78 "?func", file = 0x10b94f80 "?file", 
          level = 2147483649, prev = 0x0}
 
#16 0x100fd178 in mysqld_main (argc=54, argv=0xf7409e30)
 
    at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:5510
 
        ho_error = <optimized out>
 
#17 0x100ee8d0 in main (argc=<optimized out>, argv=<optimized out>)
 
    at /home/otto/mariadb-10.0-10.0.20/sql/main.cc:25
 
No locals.



 Comments   
Comment by Otto Kekäläinen [ 2015-07-23 ]

Important remarks:

When you fix it, please link to the commit that was the fix, so that that commit can be backported as a patch in Debian packaging. We cannot wait for 10.0.21 to release, this should be fixed ASAP.

Comment by Otto Kekäläinen [ 2015-07-23 ]

More build logs and trace if useful: http://labs.seravo.fi/~otto/mariadb-repo/logs/powerpc/

Comment by Otto Kekäläinen [ 2015-07-23 ]

For the record, here is the overview of minor Debian platform's builds: http://buildd.debian-ports.org/status/package.php?p=mariadb-10.0

There is platform ppc64 (big endian 64-bit PowerPC) which fails in similar ways:
http://buildd.debian-ports.org/status/fetch.php?pkg=mariadb-10.0&arch=ppc64&ver=10.0.20-2&stamp=1436339590

For some strange reason the failing test suite does not stop the ppc64 build from proceeding as it normally does in MariaDB builds..

Comment by Jan Lindström (Inactive) [ 2015-07-24 ]

Does this crash happen if you use innodb_plugin for mtr bootstrap ?

Comment by Sergey Vojtovich [ 2015-07-24 ]

jplindst, from the log I can guess it was XtraDB:

...
 
150723 18:59:06 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.24-72.2 started; log sequence number 1629494
 
...

Comment by Otto Kekäläinen [ 2015-07-24 ]

Thanks jplindst for a quick response.

I've spent way too much time on this already, so if the provided logs and traces are not enough to pinpoint what broke in .20 and you need me to do additional debugging, please provide me with copy-paste commands or easily applicable patches so I don't need to spend time researching how to do something I've never done before, like for example switch xtradb to innodb (if that is what use of innodb_plugin implies).

Comment by Jan Lindström (Inactive) [ 2015-07-24 ]

I did read diff for 10.0.19 -> 10.0.20 but failed to see anything clear. So why this fails and only on ppc is mystery still, hard to give any patches as we are just guessing. Try this to use innodb_plugin:

--no-defaults --bootstrap --basedir=/home/otto/mariadb-10.0-10.0.20 --datadir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/install.db --default-storage-engine=myisam --skip-plugin-archive --skip-plugin-aria --skip-plugin-blackhole --skip-plugin-federated --skip-plugin-feedback --skip-plugin-innodb --skip-plugin-innodb-buffer-page --skip-plugin-innodb-buffer-page-lru --skip-plugin-innodb-buffer-pool-stats --skip-plugin-innodb-changed-pages --skip-plugin-innodb-cmp --skip-plugin-innodb-cmp-per-index --skip-plugin-innodb-cmp-per-index-reset --skip-plugin-innodb-cmp-reset --skip-plugin-innodb-cmpmem --skip-plugin-innodb-cmpmem-reset --skip-plugin-innodb-ft-being-deleted --skip-plugin-innodb-ft-config --skip-plugin-innodb-ft-default-stopword --skip-plugin-innodb-ft-deleted --skip-plugin-innodb-ft-index-cache --skip-plugin-innodb-ft-index-table --skip-plugin-innodb-lock-waits --skip-plugin-innodb-locks --skip-plugin-innodb-metrics --skip-plugin-innodb-sys-columns --skip-plugin-innodb-sys-datafiles --skip-plugin-innodb-sys-fields --skip-plugin-innodb-sys-foreign --skip-plugin-innodb-sys-foreign-cols --skip-plugin-innodb-sys-indexes --skip-plugin-innodb-sys-tables --skip-plugin-innodb-sys-tablespaces --skip-plugin-innodb-sys-tablestats --skip-plugin-innodb-trx --skip-plugin-partition --skip-plugin-xtradb-internal-hash-tables --skip-plugin-xtradb-read-view --skip-plugin-xtradb-rseg --loose-innodb --ignore-builtin-innodb --plugin-load=innodb=ha_innodb.so --plugin_dir=/home/otto/mariadb-10.0-10.0.20/builddir/storage/innobase --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/tmp/ --core-file --lc-messages-dir=/home/otto/mariadb-10.0-10.0.20/builddir/sql/share/ --character-sets-dir=/home/otto/mariadb-10.0-10.0.20/sql/share/charsets --debug --stack-trace

Comment by Otto Kekäläinen [ 2015-07-24 ]

Now it crashes directly then:

~/builddir-debug$ gdb sql/mysqld
 
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
 
Copyright (C) 2014 Free Software Foundation, Inc.
 
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 
This is free software: you are free to change and redistribute it.
 
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 
and "show warranty" for details.
 
This GDB was configured as "powerpc-linux-gnu".
 
Type "show configuration" for configuration details.
 
For bug reporting instructions, please see:
 
<http://www.gnu.org/software/gdb/bugs/>.
 
Find the GDB manual and other documentation resources online at:
 
<http://www.gnu.org/software/gdb/documentation/>.
 
For help, type "help".
 
Type "apropos word" to search for commands related to "word"...
 
Reading symbols from sql/mysqld...done.
 
(gdb) set height 0
 
(gdb) set logging file /tmp/mysqld.log
 
(gdb) set logging on
 
Copying output to /tmp/mysqld.log.
 
(gdb) run --no-defaults --bootstrap --basedir=/home/otto/mariadb-10.0-10.0.20 --datadir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/install.db --default-storage-engine=myisam --skip-plugin-archive --skip-plugin-aria --skip-plugin-blackhole --skip-plugin-federated --skip-plugin-feedback --skip-plugin-innodb --skip-plugin-innodb-buffer-page --skip-plugin-innodb-buffer-page-lru --skip-plugin-innodb-buffer-pool-stats --skip-plugin-innodb-changed-pages --skip-plugin-innodb-cmp --skip-plugin-innodb-cmp-per-index --skip-plugin-innodb-cmp-per-index-reset --skip-plugin-innodb-cmp-reset --skip-plugin-innodb-cmpmem --skip-plugin-innodb-cmpmem-reset --skip-plugin-innodb-ft-being-deleted --skip-plugin-innodb-ft-config --skip-plugin-innodb-ft-default-stopword --skip-plugin-innodb-ft-deleted --skip-plugin-innodb-ft-index-cache --skip-plugin-innodb-ft-index-table --skip-plugin-innodb-lock-waits --skip-plugin-innodb-locks --skip-plugin-innodb-metrics --skip-plugin-innodb-sys-columns --skip-plugin-innodb-sys-datafiles --skip-plugin-innodb-sys-fields --skip-plugin-innodb-sys-foreign --skip-plugin-innodb-sys-foreign-cols --skip-plugin-innodb-sys-indexes --skip-plugin-innodb-sys-tables --skip-plugin-innodb-sys-tablespaces --skip-plugin-innodb-sys-tablestats --skip-plugin-innodb-trx --skip-plugin-partition --skip-plugin-xtradb-internal-hash-tables --skip-plugin-xtradb-read-view --skip-plugin-xtradb-rseg --loose-innodb --ignore-builtin-innodb --plugin-load=innodb=ha_innodb.so --plugin_dir=/home/otto/mariadb-10.0-10.0.20/builddir/storage/innobase --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/tmp/ --core-file --lc-messages-dir=/home/otto/mariadb-10.0-10.0.20/builddir/sql/share/ --character-sets-dir=/home/otto/mariadb-10.0-10.0.20/sql/share/charsets --debug --stack-trace
 
Starting program: /home/otto/builddir-debug/sql/mysqld --no-defaults --bootstrap --basedir=/home/otto/mariadb-10.0-10.0.20 --datadir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/install.db --default-storage-engine=myisam --skip-plugin-archive --skip-plugin-aria --skip-plugin-blackhole --skip-plugin-federated --skip-plugin-feedback --skip-plugin-innodb --skip-plugin-innodb-buffer-page --skip-plugin-innodb-buffer-page-lru --skip-plugin-innodb-buffer-pool-stats --skip-plugin-innodb-changed-pages --skip-plugin-innodb-cmp --skip-plugin-innodb-cmp-per-index --skip-plugin-innodb-cmp-per-index-reset --skip-plugin-innodb-cmp-reset --skip-plugin-innodb-cmpmem --skip-plugin-innodb-cmpmem-reset --skip-plugin-innodb-ft-being-deleted --skip-plugin-innodb-ft-config --skip-plugin-innodb-ft-default-stopword --skip-plugin-innodb-ft-deleted --skip-plugin-innodb-ft-index-cache --skip-plugin-innodb-ft-index-table --skip-plugin-innodb-lock-waits --skip-plugin-innodb-locks --skip-plugin-innodb-metrics --skip-plugin-innodb-sys-columns --skip-plugin-innodb-sys-datafiles --skip-plugin-innodb-sys-fields --skip-plugin-innodb-sys-foreign --skip-plugin-innodb-sys-foreign-cols --skip-plugin-innodb-sys-indexes --skip-plugin-innodb-sys-tables --skip-plugin-innodb-sys-tablespaces --skip-plugin-innodb-sys-tablestats --skip-plugin-innodb-trx --skip-plugin-partition --skip-plugin-xtradb-internal-hash-tables --skip-plugin-xtradb-read-view --skip-plugin-xtradb-rseg --loose-innodb --ignore-builtin-innodb --plugin-load=innodb=ha_innodb.so --plugin_dir=/home/otto/mariadb-10.0-10.0.20/builddir/storage/innobase --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/tmp/ --core-file --lc-messages-dir=/home/otto/mariadb-10.0-10.0.20/builddir/sql/share/ --character-sets-dir=/home/otto/mariadb-10.0-10.0.20/sql/share/charsets --debug --stack-trace
 
warning: Could not load shared library symbols for linux-vdso32.so.1.
 
Do you need "set solib-search-path" or "set sysroot"?
 
[Thread debugging using libthread_db enabled]
 
Using host libthread_db library "/lib/powerpc-linux-gnu/libthread_db.so.1".
 
150724  8:27:16 [Note] /home/otto/builddir-debug/sql/mysqld (mysqld 10.0.20-MariaDB-0+deb8u1-debug) starting as process 17729 ...
 
150724  8:27:16 [Note] Plugin 'ARCHIVE' is disabled.
 
150724  8:27:16 [Note] Plugin 'Aria' is disabled.
 
150724  8:27:16 [Note] Plugin 'BLACKHOLE' is disabled.
 
150724  8:27:16 [Note] Plugin 'XTRADB_READ_VIEW' is disabled.
 
150724  8:27:16 [Note] Plugin 'XTRADB_INTERNAL_HASH_TABLES' is disabled.
 
150724  8:27:16 [Note] Plugin 'XTRADB_RSEG' is disabled.
 
150724  8:27:16 [Note] Plugin 'FEDERATED' is disabled.
 
150724  8:27:16 [Note] Plugin 'FEEDBACK' is disabled.
 
150724  8:27:16 [Note] Plugin 'partition' is disabled.
 
150724  8:27:17 [Note] InnoDB: Using mutexes to ref count buffer pool pages
 
2015-07-24 08:27:17 f7fe78d0 InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
 
2015-07-24 08:27:17 f7fe78d0 InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!
 
150724  8:27:17 [Note] InnoDB: The InnoDB memory heap is disabled
 
150724  8:27:17 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
150724  8:27:17 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
 
150724  8:27:17 [Note] InnoDB: Compressed tables use zlib 1.2.8
 
150724  8:27:17 [Note] InnoDB: Using Linux native AIO
 
150724  8:27:17 [Note] InnoDB: Not using CPU crc32 instructions
 
150724  8:27:17 [Note] InnoDB: Initializing buffer pool, size = 128.0M
 
150724  8:27:17 [Note] InnoDB: Completed initialization of buffer pool
 
[New Thread 0xe57ff430 (LWP 17734)]
 
[New Thread 0xe4fff430 (LWP 17735)]
 
[New Thread 0xe47ff430 (LWP 17736)]
 
[New Thread 0xe3fff430 (LWP 17737)]
 
[New Thread 0xe37ff430 (LWP 17738)]
 
[New Thread 0xe2fff430 (LWP 17739)]
 
[New Thread 0xe27ff430 (LWP 17740)]
 
[New Thread 0xe1fff430 (LWP 17741)]
 
[New Thread 0xe17ff430 (LWP 17742)]
 
[New Thread 0xe0fff430 (LWP 17743)]
 
150724  8:27:17 [Note] InnoDB: Highest supported file format is Barracuda.
 
[New Thread 0xdffff430 (LWP 17758)]
 
[Thread 0xdffff430 (LWP 17758) exited]
 
150724  8:27:18 [Note] InnoDB: 128 rollback segment(s) are active.
 
[New Thread 0xdffff430 (LWP 17759)]
 
[New Thread 0xdf3ff430 (LWP 17760)]
 
[New Thread 0xdebff430 (LWP 17761)]
 
[New Thread 0xde3ff430 (LWP 17762)]
 
[New Thread 0xddbff430 (LWP 17763)]
 
150724  8:27:18 [Note] InnoDB: Waiting for purge to start
 
[New Thread 0xdcbff430 (LWP 17764)]
 
150724  8:27:18 [Note] InnoDB: 5.6.25 started; log sequence number 1629504
 
[New Thread 0xdc3ff430 (LWP 17765)]
 
[New Thread 0xdbbff430 (LWP 17766)]
 
[New Thread 0xdb3ff430 (LWP 17767)]
 
150724  8:27:18 [ERROR] /home/otto/builddir-debug/sql/mysqld: unknown option '--skip-plugin-innodb-buffer-page'
 
150724  8:27:18 [ERROR] Aborting
 
 
 
150724  8:27:18 [Note] InnoDB: FTS optimize thread exiting.
 
150724  8:27:18 [Note] InnoDB: Starting shutdown...
 
[Thread 0xdb3ff430 (LWP 17767) exited]
 
[Thread 0xdc3ff430 (LWP 17765) exited]
 
[Thread 0xdbbff430 (LWP 17766) exited]
 
[Thread 0xdffff430 (LWP 17759) exited]
 
[Thread 0xdebff430 (LWP 17761) exited]
 
[Thread 0xdf3ff430 (LWP 17760) exited]
 
[Thread 0xddbff430 (LWP 17763) exited]
 
[Thread 0xdcbff430 (LWP 17764) exited]
 
[Thread 0xde3ff430 (LWP 17762) exited]
 
[Thread 0xe57ff430 (LWP 17734) exited]
 
[Thread 0xe0fff430 (LWP 17743) exited]
 
[Thread 0xe47ff430 (LWP 17736) exited]
 
[Thread 0xe17ff430 (LWP 17742) exited]
 
[Thread 0xe3fff430 (LWP 17737) exited]
 
[Thread 0xe1fff430 (LWP 17741) exited]
 
[Thread 0xe4fff430 (LWP 17735) exited]
 
[Thread 0xe27ff430 (LWP 17740) exited]
 
[Thread 0xe2fff430 (LWP 17739) exited]
 
[Thread 0xe37ff430 (LWP 17738) exited]
 
 
 
Program received signal SIGSEGV, Segmentation fault.
 
0x109ea740 in je_arena_dalloc_bin_locked ()
 
(gdb) bt
 
#0  0x109ea740 in je_arena_dalloc_bin_locked ()
 
#1  0x10a0566c in je_tcache_bin_flush_small ()
 
#2  0x109df5f8 in free ()
 
#3  0x0f959b80 in ut_free (ptr=ptr@entry=0xf7468460) at /home/otto/mariadb-10.0-10.0.20/storage/innobase/ut/ut0mem.cc:223
 
#4  0x0f84ecbc in os_event_free (event=0xf7468460) at /home/otto/mariadb-10.0-10.0.20/storage/innobase/os/os0sync.cc:540
 
#5  0x0f924008 in mutex_free_func (mutex=mutex@entry=0xf9d5ba0 <row_drop_list_mutex>) at /home/otto/mariadb-10.0-10.0.20/storage/innobase/sync/sync0sync.cc:361
 
#6  0x0f8c2cbc in pfs_mutex_free_func (mutex=0xf9d5ba0 <row_drop_list_mutex>) at /home/otto/mariadb-10.0-10.0.20/storage/innobase/include/sync0sync.ic:362
 
#7  row_mysql_close () at /home/otto/mariadb-10.0-10.0.20/storage/innobase/row/row0mysql.cc:5458
 
#8  0x0f91c6c8 in innobase_shutdown_for_mysql () at /home/otto/mariadb-10.0-10.0.20/storage/innobase/srv/srv0start.cc:3130
 
#9  0x0f7ca0d0 in innobase_end (hton=<optimized out>, type=<optimized out>) at /home/otto/mariadb-10.0-10.0.20/storage/innobase/handler/ha_innodb.cc:3476
 
#10 0x1034a458 in ha_finalize_handlerton (plugin=0xf745c5c0) at /home/otto/mariadb-10.0-10.0.20/sql/handler.cc:440
 
#11 0x101a8e80 in plugin_deinitialize (plugin=0xf745c5c0, ref_check=ref_check@entry=true) at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1176
 
#12 0x101acf58 in reap_plugins () at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1249
 
#13 0x101ad96c in plugin_shutdown () at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1895
 
#14 0x100ef320 in clean_up (print_message=<optimized out>) at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:2029
 
#15 0x100f6c54 in unireg_abort (exit_code=2) at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:1954
 
#16 0x100f804c in init_server_components () at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:4862
 
#17 0x100fcea8 in mysqld_main (argc=57, argv=0xf7409e30) at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:5434
 
#18 0x100ee8d0 in main (argc=<optimized out>, argv=<optimized out>) at /home/otto/mariadb-10.0-10.0.20/sql/main.cc:25
 
(gdb) thr apply all bt full
 
 
 
Thread 1 (Thread 0xf7fe78d0 (LWP 17729)):
 
#0  0x109ea740 in je_arena_dalloc_bin_locked ()
 
No symbol table info available.
 
#1  0x10a0566c in je_tcache_bin_flush_small ()
 
No symbol table info available.
 
#2  0x109df5f8 in free ()
 
No symbol table info available.
 
#3  0x0f959b80 in ut_free (ptr=ptr@entry=0xf7468460) at /home/otto/mariadb-10.0-10.0.20/storage/innobase/ut/ut0mem.cc:223
 
        block = <optimized out>
 
#4  0x0f84ecbc in os_event_free (event=0xf7468460) at /home/otto/mariadb-10.0-10.0.20/storage/innobase/os/os0sync.cc:540
 
No locals.
 
#5  0x0f924008 in mutex_free_func (mutex=mutex@entry=0xf9d5ba0 <row_drop_list_mutex>) at /home/otto/mariadb-10.0-10.0.20/storage/innobase/sync/sync0sync.cc:361
 
No locals.
 
#6  0x0f8c2cbc in pfs_mutex_free_func (mutex=0xf9d5ba0 <row_drop_list_mutex>) at /home/otto/mariadb-10.0-10.0.20/storage/innobase/include/sync0sync.ic:362
 
No locals.
 
#7  row_mysql_close () at /home/otto/mariadb-10.0-10.0.20/storage/innobase/row/row0mysql.cc:5458
 
No locals.
 
#8  0x0f91c6c8 in innobase_shutdown_for_mysql () at /home/otto/mariadb-10.0-10.0.20/storage/innobase/srv/srv0start.cc:3130
 
        i = <optimized out>
 
#9  0x0f7ca0d0 in innobase_end (hton=<optimized out>, type=<optimized out>) at /home/otto/mariadb-10.0-10.0.20/storage/innobase/handler/ha_innodb.cc:3476
 
        err = 0
 
        _db_stack_frame_ = {func = 0x10ac98d8 "ha_finalize_handlerton", file = 0x10ac9418 "/home/otto/mariadb-10.0-10.0.20/sql/handler.cc", level = 2147483653, prev = 0xfffee7f8}
 
        __PRETTY_FUNCTION__ = "int innobase_end(handlerton*, ha_panic_function)"
 
#10 0x1034a458 in ha_finalize_handlerton (plugin=0xf745c5c0) at /home/otto/mariadb-10.0-10.0.20/sql/handler.cc:440
 
        hton = 0xf7416838
 
        _db_stack_frame_ = {func = 0x10a33bec "plugin_shutdown", file = 0x10a3347c "/home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc", level = 2147483652, prev = 0xfffee928}
 
        __PRETTY_FUNCTION__ = "int ha_finalize_handlerton(st_plugin_int*)"
 
#11 0x101a8e80 in plugin_deinitialize (plugin=0xf745c5c0, ref_check=ref_check@entry=true) at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1176
 
        __PRETTY_FUNCTION__ = "void plugin_deinitialize(st_plugin_int*, bool)"
 
#12 0x101acf58 in reap_plugins () at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1249
 
        count = <optimized out>
 
        idx = <optimized out>
 
        reap = 0xfffee894
 
        list = 0xfffee890
 
        __PRETTY_FUNCTION__ = "void reap_plugins()"
 
        plugin = <optimized out>
 
#13 0x101ad96c in plugin_shutdown () at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1895
 
        i = <optimized out>
 
        count = 18
 
        dl = <optimized out>
 
        plugins = <optimized out>
 
        plugin = <optimized out>
 
        _db_stack_frame_ = {func = 0x10a1126c "unireg_abort", file = 0x10a107c0 "/home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc", level = 2147483651, prev = 0xfffee9b8}
 
#14 0x100ef320 in clean_up (print_message=<optimized out>) at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:2029
 
No locals.
 
#15 0x100f6c54 in unireg_abort (exit_code=2) at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:1954
 
        _db_stack_frame_ = {func = 0x10a1178c "init_server_components", file = 0x10a107c0 "/home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc", level = 2147483650, prev = 0xfffeebe8}
 
#16 0x100f804c in init_server_components () at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:4862
 
        ho_error = <optimized out>
 
        no_opts = {{name = 0x0, id = 0, comment = 0x0, value = 0x0, u_max_value = 0x0, typelib = 0x0, var_type = 1, arg_type = NO_ARG, def_value = 0, min_value = 0, max_value = 0, sub_size = 0, block_size = 0, 
            app_type = 0x0}}
 
        __PRETTY_FUNCTION__ = "int init_server_components()"
 
        _db_stack_frame_ = {func = 0x10b94f78 "?func", file = 0x10b94f80 "?file", level = 2147483649, prev = 0x0}
 
        name = {str = 0x10e96ad8 <sql_functions_map+5412> "", length = 4294896640}
 
        plugin = <optimized out>
 
#17 0x100fcea8 in mysqld_main (argc=57, argv=0xf7409e30) at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:5434
 
        ho_error = <optimized out>
 
#18 0x100ee8d0 in main (argc=<optimized out>, argv=<optimized out>) at /home/otto/mariadb-10.0-10.0.20/sql/main.cc:25
 
No locals.

Comment by Jan Lindström (Inactive) [ 2015-07-24 ]

These two crashes are on very different places in shutdown, but indicates memory corruption earlier.

Comment by Otto Kekäläinen [ 2015-07-24 ]

Looking more into powerpc logs I noticed that even though this failure is not visible in 10.0.19 builds, it is visible in some older builds that that.

See https://buildd.debian.org/status/logs.php?pkg=mariadb-10.0&arch=powerpc

  • first ever build 10.0.10-1 passed despite test suite failures in: rpl.rpl_gtid_until rpl.rpl_partition_innodb rpl.rpl_innodb_bug30888 rpl.rpl_mixed_binlog_max_cache_size rpl.rpl_mixed_implicit_commit_binlog rpl.rpl_relayrotate rpl.rpl_checksum_cache rpl.rpl_relay_space_innodb
  • These were fixed and since 10.0.14-1 the test suite has failed with: connect.part_file connect.part_table connect.alter connect.bin connect.dbf connect.index connect.updelx main.key_cache. These connect failures are most likely still there, though we haven't paid notice to them since the build passes (which is wrong, failing tests should stop the build from proceeding)
  • In build 10.0.15-1 build failed at 47% with: /«PKGBUILDDIR»/sql/rpl_parallel.cc:1126:39: error: 'class Relay_log_info' has no member named 'inuse_relaylog_atomic_lock'
  • the above was fixed in 10.0.15-2~exp1 and the compilation passes, but test suite fails to start with the exact same symptom as in 10.0.20 still present: hangs on "Installing system database..."
  • In 10.0.15-2 the tests start again and same connect tests from 10.0.14-1 fail.
  • In the rest of the builds from time to time the testsuite fails to start, or starts but connect tests fail to pass
Comment by Otto Kekäläinen [ 2015-07-24 ]

Grepping though the commit log for powerpc I found this:
https://github.com/ottok/mariadb-10.0/commit/3b66ec189bdda12c2bf899fec87fcff7ee4f2bbf

Author: Kristian Nielsen <knielsen@knielsen-hq.org>
 
Subject: Fix for FTBFS on mips, mipsel, powerpc introduced in 10.0.15
 
 Backported by Otto Kekäläinen from 10.0.16
 
Last-Update: 2014-12-12
 
 
 
=== modified file 'sql/rpl_parallel.cc'
 
--- a/sql/rpl_parallel.cc	2014-12-01 12:53:57 +0000
 
+++ b/sql/rpl_parallel.cc	2014-12-12 12:48:04 +0000
 
@@ -1123,9 +1123,9 @@ rpl_parallel_thread::inuse_relaylog_refc
 
   inuse_relaylog *ir= accumulated_ir_last;
 
   if (ir)
 
   {
 
-    my_atomic_rwlock_wrlock(&ir->rli->inuse_relaylog_atomic_lock);
 
+    my_atomic_rwlock_wrlock(&ir->inuse_relaylog_atomic_lock);
 
     my_atomic_add64(&ir->dequeued_count, accumulated_ir_count);
 
-    my_atomic_rwlock_wrunlock(&ir->rli->inuse_relaylog_atomic_lock);
 
+    my_atomic_rwlock_wrunlock(&ir->inuse_relaylog_atomic_lock);
 
     accumulated_ir_count= 0;
 
     accumulated_ir_last= NULL;
 
   }

Comment by Otto Kekäläinen [ 2015-07-24 ]

Here is a i386 build log where the test suite failed and it correctly stopped the whole build: https://buildd.debian.org/status/fetch.php?pkg=mariadb-10.0&arch=i386&ver=10.0.17-1~exp1&stamp=1425382881

I don't know why test suite failures don't stop builds on powerpc. Ideas?

Even a single test failure should always stop the build from completing.

Comment by Otto Kekäläinen [ 2015-07-27 ]

Now the build passed successfully: https://buildd.debian.org/status/fetch.php?pkg=mariadb-10.0&arch=powerpc&ver=10.0.20-3&stamp=1437946958

There was however one failing test: connect.json_udf

Conclusions:

  • this error has certain randomness
  • build should not complete if there is even a single test suite failure, otherwise the false positives will trigger false negatives in build status changes later
Comment by Otto Kekäläinen [ 2015-07-28 ]

Here is a build log from ppc64 (big endian, just like powerpc?) that has ended in the same way, that is test suite never starts but stalls at "Installing system databases..":

make[5]: Leaving directory '/«PKGBUILDDIR»/builddir'
 
/usr/bin/make -f mysql-test/CMakeFiles/test-force.dir/build.make mysql-test/CMakeFiles/test-force.dir/build
 
make[5]: Entering directory '/«PKGBUILDDIR»/builddir'
 
cd /«PKGBUILDDIR»/builddir/mysql-test && echo Running tests
 
Running tests
 
cd /«PKGBUILDDIR»/builddir/mysql-test && echo OS=Linux
 
OS=Linux
 
cd /«PKGBUILDDIR»/builddir/mysql-test && export MTR_BUILD_THREAD=auto
 
cd /«PKGBUILDDIR»/builddir/mysql-test && perl ./mysql-test-run.pl --force --testcase-timeout=30 --suite-timeout=540
 
Logging: /«PKGBUILDDIR»/mysql-test/mysql-test-run.pl  --force --testcase-timeout=30 --suite-timeout=540
 
vardir: /«PKGBUILDDIR»/builddir/mysql-test/var
 
Removing old var directory...
 
Creating var directory '/«PKGBUILDDIR»/builddir/mysql-test/var'...
 
Checking supported features...
 
MariaDB Version 10.0.20-MariaDB-3
 
 - SSL connections supported
 
Using suites: main-,archive-,binlog-,csv-,federated-,funcs_1-,funcs_2-,handler-,heap-,innodb-,innodb_fts-,innodb_zip-,maria-,multi_source-,optimizer_unfixed_bugs-,parts-,percona-,perfschema-,plugins-,roles-,rpl-,sys_vars-,unit-,vcol-,connect,oqgraph,sequence,spider,spider/bg,sql_discovery,metadata_lock_info,query_response_time
 
Collecting tests...
 
Installing system database...
 
...
 
Build killed with signal TERM after 300 minutes of inactivity

http://buildd.debian-ports.org/status/fetch.php?pkg=mariadb-10.0&arch=ppc64&ver=10.0.20-3&stamp=1437975386

Comment by Otto Kekäläinen [ 2015-07-28 ]

Test suite false positives is due to https://github.com/ottok/mariadb-10.0/blob/master/debian/rules#L68

ifneq (,$(filter $(ARCH), amd64 kfreebsd-amd64 i386 kfreebsd-i386 ia64 s390 s390x))
 
    TESTSUITE_FAIL_CMD:=exit 1
 
else
 
    TESTSUITE_FAIL_CMD:=true
 
endif

This is inherited from mysql-5.6 packaging. Since mysql-5.6 maintainers have even chosen to no fail on any platform due to test suite failures...
https://anonscm.debian.org/cgit/pkg-mysql/mysql-5.6.git/tree/debian/rules?id=refs/heads/master#n28

Comment by Otto Kekäläinen [ 2015-08-28 ]

jplindst For the record, ppc64 still crashes randomly when trying to start the test suite. See latest log at https://buildd.debian.org/status/fetch.php?pkg=mariadb-10.0&arch=ppc64&ver=10.0.21-2&stamp=1440738838

Comment by Jan Lindström (Inactive) [ 2015-10-27 ]

I have now done the diagnostics, can you try this on powerpc and ppc64, branch is bb-10.0-jplindst, If still repeats, I need output from bootstrap.log

Comment by Otto Kekäläinen [ 2015-10-27 ]

Sorry, but getting the branch, preparing tar.gz dump, uploading to to different machines (and getting access to them), building manually etc is a bit too much right now.

What kind of fixes did you do, can you package them into a patch that would be uploaded along the next Debian upload? (see example https://github.com/ottok/mariadb-10.0/commit/87995f846df919b77de389ad509532a82c3ca91c)

Is bootstrap.log generally useful as diagnostics, should the build process on Debian systems maybe always cat the bootstrap.log into the build log output?

Comment by Jan Lindström (Inactive) [ 2015-10-27 ]

This just contains the additional diagnostics, not intended to remain there when the real fix is found. I do not know the reason, I do not have machine where this repeats, thus I need something. Yes, boostrap.log output could be useful, but not for this case without additional info I added. I do not want that this change is added to real packages.

Comment by Otto Kekäläinen [ 2015-10-27 ]

I can have the patch for a while in the Debian master branch and upload to Debian experimental, from where it will not propagate to "real" packages.

Comment by Jan Lindström (Inactive) [ 2015-10-27 ]

https://github.com/ottok/mariadb-10.0/pull/29

Comment by Jan Lindström (Inactive) [ 2015-10-28 ]

https://github.com/ottok/mariadb-10.0/pull/30

Comment by Jan Lindström (Inactive) [ 2015-12-09 ]

Hi,

Yes, the diagnostic patch revealed the fact that some of the syncronization primitives does not work in these platforms, I do not know how to fix that as the actual hw is not familiar to me, maybe Sergey Vojtovich, would know where the actual problem is.

R: Jan

Comment by Sergey Vojtovich [ 2016-02-11 ]

I may try looking into this, but I need more details:

  • it's unclear what is the problem we're aiming to fix within the scope of this bug. Some comments say hang, some crash, some link to pages which are unavailable. Could you confirm we're still concentrated on the originally reported problem.
  • it's unclear if it's regression or not. It's safe to assume it is not, but if some versions are known to work we could reduce scope of the search.
  • it's unclear how exactly synchronization primitives can affect originally reported crash

If that's really about synchronization primitives, chances that we get this bug fixed without full access to affected system are rather low.

Comment by Otto Kekäläinen [ 2016-02-11 ]

The build history of powerpc in Debian is visible at: https://buildd.debian.org/status/logs.php?pkg=mariadb-10.0&arch=powerpc

According to that page, the powerpc builds failed in 20-1 and 20-2 but have been green since 10.0.20-3. Changes between 20-2 and 20-3 are visible at https://github.com/ottok/mariadb-10.0/compare/debian/10.0.20-2...debian/10.0.20-3

What was the root cause and the fix? I don't know. I don't see anything obivious in the changes at https://github.com/ottok/mariadb-10.0/compare/debian/10.0.20-2...debian/10.0.20-3
Are we maybe mixing two different issues in one. There are so many comments that it is difficult to get on track here.

Comment by Sergey Vojtovich [ 2016-02-11 ]

otto, jplindst, if the problem is not reproducible anymore should we close this bug?

Comment by Otto Kekäläinen [ 2016-02-12 ]

Let's wait a bit, I'll upload a debug build to Debian experimental in coming weeks and then review all bugs labelled Platform Debian to close issues that don't reproduce anymore.

Comment by Sergei Golubchik [ 2017-06-26 ]

otto, it's been more than a year. Shall we close it?

Comment by Otto Kekäläinen [ 2017-06-27 ]

Looking at https://buildd.debian.org/status/logs.php?pkg=mariadb-10.0&arch=powerpc we can see many successfull builds since 10.0.20-3.

The latest build https://buildd.debian.org/status/fetch.php?pkg=mariadb-10.0&arch=powerpc&ver=10.0.30-0%2Bdeb8u2&stamp=1493574340&raw=0 failed, but to other reasons than described in this bug report.

I don't see this issue repeating anymore, so closing..

Generated at Thu Feb 08 07:27:53 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.