[MDEV-8536] Crash in os_event_free on shutdown - Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 10.0.20
Fix Version/s: N/A
Component/s: Platform Debian, Platform Power
Labels:
None

Sprint:
10.0.22, 10.1.9-1, 10.1.9-2

Description

This crash comes from debian build system: https://buildd.debian.org/status/package.php?p=mariadb-10.0

Server crashes during shutdown of mtr bootstrap instance. Looks like either double free or memory corruption.

(gdb) --no-defaults --bootstrap --basedir=/home/otto/mariadb-10.0-10.0.20 --datadir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/install.db --default-storage-engine=myisam --skip-plugin-archive --skip-plugin-aria --skip-plugin-blackhole --skip-plugin-federated --skip-plugin-feedback --skip-plugin-innodb --skip-plugin-innodb-buffer-page --skip-plugin-innodb-buffer-page-lru --skip-plugin-innodb-buffer-pool-stats --skip-plugin-innodb-changed-pages --skip-plugin-innodb-cmp --skip-plugin-innodb-cmp-per-index --skip-plugin-innodb-cmp-per-index-reset --skip-plugin-innodb-cmp-reset --skip-plugin-innodb-cmpmem --skip-plugin-innodb-cmpmem-reset --skip-plugin-innodb-ft-being-deleted --skip-plugin-innodb-ft-config --skip-plugin-innodb-ft-default-stopword --skip-plugin-innodb-ft-deleted --skip-plugin-innodb-ft-index-cache --skip-plugin-innodb-ft-index-table --skip-plugin-innodb-lock-waits --skip-plugin-innodb-locks --skip-plugin-innodb-metrics --skip-plugin-innodb-sys-columns --skip-plugin-innodb-sys-datafiles --skip-plugin-innodb-sys-fields --skip-plugin-innodb-sys-foreign --skip-plugin-innodb-sys-foreign-cols --skip-plugin-innodb-sys-indexes --skip-plugin-innodb-sys-tables --skip-plugin-innodb-sys-tablespaces --skip-plugin-innodb-sys-tablestats --skip-plugin-innodb-trx --skip-plugin-partition --skip-plugin-xtradb-internal-hash-tables --skip-plugin-xtradb-read-view --skip-plugin-xtradb-rseg --loose-innodb --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/tmp/ --core-file --lc-messages-dir=/home/otto/mariadb-10.0-10.0.20/builddir/sql/share/ --character-sets-dir=/home/otto/mariadb-10.0-10.0.20/sql/share/charsets --debug --stack-trace

Undefined command: "--no-defaults".  Try "help".

(gdb) run --no-defaults --bootstrap --basedir=/home/otto/mariadb-10.0-10.0.20 --datadir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/install.db --default-storage-engine=myisam --skip-plugin-archive --skip-plugin-aria --skip-plugin-blackhole --skip-plugin-federated --skip-plugin-feedback --skip-plugin-innodb --skip-plugin-innodb-buffer-page --skip-plugin-innodb-buffer-page-lru --skip-plugin-innodb-buffer-pool-stats --skip-plugin-innodb-changed-pages --skip-plugin-innodb-cmp --skip-plugin-innodb-cmp-per-index --skip-plugin-innodb-cmp-per-index-reset --skip-plugin-innodb-cmp-reset --skip-plugin-innodb-cmpmem --skip-plugin-innodb-cmpmem-reset --skip-plugin-innodb-ft-being-deleted --skip-plugin-innodb-ft-config --skip-plugin-innodb-ft-default-stopword --skip-plugin-innodb-ft-deleted --skip-plugin-innodb-ft-index-cache --skip-plugin-innodb-ft-index-table --skip-plugin-innodb-lock-waits --skip-plugin-innodb-locks --skip-plugin-innodb-metrics --skip-plugin-innodb-sys-columns --skip-plugin-innodb-sys-datafiles --skip-plugin-innodb-sys-fields --skip-plugin-innodb-sys-foreign --skip-plugin-innodb-sys-foreign-cols --skip-plugin-innodb-sys-indexes --skip-plugin-innodb-sys-tables --skip-plugin-innodb-sys-tablespaces --skip-plugin-innodb-sys-tablestats --skip-plugin-innodb-trx --skip-plugin-partition --skip-plugin-xtradb-internal-hash-tables --skip-plugin-xtradb-read-view --skip-plugin-xtradb-rseg --loose-innodb --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/tmp/ --core-file --lc-messages-dir=/home/otto/mariadb-10.0-10.0.20/builddir/sql/share/ --character-sets-dir=/home/otto/mariadb-10.0-10.0.20/sql/share/charsets --debug --stack-trace

Starting program: /home/otto/builddir-debug/sql/mysqld --no-defaults --bootstrap --basedir=/home/otto/mariadb-10.0-10.0.20 --datadir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/install.db --default-storage-engine=myisam --skip-plugin-archive --skip-plugin-aria --skip-plugin-blackhole --skip-plugin-federated --skip-plugin-feedback --skip-plugin-innodb --skip-plugin-innodb-buffer-page --skip-plugin-innodb-buffer-page-lru --skip-plugin-innodb-buffer-pool-stats --skip-plugin-innodb-changed-pages --skip-plugin-innodb-cmp --skip-plugin-innodb-cmp-per-index --skip-plugin-innodb-cmp-per-index-reset --skip-plugin-innodb-cmp-reset --skip-plugin-innodb-cmpmem --skip-plugin-innodb-cmpmem-reset --skip-plugin-innodb-ft-being-deleted --skip-plugin-innodb-ft-config --skip-plugin-innodb-ft-default-stopword --skip-plugin-innodb-ft-deleted --skip-plugin-innodb-ft-index-cache --skip-plugin-innodb-ft-index-table --skip-plugin-innodb-lock-waits --skip-plugin-innodb-locks --skip-plugin-innodb-metrics --skip-plugin-innodb-sys-columns --skip-plugin-innodb-sys-datafiles --skip-plugin-innodb-sys-fields --skip-plugin-innodb-sys-foreign --skip-plugin-innodb-sys-foreign-cols --skip-plugin-innodb-sys-indexes --skip-plugin-innodb-sys-tables --skip-plugin-innodb-sys-tablespaces --skip-plugin-innodb-sys-tablestats --skip-plugin-innodb-trx --skip-plugin-partition --skip-plugin-xtradb-internal-hash-tables --skip-plugin-xtradb-read-view --skip-plugin-xtradb-rseg --loose-innodb --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/tmp/ --core-file --lc-messages-dir=/home/otto/mariadb-10.0-10.0.20/builddir/sql/share/ --character-sets-dir=/home/otto/mariadb-10.0-10.0.20/sql/share/charsets --debug --stack-trace

warning: Could not load shared library symbols for linux-vdso32.so.1.

Do you need "set solib-search-path" or "set sysroot"?

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/powerpc-linux-gnu/libthread_db.so.1".

150723 18:59:05 [Note] /home/otto/builddir-debug/sql/mysqld (mysqld 10.0.20-MariaDB-0+deb8u1-debug) starting as process 13168 ...

150723 18:59:05 [Note] Plugin 'ARCHIVE' is disabled.

150723 18:59:05 [Note] Plugin 'Aria' is disabled.

150723 18:59:05 [Note] Plugin 'BLACKHOLE' is disabled.

150723 18:59:05 [Note] InnoDB: Using mutexes to ref count buffer pool pages

2015-07-23 18:59:05 f7fe78d0 InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!

2015-07-23 18:59:05 f7fe78d0 InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!

150723 18:59:05 [Note] InnoDB: The InnoDB memory heap is disabled

150723 18:59:05 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

150723 18:59:05 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier

150723 18:59:05 [Note] InnoDB: Compressed tables use zlib 1.2.8

150723 18:59:05 [Note] InnoDB: Using Linux native AIO

150723 18:59:05 [Note] InnoDB: Not using CPU crc32 instructions

150723 18:59:05 [Note] InnoDB: Initializing buffer pool, size = 128.0M

150723 18:59:05 [Note] InnoDB: Completed initialization of buffer pool

[New Thread 0xe53ff430 (LWP 13173)]

[New Thread 0xe4bff430 (LWP 13174)]

[New Thread 0xe43ff430 (LWP 13175)]

[New Thread 0xe3bff430 (LWP 13176)]

[New Thread 0xe33ff430 (LWP 13177)]

[New Thread 0xe2bff430 (LWP 13178)]

[New Thread 0xe23ff430 (LWP 13179)]

[New Thread 0xe1bff430 (LWP 13180)]

[New Thread 0xe13ff430 (LWP 13181)]

[New Thread 0xe0bff430 (LWP 13182)]

150723 18:59:05 [Note] InnoDB: Highest supported file format is Barracuda.

[New Thread 0xdfbff430 (LWP 13183)]

150723 18:59:06 [Note] InnoDB: 128 rollback segment(s) are active.

[Thread 0xdfbff430 (LWP 13183) exited]

[New Thread 0xdfbff430 (LWP 13184)]

[New Thread 0xdefff430 (LWP 13185)]

[New Thread 0xde7ff430 (LWP 13186)]

[New Thread 0xddfff430 (LWP 13187)]

[New Thread 0xdd7ff430 (LWP 13188)]

150723 18:59:06 [Note] InnoDB: Waiting for purge to start

[New Thread 0xdc7ff430 (LWP 13189)]

[New Thread 0xdbfff430 (LWP 13190)]

150723 18:59:06 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.24-72.2 started; log sequence number 1629494

[New Thread 0xdb7ff430 (LWP 13191)]

[New Thread 0xdafff430 (LWP 13192)]

[New Thread 0xda7ff430 (LWP 13193)]

150723 18:59:06 [Note] Plugin 'XTRADB_READ_VIEW' is disabled.

150723 18:59:06 [Note] Plugin 'XTRADB_INTERNAL_HASH_TABLES' is disabled.

150723 18:59:06 [Note] Plugin 'XTRADB_RSEG' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_TRX' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_CMP_RESET' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_CMPMEM' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_METRICS' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_FT_DELETED' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.

150723 18:59:06 [Note] Plugin 'INNODB_CHANGED_PAGES' is disabled.

150723 18:59:06 [Note] Plugin 'FEDERATED' is disabled.

150723 18:59:06 [Note] Plugin 'FEEDBACK' is disabled.

150723 18:59:06 [Note] Plugin 'partition' is disabled.

[New Thread 0xf7e87430 (LWP 13194)]

[New Thread 0xf7e27430 (LWP 13195)]

bt

exit;

ERROR: 1064  You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'bt

exit;' at line 1

150723 19:00:29 [ERROR] Aborting

[Thread 0xf7e27430 (LWP 13195) exited]

150723 19:00:29 [Note] InnoDB: FTS optimize thread exiting.

150723 19:00:29 [Note] InnoDB: Starting shutdown...

[Thread 0xda7ff430 (LWP 13193) exited]

[Thread 0xde7ff430 (LWP 13186) exited]

[Thread 0xdb7ff430 (LWP 13191) exited]

[Thread 0xdefff430 (LWP 13185) exited]

[Thread 0xdafff430 (LWP 13192) exited]

[Thread 0xdfbff430 (LWP 13184) exited]

[Thread 0xdd7ff430 (LWP 13188) exited]

[Thread 0xdc7ff430 (LWP 13189) exited]

[Thread 0xddfff430 (LWP 13187) exited]

[Thread 0xe3bff430 (LWP 13176) exited]

[Thread 0xe0bff430 (LWP 13182) exited]

[Thread 0xe53ff430 (LWP 13173) exited]

[Thread 0xe13ff430 (LWP 13181) exited]

[Thread 0xe23ff430 (LWP 13179) exited]

[Thread 0xe4bff430 (LWP 13174) exited]

[Thread 0xe43ff430 (LWP 13175) exited]

[Thread 0xe1bff430 (LWP 13180) exited]

[Thread 0xe33ff430 (LWP 13177) exited]

[Thread 0xe2bff430 (LWP 13178) exited]

[Thread 0xdbfff430 (LWP 13190) exited]

Program received signal SIGSEGV, Segmentation fault.

0x109e8b8c in arena_purge ()

(gdb) bt

#0  0x109e8b8c in arena_purge ()

#1  0x109e8f60 in arena_dalloc_bin_run ()

#2  0x109ea7d8 in je_arena_dalloc_bin_locked ()

#3  0x10a0566c in je_tcache_bin_flush_small ()

#4  0x109df5f8 in free ()

#5  0x10754780 in ut_free (ptr=ptr@entry=0xf77e58e0)

    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/ut/ut0mem.cc:223

#6  0x1065132c in os_event_free (event=0xf77e58e0)

    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/os/os0sync.cc:540

#7  0x106517f4 in os_sync_free ()

    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/os/os0sync.cc:321

#8  0x10713c04 in innobase_shutdown_for_mysql ()

    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/srv/srv0start.cc:3227

#9  0x105be570 in innobase_end (hton=<optimized out>, type=<optimized out>)

    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/handler/ha_innodb.cc:3883

#10 0x1034a458 in ha_finalize_handlerton (plugin=0xf745c368)

    at /home/otto/mariadb-10.0-10.0.20/sql/handler.cc:440

#11 0x101a8e80 in plugin_deinitialize (plugin=0xf745c368,

    ref_check=ref_check@entry=true)

    at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1176

#12 0x101acf58 in reap_plugins ()

    at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1249

#13 0x101ad96c in plugin_shutdown ()

    at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1895

#14 0x100ef320 in clean_up (print_message=<optimized out>)

    at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:2029

#15 0x100f6c54 in unireg_abort (exit_code=1)

    at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:1954

#16 0x100fd178 in mysqld_main (argc=54, argv=0xf7409e30)

    at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:5510

#17 0x100ee8d0 in main (argc=<optimized out>, argv=<optimized out>)

    at /home/otto/mariadb-10.0-10.0.20/sql/main.cc:25

(gdb) x/10c 0xf77e58e0

0xf77e58e0:	0 '\000'	0 '\000'	0 '\000'	0 '\000'	0 '\000'	0 '\000'	0 '\000'	0 '\000'

0xf77e58e8:	0 '\000'	0 '\000'

(gdb) thr apply all bt full

Thread 23 (Thread 0xf7e87430 (LWP 13194)):

#0  0x0ffd25f0 in do_sigwait () from /lib/powerpc-linux-gnu/libpthread.so.0

No symbol table info available.

#1  0x0ffd26a8 in sigwait () from /lib/powerpc-linux-gnu/libpthread.so.0

No symbol table info available.

#2  0x100f35cc in signal_hand (arg=<optimized out>)

    at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:3346

        error = <optimized out>

        set = {__val = {548869, 0 <repeats 31 times>}}

        sig = 0

        _db_stack_frame_ = {func = 0x10b94f78 "?func", file = 0x10b94f80 "?file",

          level = 2147483649, prev = 0x0}

#3  0x0ffc7c3c in start_thread () from /lib/powerpc-linux-gnu/libpthread.so.0

No symbol table info available.

#4  0x0fb56470 in clone () from /lib/powerpc-linux-gnu/libc.so.6

No symbol table info available.

Thread 1 (Thread 0xf7fe78d0 (LWP 13168)):

#0  0x109e8b8c in arena_purge ()

No symbol table info available.

#1  0x109e8f60 in arena_dalloc_bin_run ()

No symbol table info available.

#2  0x109ea7d8 in je_arena_dalloc_bin_locked ()

No symbol table info available.

#3  0x10a0566c in je_tcache_bin_flush_small ()

No symbol table info available.

#4  0x109df5f8 in free ()

No symbol table info available.

#5  0x10754780 in ut_free (ptr=ptr@entry=0xf77e58e0)

    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/ut/ut0mem.cc:223

        block = <optimized out>

#6  0x1065132c in os_event_free (event=0xf77e58e0)

    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/os/os0sync.cc:540

No locals.

#7  0x106517f4 in os_sync_free ()

    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/os/os0sync.cc:321

        event = <optimized out>

        mutex = <optimized out>

#8  0x10713c04 in innobase_shutdown_for_mysql ()

    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/srv/srv0start.cc:3227

        i = <optimized out>

#9  0x105be570 in innobase_end (hton=<optimized out>, type=<optimized out>)

    at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/handler/ha_innodb.cc:3883

        err = 0

        _db_stack_frame_ = {func = 0x10ac98d8 "ha_finalize_handlerton",

          file = 0x10ac9418 "/home/otto/mariadb-10.0-10.0.20/sql/handler.cc",

          level = 2147483652, prev = 0xfffeea58}

        __PRETTY_FUNCTION__ = "int innobase_end(handlerton*, ha_panic_function)"

#10 0x1034a458 in ha_finalize_handlerton (plugin=0xf745c368)

    at /home/otto/mariadb-10.0-10.0.20/sql/handler.cc:440

        hton = 0xf7416738

        _db_stack_frame_ = {func = 0x10a33bec "plugin_shutdown",

          file = 0x10a3347c "/home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc",

          level = 2147483651, prev = 0xfffeebf8}

        __PRETTY_FUNCTION__ = "int ha_finalize_handlerton(st_plugin_int*)"

#11 0x101a8e80 in plugin_deinitialize (plugin=0xf745c368,

    ref_check=ref_check@entry=true)

    at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1176

        __PRETTY_FUNCTION__ = "void plugin_deinitialize(st_plugin_int*, bool)"

#12 0x101acf58 in reap_plugins ()

    at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1249

        count = <optimized out>

        idx = <optimized out>

        reap = 0xfffeeb00

        list = 0xfffeeaec

        __PRETTY_FUNCTION__ = "void reap_plugins()"

        plugin = <optimized out>

#13 0x101ad96c in plugin_shutdown ()

    at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1895

        i = <optimized out>

        count = 47

        dl = <optimized out>

        plugins = <optimized out>

        plugin = <optimized out>

        _db_stack_frame_ = {func = 0x10a1126c "unireg_abort",

          file = 0x10a107c0 "/home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc",

          level = 2147483650, prev = 0xfffeec88}

#14 0x100ef320 in clean_up (print_message=<optimized out>)

    at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:2029

No locals.

#15 0x100f6c54 in unireg_abort (exit_code=1)

    at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:1954

        _db_stack_frame_ = {func = 0x10b94f78 "?func", file = 0x10b94f80 "?file",

          level = 2147483649, prev = 0x0}

#16 0x100fd178 in mysqld_main (argc=54, argv=0xf7409e30)

    at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:5510

        ho_error = <optimized out>

#17 0x100ee8d0 in main (argc=<optimized out>, argv=<optimized out>)

    at /home/otto/mariadb-10.0-10.0.20/sql/main.cc:25

No locals.

Attachments

Activity

People

Assignee:: Otto Kekäläinen

Reporter:: Sergey Vojtovich

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 2015-07-23 23:02

Updated:: 2017-06-27 09:07

Resolved:: 2017-06-27 06:16

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.