Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-8536

Crash in os_event_free on shutdown

    XMLWordPrintable

    Details

    • Sprint:
      10.0.22, 10.1.9-1, 10.1.9-2

      Description

      This crash comes from debian build system: https://buildd.debian.org/status/package.php?p=mariadb-10.0

      Server crashes during shutdown of mtr bootstrap instance. Looks like either double free or memory corruption.

      (gdb) --no-defaults --bootstrap --basedir=/home/otto/mariadb-10.0-10.0.20 --datadir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/install.db --default-storage-engine=myisam --skip-plugin-archive --skip-plugin-aria --skip-plugin-blackhole --skip-plugin-federated --skip-plugin-feedback --skip-plugin-innodb --skip-plugin-innodb-buffer-page --skip-plugin-innodb-buffer-page-lru --skip-plugin-innodb-buffer-pool-stats --skip-plugin-innodb-changed-pages --skip-plugin-innodb-cmp --skip-plugin-innodb-cmp-per-index --skip-plugin-innodb-cmp-per-index-reset --skip-plugin-innodb-cmp-reset --skip-plugin-innodb-cmpmem --skip-plugin-innodb-cmpmem-reset --skip-plugin-innodb-ft-being-deleted --skip-plugin-innodb-ft-config --skip-plugin-innodb-ft-default-stopword --skip-plugin-innodb-ft-deleted --skip-plugin-innodb-ft-index-cache --skip-plugin-innodb-ft-index-table --skip-plugin-innodb-lock-waits --skip-plugin-innodb-locks --skip-plugin-innodb-metrics --skip-plugin-innodb-sys-columns --skip-plugin-innodb-sys-datafiles --skip-plugin-innodb-sys-fields --skip-plugin-innodb-sys-foreign --skip-plugin-innodb-sys-foreign-cols --skip-plugin-innodb-sys-indexes --skip-plugin-innodb-sys-tables --skip-plugin-innodb-sys-tablespaces --skip-plugin-innodb-sys-tablestats --skip-plugin-innodb-trx --skip-plugin-partition --skip-plugin-xtradb-internal-hash-tables --skip-plugin-xtradb-read-view --skip-plugin-xtradb-rseg --loose-innodb --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/tmp/ --core-file --lc-messages-dir=/home/otto/mariadb-10.0-10.0.20/builddir/sql/share/ --character-sets-dir=/home/otto/mariadb-10.0-10.0.20/sql/share/charsets --debug --stack-trace
      Undefined command: "--no-defaults".  Try "help".
      (gdb) run --no-defaults --bootstrap --basedir=/home/otto/mariadb-10.0-10.0.20 --datadir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/install.db --default-storage-engine=myisam --skip-plugin-archive --skip-plugin-aria --skip-plugin-blackhole --skip-plugin-federated --skip-plugin-feedback --skip-plugin-innodb --skip-plugin-innodb-buffer-page --skip-plugin-innodb-buffer-page-lru --skip-plugin-innodb-buffer-pool-stats --skip-plugin-innodb-changed-pages --skip-plugin-innodb-cmp --skip-plugin-innodb-cmp-per-index --skip-plugin-innodb-cmp-per-index-reset --skip-plugin-innodb-cmp-reset --skip-plugin-innodb-cmpmem --skip-plugin-innodb-cmpmem-reset --skip-plugin-innodb-ft-being-deleted --skip-plugin-innodb-ft-config --skip-plugin-innodb-ft-default-stopword --skip-plugin-innodb-ft-deleted --skip-plugin-innodb-ft-index-cache --skip-plugin-innodb-ft-index-table --skip-plugin-innodb-lock-waits --skip-plugin-innodb-locks --skip-plugin-innodb-metrics --skip-plugin-innodb-sys-columns --skip-plugin-innodb-sys-datafiles --skip-plugin-innodb-sys-fields --skip-plugin-innodb-sys-foreign --skip-plugin-innodb-sys-foreign-cols --skip-plugin-innodb-sys-indexes --skip-plugin-innodb-sys-tables --skip-plugin-innodb-sys-tablespaces --skip-plugin-innodb-sys-tablestats --skip-plugin-innodb-trx --skip-plugin-partition --skip-plugin-xtradb-internal-hash-tables --skip-plugin-xtradb-read-view --skip-plugin-xtradb-rseg --loose-innodb --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/tmp/ --core-file --lc-messages-dir=/home/otto/mariadb-10.0-10.0.20/builddir/sql/share/ --character-sets-dir=/home/otto/mariadb-10.0-10.0.20/sql/share/charsets --debug --stack-trace
      Starting program: /home/otto/builddir-debug/sql/mysqld --no-defaults --bootstrap --basedir=/home/otto/mariadb-10.0-10.0.20 --datadir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/install.db --default-storage-engine=myisam --skip-plugin-archive --skip-plugin-aria --skip-plugin-blackhole --skip-plugin-federated --skip-plugin-feedback --skip-plugin-innodb --skip-plugin-innodb-buffer-page --skip-plugin-innodb-buffer-page-lru --skip-plugin-innodb-buffer-pool-stats --skip-plugin-innodb-changed-pages --skip-plugin-innodb-cmp --skip-plugin-innodb-cmp-per-index --skip-plugin-innodb-cmp-per-index-reset --skip-plugin-innodb-cmp-reset --skip-plugin-innodb-cmpmem --skip-plugin-innodb-cmpmem-reset --skip-plugin-innodb-ft-being-deleted --skip-plugin-innodb-ft-config --skip-plugin-innodb-ft-default-stopword --skip-plugin-innodb-ft-deleted --skip-plugin-innodb-ft-index-cache --skip-plugin-innodb-ft-index-table --skip-plugin-innodb-lock-waits --skip-plugin-innodb-locks --skip-plugin-innodb-metrics --skip-plugin-innodb-sys-columns --skip-plugin-innodb-sys-datafiles --skip-plugin-innodb-sys-fields --skip-plugin-innodb-sys-foreign --skip-plugin-innodb-sys-foreign-cols --skip-plugin-innodb-sys-indexes --skip-plugin-innodb-sys-tables --skip-plugin-innodb-sys-tablespaces --skip-plugin-innodb-sys-tablestats --skip-plugin-innodb-trx --skip-plugin-partition --skip-plugin-xtradb-internal-hash-tables --skip-plugin-xtradb-read-view --skip-plugin-xtradb-rseg --loose-innodb --loose-innodb-log-file-size=5M --disable-sync-frm --tmpdir=/home/otto/mariadb-10.0-10.0.20/builddir/mysql-test/var/tmp/ --core-file --lc-messages-dir=/home/otto/mariadb-10.0-10.0.20/builddir/sql/share/ --character-sets-dir=/home/otto/mariadb-10.0-10.0.20/sql/share/charsets --debug --stack-trace
      warning: Could not load shared library symbols for linux-vdso32.so.1.
      Do you need "set solib-search-path" or "set sysroot"?
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/powerpc-linux-gnu/libthread_db.so.1".
      150723 18:59:05 [Note] /home/otto/builddir-debug/sql/mysqld (mysqld 10.0.20-MariaDB-0+deb8u1-debug) starting as process 13168 ...
      150723 18:59:05 [Note] Plugin 'ARCHIVE' is disabled.
      150723 18:59:05 [Note] Plugin 'Aria' is disabled.
      150723 18:59:05 [Note] Plugin 'BLACKHOLE' is disabled.
      150723 18:59:05 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      2015-07-23 18:59:05 f7fe78d0 InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
      2015-07-23 18:59:05 f7fe78d0 InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!
      150723 18:59:05 [Note] InnoDB: The InnoDB memory heap is disabled
      150723 18:59:05 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      150723 18:59:05 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
      150723 18:59:05 [Note] InnoDB: Compressed tables use zlib 1.2.8
      150723 18:59:05 [Note] InnoDB: Using Linux native AIO
      150723 18:59:05 [Note] InnoDB: Not using CPU crc32 instructions
      150723 18:59:05 [Note] InnoDB: Initializing buffer pool, size = 128.0M
      150723 18:59:05 [Note] InnoDB: Completed initialization of buffer pool
      [New Thread 0xe53ff430 (LWP 13173)]
      [New Thread 0xe4bff430 (LWP 13174)]
      [New Thread 0xe43ff430 (LWP 13175)]
      [New Thread 0xe3bff430 (LWP 13176)]
      [New Thread 0xe33ff430 (LWP 13177)]
      [New Thread 0xe2bff430 (LWP 13178)]
      [New Thread 0xe23ff430 (LWP 13179)]
      [New Thread 0xe1bff430 (LWP 13180)]
      [New Thread 0xe13ff430 (LWP 13181)]
      [New Thread 0xe0bff430 (LWP 13182)]
      150723 18:59:05 [Note] InnoDB: Highest supported file format is Barracuda.
      [New Thread 0xdfbff430 (LWP 13183)]
      150723 18:59:06 [Note] InnoDB: 128 rollback segment(s) are active.
      [Thread 0xdfbff430 (LWP 13183) exited]
      [New Thread 0xdfbff430 (LWP 13184)]
      [New Thread 0xdefff430 (LWP 13185)]
      [New Thread 0xde7ff430 (LWP 13186)]
      [New Thread 0xddfff430 (LWP 13187)]
      [New Thread 0xdd7ff430 (LWP 13188)]
      150723 18:59:06 [Note] InnoDB: Waiting for purge to start
      [New Thread 0xdc7ff430 (LWP 13189)]
      [New Thread 0xdbfff430 (LWP 13190)]
      150723 18:59:06 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.24-72.2 started; log sequence number 1629494
      [New Thread 0xdb7ff430 (LWP 13191)]
      [New Thread 0xdafff430 (LWP 13192)]
      [New Thread 0xda7ff430 (LWP 13193)]
      150723 18:59:06 [Note] Plugin 'XTRADB_READ_VIEW' is disabled.
      150723 18:59:06 [Note] Plugin 'XTRADB_INTERNAL_HASH_TABLES' is disabled.
      150723 18:59:06 [Note] Plugin 'XTRADB_RSEG' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_TRX' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_CMPMEM' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_METRICS' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
      150723 18:59:06 [Note] Plugin 'INNODB_CHANGED_PAGES' is disabled.
      150723 18:59:06 [Note] Plugin 'FEDERATED' is disabled.
      150723 18:59:06 [Note] Plugin 'FEEDBACK' is disabled.
      150723 18:59:06 [Note] Plugin 'partition' is disabled.
      [New Thread 0xf7e87430 (LWP 13194)]
      [New Thread 0xf7e27430 (LWP 13195)]
       
      bt
      exit;
      ERROR: 1064  You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'bt
      exit;' at line 1
      150723 19:00:29 [ERROR] Aborting
       
      [Thread 0xf7e27430 (LWP 13195) exited]
      150723 19:00:29 [Note] InnoDB: FTS optimize thread exiting.
      150723 19:00:29 [Note] InnoDB: Starting shutdown...
      [Thread 0xda7ff430 (LWP 13193) exited]
      [Thread 0xde7ff430 (LWP 13186) exited]
      [Thread 0xdb7ff430 (LWP 13191) exited]
      [Thread 0xdefff430 (LWP 13185) exited]
      [Thread 0xdafff430 (LWP 13192) exited]
      [Thread 0xdfbff430 (LWP 13184) exited]
      [Thread 0xdd7ff430 (LWP 13188) exited]
      [Thread 0xdc7ff430 (LWP 13189) exited]
      [Thread 0xddfff430 (LWP 13187) exited]
      [Thread 0xe3bff430 (LWP 13176) exited]
      [Thread 0xe0bff430 (LWP 13182) exited]
      [Thread 0xe53ff430 (LWP 13173) exited]
      [Thread 0xe13ff430 (LWP 13181) exited]
      [Thread 0xe23ff430 (LWP 13179) exited]
      [Thread 0xe4bff430 (LWP 13174) exited]
      [Thread 0xe43ff430 (LWP 13175) exited]
      [Thread 0xe1bff430 (LWP 13180) exited]
      [Thread 0xe33ff430 (LWP 13177) exited]
      [Thread 0xe2bff430 (LWP 13178) exited]
      [Thread 0xdbfff430 (LWP 13190) exited]
       
      Program received signal SIGSEGV, Segmentation fault.
      0x109e8b8c in arena_purge ()
      (gdb) bt
      #0  0x109e8b8c in arena_purge ()
      #1  0x109e8f60 in arena_dalloc_bin_run ()
      #2  0x109ea7d8 in je_arena_dalloc_bin_locked ()
      #3  0x10a0566c in je_tcache_bin_flush_small ()
      #4  0x109df5f8 in free ()
      #5  0x10754780 in ut_free (ptr=ptr@entry=0xf77e58e0)
          at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/ut/ut0mem.cc:223
      #6  0x1065132c in os_event_free (event=0xf77e58e0)
          at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/os/os0sync.cc:540
      #7  0x106517f4 in os_sync_free ()
          at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/os/os0sync.cc:321
      #8  0x10713c04 in innobase_shutdown_for_mysql ()
          at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/srv/srv0start.cc:3227
      #9  0x105be570 in innobase_end (hton=<optimized out>, type=<optimized out>)
          at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/handler/ha_innodb.cc:3883
      #10 0x1034a458 in ha_finalize_handlerton (plugin=0xf745c368)
          at /home/otto/mariadb-10.0-10.0.20/sql/handler.cc:440
      #11 0x101a8e80 in plugin_deinitialize (plugin=0xf745c368, 
          ref_check=ref_check@entry=true)
          at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1176
      #12 0x101acf58 in reap_plugins ()
          at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1249
      #13 0x101ad96c in plugin_shutdown ()
          at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1895
      #14 0x100ef320 in clean_up (print_message=<optimized out>)
          at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:2029
      #15 0x100f6c54 in unireg_abort (exit_code=1)
          at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:1954
      #16 0x100fd178 in mysqld_main (argc=54, argv=0xf7409e30)
          at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:5510
      #17 0x100ee8d0 in main (argc=<optimized out>, argv=<optimized out>)
          at /home/otto/mariadb-10.0-10.0.20/sql/main.cc:25
      (gdb) x/10c 0xf77e58e0
      0xf77e58e0:	0 '\000'	0 '\000'	0 '\000'	0 '\000'	0 '\000'	0 '\000'	0 '\000'	0 '\000'
      0xf77e58e8:	0 '\000'	0 '\000'
      (gdb) thr apply all bt full
       
      Thread 23 (Thread 0xf7e87430 (LWP 13194)):
      #0  0x0ffd25f0 in do_sigwait () from /lib/powerpc-linux-gnu/libpthread.so.0
      No symbol table info available.
      #1  0x0ffd26a8 in sigwait () from /lib/powerpc-linux-gnu/libpthread.so.0
      No symbol table info available.
      #2  0x100f35cc in signal_hand (arg=<optimized out>)
          at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:3346
              error = <optimized out>
              set = {__val = {548869, 0 <repeats 31 times>}}
              sig = 0
              _db_stack_frame_ = {func = 0x10b94f78 "?func", file = 0x10b94f80 "?file", 
                level = 2147483649, prev = 0x0}
      #3  0x0ffc7c3c in start_thread () from /lib/powerpc-linux-gnu/libpthread.so.0
      No symbol table info available.
      #4  0x0fb56470 in clone () from /lib/powerpc-linux-gnu/libc.so.6
      No symbol table info available.
       
      Thread 1 (Thread 0xf7fe78d0 (LWP 13168)):
      #0  0x109e8b8c in arena_purge ()
      No symbol table info available.
      #1  0x109e8f60 in arena_dalloc_bin_run ()
      No symbol table info available.
      #2  0x109ea7d8 in je_arena_dalloc_bin_locked ()
      No symbol table info available.
      #3  0x10a0566c in je_tcache_bin_flush_small ()
      No symbol table info available.
      #4  0x109df5f8 in free ()
      No symbol table info available.
      #5  0x10754780 in ut_free (ptr=ptr@entry=0xf77e58e0)
          at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/ut/ut0mem.cc:223
              block = <optimized out>
      #6  0x1065132c in os_event_free (event=0xf77e58e0)
          at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/os/os0sync.cc:540
      No locals.
      #7  0x106517f4 in os_sync_free ()
          at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/os/os0sync.cc:321
              event = <optimized out>
              mutex = <optimized out>
      #8  0x10713c04 in innobase_shutdown_for_mysql ()
          at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/srv/srv0start.cc:3227
              i = <optimized out>
      #9  0x105be570 in innobase_end (hton=<optimized out>, type=<optimized out>)
          at /home/otto/mariadb-10.0-10.0.20/storage/xtradb/handler/ha_innodb.cc:3883
              err = 0
              _db_stack_frame_ = {func = 0x10ac98d8 "ha_finalize_handlerton", 
                file = 0x10ac9418 "/home/otto/mariadb-10.0-10.0.20/sql/handler.cc", 
                level = 2147483652, prev = 0xfffeea58}
              __PRETTY_FUNCTION__ = "int innobase_end(handlerton*, ha_panic_function)"
      #10 0x1034a458 in ha_finalize_handlerton (plugin=0xf745c368)
          at /home/otto/mariadb-10.0-10.0.20/sql/handler.cc:440
              hton = 0xf7416738
              _db_stack_frame_ = {func = 0x10a33bec "plugin_shutdown", 
                file = 0x10a3347c "/home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc", 
                level = 2147483651, prev = 0xfffeebf8}
              __PRETTY_FUNCTION__ = "int ha_finalize_handlerton(st_plugin_int*)"
      #11 0x101a8e80 in plugin_deinitialize (plugin=0xf745c368, 
          ref_check=ref_check@entry=true)
          at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1176
              __PRETTY_FUNCTION__ = "void plugin_deinitialize(st_plugin_int*, bool)"
      #12 0x101acf58 in reap_plugins ()
          at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1249
              count = <optimized out>
              idx = <optimized out>
              reap = 0xfffeeb00
              list = 0xfffeeaec
              __PRETTY_FUNCTION__ = "void reap_plugins()"
              plugin = <optimized out>
      #13 0x101ad96c in plugin_shutdown ()
          at /home/otto/mariadb-10.0-10.0.20/sql/sql_plugin.cc:1895
              i = <optimized out>
              count = 47
              dl = <optimized out>
              plugins = <optimized out>
              plugin = <optimized out>
              _db_stack_frame_ = {func = 0x10a1126c "unireg_abort", 
                file = 0x10a107c0 "/home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc", 
                level = 2147483650, prev = 0xfffeec88}
      #14 0x100ef320 in clean_up (print_message=<optimized out>)
          at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:2029
      No locals.
      #15 0x100f6c54 in unireg_abort (exit_code=1)
          at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:1954
              _db_stack_frame_ = {func = 0x10b94f78 "?func", file = 0x10b94f80 "?file", 
                level = 2147483649, prev = 0x0}
      #16 0x100fd178 in mysqld_main (argc=54, argv=0xf7409e30)
          at /home/otto/mariadb-10.0-10.0.20/sql/mysqld.cc:5510
              ho_error = <optimized out>
      #17 0x100ee8d0 in main (argc=<optimized out>, argv=<optimized out>)
          at /home/otto/mariadb-10.0-10.0.20/sql/main.cc:25
      No locals.

        Attachments

          Activity

            People

            Assignee:
            otto Otto Kekäläinen
            Reporter:
            svoj Sergey Vojtovich
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: