Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-16767

InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/rem/rem0rec.cc line 82

    XMLWordPrintable

Details

    Description

      Hi there,
      After a restart of one node in a galera cluster of 3 nodes (running inside a 1GB LAN) the node was no longer able to start with (see traceback below). During restart server was put under a load (mainly inserts and updates).
      ===
      Jul 16 15:12:24 node2 systemd[1]: Starting MariaDB 10.3.8 database server...
      Jul 16 15:12:45 node2 mysqld[14057]: WSREP: Failed to start mysqld for wsrep recovery: '2018-07-16 15:12:24 0 [Note] /usr/sbin/mysqld (mysqld 10.3.8-MariaDB-1:10.3.8+maria~xenial-log) starting as process 14230 ...
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: For Galera, using innodb_lock_schedule_algorithm=fcfs
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Using Linux native AIO
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Uses event mutexes
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Number of pools: 1
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Using generic crc32 instructions
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Completed initialization of buffer pool
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:25 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=17920587334
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:36 0 [Note] InnoDB: 6 transaction(s) which must be rolled back or cleaned up in total 4200 row operations to undo
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:36 0 [Note] InnoDB: Trx id counter is 123430
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:36 0 [Note] InnoDB: Starting final batch to recover 1808 pages from redo log.
      Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:36 0x7f83c4f03700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/rem/rem0rec.cc line 828
      Jul 16 15:12:45 node2 mysqld[14057]: InnoDB: We intentionally generate a memory trap.
      Jul 16 15:12:45 node2 mysqld[14057]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      Jul 16 15:12:45 node2 mysqld[14057]: InnoDB: If you get repeated assertion failures or crashes, even
      Jul 16 15:12:45 node2 mysqld[14057]: InnoDB: immediately after the mysqld startup, there may be
      Jul 16 15:12:45 node2 mysqld[14057]: InnoDB: corruption in the InnoDB tablespace. Please refer to
      Jul 16 15:12:45 node2 mysqld[14057]: InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/
      Jul 16 15:12:45 node2 mysqld[14057]: InnoDB: about forcing recovery.
      Jul 16 15:12:45 node2 mysqld[14057]: 180716 15:12:36 [ERROR] mysqld got signal 6 ;
      Jul 16 15:12:45 node2 mysqld[14057]: This could be because you hit a bug. It is also possible that this binary
      Jul 16 15:12:45 node2 mysqld[14057]: or one of the libraries it was linked against is corrupt, improperly built,
      Jul 16 15:12:45 node2 mysqld[14057]: or misconfigured. This error can also be caused by malfunctioning hardware.
      Jul 16 15:12:45 node2 mysqld[14057]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      Jul 16 15:12:45 node2 mysqld[14057]: We will try our best to scrape up some info that will hopefully help
      Jul 16 15:12:45 node2 mysqld[14057]: diagnose the problem, but since we have already crashed,
      Jul 16 15:12:45 node2 mysqld[14057]: something is definitely wrong and this may fail.
      Jul 16 15:12:45 node2 mysqld[14057]: Server version: 10.3.8-MariaDB-1:10.3.8+maria~xenial-log
      Jul 16 15:12:45 node2 mysqld[14057]: key_buffer_size=134217728
      Jul 16 15:12:45 node2 mysqld[14057]: read_buffer_size=2097152
      Jul 16 15:12:45 node2 mysqld[14057]: max_used_connections=0
      Jul 16 15:12:45 node2 mysqld[14057]: max_threads=1002
      Jul 16 15:12:45 node2 mysqld[14057]: thread_count=0
      Jul 16 15:12:45 node2 mysqld[14057]: It is possible that mysqld could use up to
      Jul 16 15:12:45 node2 mysqld[14057]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6309726 K bytes of memory
      Jul 16 15:12:45 node2 mysqld[14057]: Hope that's ok; if not, decrease some variables in the equation.
      Jul 16 15:12:45 node2 mysqld[14057]: Thread pointer: 0x0
      Jul 16 15:12:45 node2 mysqld[14057]: Attempting backtrace. You can use the following information to find out
      Jul 16 15:12:45 node2 mysqld[14057]: where mysqld died. If you see no messages after this, something went
      Jul 16 15:12:45 node2 mysqld[14057]: terribly wrong...
      Jul 16 15:12:45 node2 mysqld[14057]: stack_bottom = 0x0 thread_stack 0x49000
      Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56420ac5c32e]
      Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(handle_fatal_signal+0x347)[0x56420a707df7]
      Jul 16 15:12:45 node2 mysqld[14057]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f846fd02390]
      Jul 16 15:12:45 node2 mysqld[14057]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f846f2d1428]
      Jul 16 15:12:45 node2 mysqld[14057]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f846f2d302a]
      Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0x49f4e2)[0x56420a4544e2]
      Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0xa373a3)[0x56420a9ec3a3]
      Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0xa15a7e)[0x56420a9caa7e]
      Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0x9f52e5)[0x56420a9aa2e5]
      Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0x9f6bd9)[0x56420a9abbd9]
      Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0xb2e847)[0x56420aae3847]
      Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0xb99e69)[0x56420ab4ee69]
      Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0xaa5dd8)[0x56420aa5add8]
      Jul 16 15:12:45 node2 mysqld[14057]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f846fcf86ba]
      Jul 16 15:12:45 node2 mysqld[14057]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f846f3a341d]
      Jul 16 15:12:45 node2 mysqld[14057]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      Jul 16 15:12:45 node2 mysqld[14057]: information that should help you find out what is causing the crash.
      Jul 16 15:12:45 node2 mysqld[14057]: Aborted (core dumped)'
      Jul 16 15:12:45 node2 systemd[1]: mariadb.service: Control process exited, code=exited status=1
      Jul 16 15:12:45 node2 systemd[1]: Failed to start MariaDB 10.3.8 database server.
      Jul 16 15:12:45 node2 systemd[1]: mariadb.service: Unit entered failed state.
      Jul 16 15:12:45 node2 systemd[1]: mariadb.service: Failed with result 'exit-code'.
      ===

      Attachments

        Activity

          People

            jplindst Jan Lindström (Inactive)
            ivan.tyagov Ivan Tyagov
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.