Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14170

InnoDB: Failing assertion: sibling_mode == RW_NO_LATCH || btr_page_get_next(get_block->frame, mtr) == page_get_page_no(page)

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Duplicate
    • 10.1.25
    • 10.7.5, 10.6.9, 10.8.4, 10.9.2
    • Admin statements
    • None
    • Zimbra Release 8.8.3.GA.1872.UBUNTU16.64 UBUNTU16_64 FOSS edition.
      Hyper-v virtual, 4 cores, 12 GB RAM, iscsi partition for zimbra

    Description

      Hello. I migrating Mails from groupwise tn zimbre. Some 500 users and 700 GB of content. Everything went well until the content was over 500 GB. Then mysql crash. I do not want to run every time.
      I have an error:

      171027 11:52:18 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
      2017-10-27 11:52:19 139677805532992 [Note] /opt/zimbra/common/sbin/mysqld (mysqld 10.1.25-MariaDB) starting as process 17178 ...
      2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      2017-10-27 11:52:19 139677805532992 [Note] InnoDB: The InnoDB memory heap is disabled
      2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      2017-10-27 11:52:19 139677805532992 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
      2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Compressed tables use zlib 1.2.3
      2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Using Linux native AIO
      2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Using generic crc32 instructions
      2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Initializing buffer pool, size = 2.0G
      2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Completed initialization of buffer pool
      2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Highest supported file format is Barracuda.
      2017-10-27 11:52:19 139677805532992 [Note] InnoDB: The log sequence number 17195140108 in ibdata file do not match the log sequence number 17195140630 in the ib_logfiles!
      2017-10-27 11:52:20 139677805532992 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
      InnoDB: 1 transaction(s) which must be rolled back or cleaned up
      InnoDB: in total 3 row operations to undo
      InnoDB: Trx id counter is 14855936
      2017-10-27 11:52:24 139677805532992 [Note] InnoDB: 128 rollback segment(s) are active.
      2017-10-27 11:52:24 139675408623360 [Note] InnoDB: Starting in background the rollback of recovered transactions
      2017-10-27 11:52:24 139677805532992 [Note] InnoDB: Waiting for purge to start
      2017-10-27 11:52:24 7f08b71d6700  InnoDB: Rolling back trx with id 14853455, 3 rows to undo
      2017-10-27 11:52:24 7f08b71d6700  InnoDB: Assertion failure in thread 139675408623360 in file btr0cur.cc line 294
      InnoDB: Failing assertion: sibling_mode == RW_NO_LATCH || btr_page_get_next(get_block->frame, mtr) == page_get_page_no(page)
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      171027 11:52:24 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,.
      something is definitely wrong and this may fail.
       
      Server version: 10.1.25-MariaDB
      key_buffer_size=134217728
      read_buffer_size=1048576
      max_used_connections=0
      max_threads=112
      thread_count=0
      It is possible that mysqld could use up to.
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 362733 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x48400
      *** buffer overflow detected ***: /opt/zimbra/common/sbin/mysqld terminated
      ======= Backtrace: =========
      /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f0943db07e5]
      /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f0943e5211c]
      /lib/x86_64-linux-gnu/libc.so.6(+0x117120)[0x7f0943e50120]
      /lib/x86_64-linux-gnu/libc.so.6(+0x119067)[0x7f0943e52067]
      /opt/zimbra/common/sbin/mysqld(my_addr_resolve+0x48)[0x562fdccbdca8]
      /opt/zimbra/common/sbin/mysqld(my_print_stacktrace+0x1e2)[0x562fdcca7e12]
      /opt/zimbra/common/sbin/mysqld(handle_fatal_signal+0x2f5)[0x562fdc8817d5]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f094479f390]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f0943d6e428]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f0943d7002a]
      /opt/zimbra/common/sbin/mysqld(+0x84c6d7)[0x562fdcb396d7]
      /opt/zimbra/common/sbin/mysqld(+0x84d89a)[0x562fdcb3a89a]
      /opt/zimbra/common/sbin/mysqld(+0x7e8305)[0x562fdcad5305]
      /opt/zimbra/common/sbin/mysqld(+0x91059d)[0x562fdcbfd59d]
      /opt/zimbra/common/sbin/mysqld(+0x9117a4)[0x562fdcbfe7a4]
      /opt/zimbra/common/sbin/mysqld(+0x7f0a74)[0x562fdcadda74]
      /opt/zimbra/common/sbin/mysqld(+0x7adcfe)[0x562fdca9acfe]
      /opt/zimbra/common/sbin/mysqld(+0x81c80c)[0x562fdcb0980c]
      /opt/zimbra/common/sbin/mysqld(+0x81cf41)[0x562fdcb09f41]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f09447956ba]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f0943e403dd]
      ======= Memory map: ========
      562fdc2ed000-562fdd15b000 r-xp 00000000 08:11 7346587                    /opt/zimbra/common/sbin/mysqld
      562fdd35b000-562fdd402000 r--p 00e6e000 08:11 7346587                    /opt/zimbra/common/sbin/mysqld
      562fdd402000-562fdd4b2000 rw-p 00f15000 08:11 7346587                    /opt/zimbra/common/sbin/mysqld
      562fdd4b2000-562fdd52b000 rw-p 00000000 00:00 0.
      562fdf3cd000-562fef594000 rw-p 00000000 00:00 0                          [heap]
      7f08b39d0000-7f08b39d1000 ---p 00000000 00:00 0.
      7f08b39d1000-7f08b41d1000 rwxp 00000000 00:00 0.
      7f08b41d1000-7f08b41d2000 ---p 00000000 00:00 0.
      7f08b41d2000-7f08b49d2000 rwxp 00000000 00:00 0.
      7f08b49d2000-7f08b49d3000 ---p 00000000 00:00 0.
      7f08b49d3000-7f08b51d3000 rwxp 00000000 00:00 0.
      7f08b51d3000-7f08b51d4000 ---p 00000000 00:00 0.
      7f08b51d4000-7f08b59d4000 rwxp 00000000 00:00 0.
      7f08b59d4000-7f08b59d5000 ---p 00000000 00:00 0.
      7f08b59d5000-7f08b61d5000 rwxp 00000000 00:00 0.
      7f08b61d5000-7f08b61d6000 ---p 00000000 00:00 0.
      7f08b61d6000-7f08b69d6000 rwxp 00000000 00:00 0.
      7f08b69d6000-7f08b69d7000 ---p 00000000 00:00 0.
      7f08b69d7000-7f08b71d7000 rwxp 00000000 00:00 0.
      7f08b71d7000-7f08b71d8000 ---p 00000000 00:00 0.
      7f08b71d8000-7f08b79d8000 rwxp 00000000 00:00 0.
      7f08b79d8000-7f08b79d9000 ---p 00000000 00:00 0.
      7f08b79d9000-7f08b81d9000 rwxp 00000000 00:00 0.
      7f08b81d9000-7f08b81da000 ---p 00000000 00:00 0.
      7f08b81da000-7f08b89da000 rwxp 00000000 00:00 0.
      7f08b89db000-7f08b91db000 rwxp 00000000 00:00 0.
      7f08b91db000-7f08b91dc000 ---p 00000000 00:00 0.
      7f08b91dc000-7f08b99dc000 rwxp 00000000 00:00 0.
      7f08b99dc000-7f08b99dd000 ---p 00000000 00:00 0.
      7f08b99dd000-7f08ba1dd000 rwxp 00000000 00:00 0.
      7f08ba1dd000-7f08ba1de000 ---p 00000000 00:00 0.
      7f08ba1de000-7f08ba9de000 rwxp 00000000 00:00 0.
      7f08ba9de000-7f08ba9df000 ---p 00000000 00:00 0.
      7f08ba9df000-7f08bb1df000 rwxp 00000000 00:00 0.
      7f08bb1df000-7f08bb1e0000 ---p 00000000 00:00 0.
      7f08bb1e0000-7f08bb9e0000 rwxp 00000000 00:00 0.
      7f08bb9e0000-7f08bb9e1000 ---p 00000000 00:00 0.
      7f08bb9e1000-7f08bc1e1000 rwxp 00000000 00:00 0.
      7f08bc1e1000-7f08bc1e2000 ---p 00000000 00:00 0.
      7f08bc1e2000-7f08bc9e2000 rwxp 00000000 00:00 0.
      7f08bc9e2000-7f0942ae2000 rw-p 00000000 00:00 0.
      7f0942ae2000-7f0942ae3000 ---p 00000000 00:00 0.
      7f0942ae3000-7f09432e3000 rwxp 00000000 00:00 0.
      7f09432e3000-7f09432ee000 r-xp 00000000 08:01 4063795                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
      7f09432ee000-7f09434ed000 ---p 0000b000 08:01 4063795                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
      7f09434ed000-7f09434ee000 r--p 0000a000 08:01 4063795                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
      7f09434ee000-7f09434ef000 rw-p 0000b000 08:01 4063795                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
      7f09434ef000-7f09434f5000 rw-p 00000000 00:00 0.
      7f09434f5000-7f0943500000 r-xp 00000000 08:01 4063799                    /lib/x86_64-linux-gnu/libnss_nis-2.23.so
      7f0943500000-7f09436ff000 ---p 0000b000 08:01 4063799                    /lib/x86_64-linux-gnu/libnss_nis-2.23.so
      7f09436ff000-7f0943700000 r--p 0000a000 08:01 4063799                    /lib/x86_64-linux-gnu/libnss_nis-2.23.so
      7f0943700000-7f0943701000 rw-p 0000b000 08:01 4063799                    /lib/x86_64-linux-gnu/libnss_nis-2.23.so
      7f0943701000-7f0943717000 r-xp 00000000 08:01 4063789                    /lib/x86_64-linux-gnu/libnsl-2.23.so
      7f0943717000-7f0943916000 ---p 00016000 08:01 4063789                    /lib/x86_64-linux-gnu/libnsl-2.23.so
      7f0943916000-7f0943917000 r--p 00015000 08:01 4063789                    /lib/x86_64-linux-gnu/libnsl-2.23.so
      7f0943917000-7f0943918000 rw-p 00016000 08:01 4063789                    /lib/x86_64-linux-gnu/libnsl-2.23.so
      7f0943918000-7f094391a000 rw-p 00000000 00:00 0.
      7f094391a000-7f0943922000 r-xp 00000000 08:01 4063791                    /lib/x86_64-linux-gnu/libnss_compat-2.23.so
      7f0943922000-7f0943b21000 ---p 00008000 08:01 4063791                    /lib/x86_64-linux-gnu/libnss_compat-2.23.so
      7f0943b21000-7f0943b22000 r--p 00007000 08:01 4063791                    /lib/x86_64-linux-gnu/libnss_compat-2.23.so
      7f0943b22000-7f0943b23000 rw-p 00008000 08:01 4063791                    /lib/x86_64-linux-gnu/libnss_compat-2.23.so
      7f0943b23000-7f0943b39000 r-xp 00000000 08:01 4063757                    /lib/x86_64-linux-gnu/libgcc_s.so.1
      7f0943b39000-7f0943d38000 ---p 00016000 08:01 4063757                    /lib/x86_64-linux-gnu/libgcc_s.so.1
      7f0943d38000-7f0943d39000 rw-p 00015000 08:01 4063757                    /lib/x86_64-linux-gnu/libgcc_s.so.1
      7f0943d39000-7f0943ef9000 r-xp 00000000 08:01 4063731                    /lib/x86_64-linux-gnu/libc-2.23.so
      7f0943ef9000-7f09440f9000 ---p 001c0000 08:01 4063731                    /lib/x86_64-linux-gnu/libc-2.23.so
      7f09440f9000-7f09440fd000 r--p 001c0000 08:01 4063731                    /lib/x86_64-linux-gnu/libc-2.23.so
      7f09440fd000-7f09440ff000 rw-p 001c4000 08:01 4063731                    /lib/x86_64-linux-gnu/libc-2.23.so
      7f09440ff000-7f0944103000 rw-p 00000000 00:00 0.
      7f0944103000-7f094420b000 r-xp 00000000 08:01 4063772                    /lib/x86_64-linux-gnu/libm-2.23.so
      7f094420b000-7f094440a000 ---p 00108000 08:01 4063772                    /lib/x86_64-linux-gnu/libm-2.23.so
      7f094440a000-7f094440b000 r--p 00107000 08:01 4063772                    /lib/x86_64-linux-gnu/libm-2.23.so
      7f094440b000-7f094440c000 rw-p 00108000 08:01 4063772                    /lib/x86_64-linux-gnu/libm-2.23.so
      7f094440c000-7f094457e000 r-xp 00000000 08:01 1835470                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
      7f094457e000-7f094477e000 ---p 00172000 08:01 1835470                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
      7f094477e000-7f0944788000 r--p 00172000 08:01 1835470                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
      7f0944788000-7f094478a000 rw-p 0017c000 08:01 1835470                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
      7f094478a000-7f094478e000 rw-p 00000000 00:00 0.
      7f094478e000-7f09447a6000 r-xp 00000000 08:01 4063818                    /lib/x86_64-linux-gnu/libpthread-2.23.so
      7f09447a6000-7f09449a5000 ---p 00018000 08:01 4063818                    /lib/x86_64-linux-gnu/libpthread-2.23.so
      7f09449a5000-7f09449a6000 r--p 00017000 08:01 4063818                    /lib/x86_64-linux-gnu/libpthread-2.23.so
      7f09449a6000-7f09449a7000 rw-p 00018000 08:01 4063818                    /lib/x86_64-linux-gnu/libpthread-2.23.so
      7f09449a7000-7f09449ab000 rw-p 00000000 00:00 0.
      7f09449ab000-7f09449ae000 r-xp 00000000 08:01 4063745                    /lib/x86_64-linux-gnu/libdl-2.23.so
      7f09449ae000-7f0944bad000 ---p 00003000 08:01 4063745                    /lib/x86_64-linux-gnu/libdl-2.23.so
      7f0944bad000-7f0944bae000 r--p 00002000 08:01 4063745                    /lib/x86_64-linux-gnu/libdl-2.23.so
      7f0944bae000-7f0944baf000 rw-p 00003000 08:01 4063745                    /lib/x86_64-linux-gnu/libdl-2.23.so
      7f0944baf000-7f0944dfd000 r-xp 00000000 08:11 7340050                    /opt/zimbra/common/lib/libcrypto.so.1.0.0
      7f0944dfd000-7f0944ffc000 ---p 0024e000 08:11 7340050                    /opt/zimbra/common/lib/libcrypto.so.1.0.0
      7f0944ffc000-7f0945018000 r--p 0024d000 08:11 7340050                    /opt/zimbra/common/lib/libcrypto.so.1.0.0
      7f0945018000-7f0945024000 rw-p 00269000 08:11 7340050                    /opt/zimbra/common/lib/libcrypto.so.1.0.0
      7f0945024000-7f0945027000 rw-p 00000000 00:00 0.
      7f0945027000-7f094508f000 r-xp 00000000 08:11 7340051                    /opt/zimbra/common/lib/libssl.so.1.0.0
      7f094508f000-7f094528e000 ---p 00068000 08:11 7340051                    /opt/zimbra/common/lib/libssl.so.1.0.0
      7f094528e000-7f0945292000 r--p 00067000 08:11 7340051                    /opt/zimbra/common/lib/libssl.so.1.0.0
      7f0945292000-7f0945299000 rw-p 0006b000 08:11 7340051                    /opt/zimbra/common/lib/libssl.so.1.0.0
      7f0945299000-7f09452a2000 r-xp 00000000 08:01 4063739                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
      7f09452a2000-7f09454a1000 ---p 00009000 08:01 4063739                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
      7f09454a1000-7f09454a2000 r--p 00008000 08:01 4063739                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
      7f09454a2000-7f09454a3000 rw-p 00009000 08:01 4063739                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
      7f09454a3000-7f09454d1000 rw-p 00000000 00:00 0.
      7f09454d1000-7f094553f000 r-xp 00000000 08:01 4063811                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
      7f094553f000-7f094573f000 ---p 0006e000 08:01 4063811                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
      7f094573f000-7f0945740000 r--p 0006e000 08:01 4063811                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
      7f0945740000-7f0945741000 rw-p 0006f000 08:01 4063811                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
      7f0945741000-7f0945742000 r-xp 00000000 08:01 4063717                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
      7f0945742000-7f0945941000 ---p 00001000 08:01 4063717                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
      7f0945941000-7f0945942000 r--p 00000000 08:01 4063717                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
      7f0945942000-7f0945943000 rw-p 00001000 08:01 4063717                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
      7f0945943000-7f0945952000 r-xp 00000000 08:01 4063730                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
      7f0945952000-7f0945b51000 ---p 0000f000 08:01 4063730                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
      7f0945b51000-7f0945b52000 r--p 0000e000 08:01 4063730                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
      7f0945b52000-7f0945b53000 rw-p 0000f000 08:01 4063730                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
      7f0945b53000-7f0945b7a000 r-xp 00000000 08:11 7340353                    /opt/zimbra/common/lib/libtcmalloc_minimal.so.4.2.6
      7f0945b7a000-7f0945d79000 ---p 00027000 08:11 7340353                    /opt/zimbra/common/lib/libtcmalloc_minimal.so.4.2.6
      7f0945d79000-7f0945d7a000 r--p 00026000 08:11 7340353                    /opt/zimbra/common/lib/libtcmalloc_minimal.so.4.2.6
      7f0945d7a000-7f0945d7b000 rw-p 00027000 08:11 7340353                    /opt/zimbra/common/lib/libtcmalloc_minimal.so.4.2.6
      7f0945d7b000-7f0945da0000 rw-p 00000000 00:00 0.
      7f0945da0000-7f0945dc6000 r-xp 00000000 08:01 4063707                    /lib/x86_64-linux-gnu/ld-2.23.so
      7f0945f7d000-7f0945f82000 rw-s 00000000 00:0d 134350                     /[aio] (deleted)
      7f0945f82000-7f0945f87000 rw-s 00000000 00:0d 134349                     /[aio] (deleted)
      7f0945f87000-7f0945f8c000 rw-s 00000000 00:0d 134348                     /[aio] (deleted)
      7f0945f8c000-7f0945f91000 rw-s 00000000 00:0d 134347                     /[aio] (deleted)
      7f0945f91000-7f0945f96000 rw-s 00000000 00:0d 134346                     /[aio] (deleted)
      7f0945f96000-7f0945f9b000 rw-s 00000000 00:0d 134345                     /[aio] (deleted)
      7f0945f9b000-7f0945fa0000 rw-s 00000000 00:0d 134344                     /[aio] (deleted)
      7f0945fa0000-7f0945fa5000 rw-s 00000000 00:0d 134343                     /[aio] (deleted)
      7f0945fa5000-7f0945faa000 rw-s 00000000 00:0d 134342                     /[aio] (deleted)
      7f0945faa000-7f0945faf000 rw-s 00000000 00:0d 134341                     /[aio] (deleted)
      7f0945faf000-7f0945fb4000 rw-s 00000000 00:0d 134340                     /[aio] (deleted)
      7f0945fb4000-7f0945fbe000 rw-p 00000000 00:00 0.
      7f0945fbe000-7f0945fc3000 rw-s 00000000 00:0d 134339                     /[aio] (deleted)
      7f0945fc3000-7f0945fc5000 rw-p 00000000 00:00 0.
      7f0945fc5000-7f0945fc6000 r--p 00025000 08:01 4063707                    /lib/x86_64-linux-gnu/ld-2.23.so
      7f0945fc6000-7f0945fc7000 rw-p 00026000 08:01 4063707                    /lib/x86_64-linux-gnu/ld-2.23.so
      7fff8fa4e000-7fff8fa6c000 rwxp 00000000 00:00 0                          [stack]
      7fff8fa6c000-7fff8fa74000 rw-p 00000000 00:00 0.
      7fff8fbc0000-7fff8fbc2000 r--p 00000000 00:00 0                          [vvar]
      7fff8fbc2000-7fff8fbc4000 r-xp 00000000 00:00 0                          [vdso]
      ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
      171027 11:52:24 mysqld_safe mysqld from pid file /opt/zimbra/log/mysql.pid ended
      

      in mysql_error.log:

      2017-10-27 08:16:23 7fbe73281700  InnoDB: Assertion failure in thread 140455952520960 in file btr0cur.cc line 294
      InnoDB: Failing assertion: sibling_mode == RW_NO_LATCH || btr_page_get_next(get_block->frame, mtr) == page_get_page_no(page)
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      171027  8:16:23 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
       
      Server version: 10.1.25-MariaDB
      key_buffer_size=134217728
      read_buffer_size=1048576
      max_used_connections=0
      max_threads=112
      thread_count=0
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 362733 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x48400
      171027 08:16:23 mysqld_safe mysqld from pid file /opt/zimbra/log/mysql.pid ended
      

      my my.cnf:

      [mysqld]
       
      basedir        = /opt/zimbra/common
      datadir        = /opt/zimbra/db/data
      socket         = /opt/zimbra/data/tmp/mysql/mysql.sock
      pid-file       = /opt/zimbra/log/mysql.pid
      bind-address   = 127.0.0.1
      port           = 7306
      user           = zimbra
      tmpdir         = /opt/zimbra/data/tmp
       
      external-locking
      slow_query_log = 1
      slow_query_log_file = /opt/zimbra/log/myslow.log
       
      general_log_file = /opt/zimbra/log/mysql-mailboxd.log
       
      long_query_time  = 1
      log_queries_not_using_indexes
       
      thread_cache_size = 110
      max_connections   = 110
       
      # We do a lot of writes, query cache turns out to be not useful.
      query_cache_type = 0
       
      key_buffer_size=134217728
      read_buffer_size=1048576
      max_used_connections=0
      max_threads=112
      thread_count=0
       
       
      sort_buffer_size = 1048576
      #read_buffer_size = 1048576
       
      # (Num mailbox groups * Num tables in each group) + padding
      table_open_cache = 1200
       
      innodb_data_file_path          = ibdata1:10M:autoextend
      innodb_buffer_pool_size        = 2G
      innodb_log_file_size           = 524288000
      innodb_log_buffer_size         = 8388608
      #
      innodb_file_per_table=1
       
      # Value is: 200 + max_connections + 2 * table_open_cache
      innodb_open_files              = 2710
       
      innodb_max_dirty_pages_pct     = 30
      innodb_flush_method            = O_DIRECT
      innodb_flush_log_at_trx_commit = 0
      max_allowed_packet             = 16777216
       
      # sysctl vm.swappiness.
      vm.swappiness = 60
       
      # Skip reverse DNS lookup of clients
      skip-name-resolve
       
      [mysqld_safe]
       
      log-error      = /opt/zimbra/log/mysqld.log
      pid-file     = /opt/zimbra/log/mysql.pid
      

      Attachments

        Issue Links

          Activity

            You did have MariaDB or MySQL before the pasted error log was produced. It's not the first start of the database server. Possibly, it's just not the beginning of the log from MariaDB which was installed with Zimbra; if so, please attach the whole log. Thanks.

            elenst Elena Stepanova added a comment - You did have MariaDB or MySQL before the pasted error log was produced. It's not the first start of the database server. Possibly, it's just not the beginning of the log from MariaDB which was installed with Zimbra; if so, please attach the whole log. Thanks.
            pokrakos Pokraczny added a comment -

            Full log in attach

            pokrakos Pokraczny added a comment - Full log in attach

            Thank you.

            So, the problems started here:

            2017-10-11  9:41:24 140521443088192 [Note] /opt/zimbra/common/sbin/mysqld: ready for connections.
            Version: '10.1.25-MariaDB'  socket: '/opt/zimbra/data/tmp/mysql/mysql.sock'  port: 7306  Zimbra MariaDB binary distribution
            171019 08:27:48 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
            2017-10-19  8:27:49 140486165006144 [Note] /opt/zimbra/common/sbin/mysqld (mysqld 10.1.25-MariaDB) starting as process 3391 ...
            2017-10-19  8:27:49 140486165006144 [Note] InnoDB: Using mutexes to ref count buffer pool pages
            2017-10-19  8:27:49 140486165006144 [Note] InnoDB: The InnoDB memory heap is disabled
            2017-10-19  8:27:49 140486165006144 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
            2017-10-19  8:27:49 140486165006144 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
            2017-10-19  8:27:49 140486165006144 [Note] InnoDB: Compressed tables use zlib 1.2.3
            2017-10-19  8:27:49 140486165006144 [Note] InnoDB: Using Linux native AIO
            2017-10-19  8:27:49 140486165006144 [Note] InnoDB: Using generic crc32 instructions
            2017-10-19  8:27:49 140486165006144 [Note] InnoDB: Initializing buffer pool, size = 1.2G
            2017-10-19  8:27:49 140486165006144 [Note] InnoDB: Completed initialization of buffer pool
            2017-10-19  8:27:50 140486165006144 [Note] InnoDB: Highest supported file format is Barracuda.
            2017-10-19  8:27:50 140486165006144 [Note] InnoDB: Starting crash recovery from checkpoint LSN=11717466889
            2017-10-19  8:28:06 140486165006144 [Note] InnoDB: Processed 1529 .ibd/.isl files
            2017-10-19  8:28:06 140486165006144 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
            2017-10-19  8:28:07 140486165006144 [Note] InnoDB: Read redo log up to LSN=11717530624
            2017-10-19  8:28:08 140486165006144 [Note] InnoDB: Starting final batch to recover 102 pages from redo log
            2017-10-19 08:28:08 7fc57bf60740 InnoDB: Error: page 1 log sequence number 11717753894
            InnoDB: is in the future! Current system log sequence number 11717530202.
            InnoDB: Your database may be corrupt or you may have copied the InnoDB
            InnoDB: tablespace but not the InnoDB log files. See
            InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
            InnoDB: for more information.
            2017-10-19 08:28:08 7fc52a48a700 InnoDB: Error: page 0 log sequence number 11717784380
            InnoDB: is in the future! Current system log sequence number 11717530202.
            InnoDB: Your database may be corrupt or you may have copied the InnoDB
            InnoDB: tablespace but not the InnoDB log files. See
            InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
            InnoDB: for more information.
            

            On October 11th, the database was still started normally. Then, at some point between October 11th and 19th, it was killed (with SIGKILL, apparently), and after restart it was complaining

            2017-10-19 08:28:08 7fc57bf60740 InnoDB: Error: page 1 log sequence number 11717753894
            InnoDB: is in the future! Current system log sequence number 11717530202.
            InnoDB: Your database may be corrupt or you may have copied the InnoDB
            InnoDB: tablespace but not the InnoDB log files. See
            

            Is it possible that between October 11th and 19th somebody indeed copied or moved around some of InnoDB files?

            elenst Elena Stepanova added a comment - Thank you. So, the problems started here: 2017-10-11 9:41:24 140521443088192 [Note] /opt/zimbra/common/sbin/mysqld: ready for connections. Version: '10.1.25-MariaDB' socket: '/opt/zimbra/data/tmp/mysql/mysql.sock' port: 7306 Zimbra MariaDB binary distribution 171019 08:27:48 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data 2017-10-19 8:27:49 140486165006144 [Note] /opt/zimbra/common/sbin/mysqld (mysqld 10.1.25-MariaDB) starting as process 3391 ... 2017-10-19 8:27:49 140486165006144 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2017-10-19 8:27:49 140486165006144 [Note] InnoDB: The InnoDB memory heap is disabled 2017-10-19 8:27:49 140486165006144 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2017-10-19 8:27:49 140486165006144 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2017-10-19 8:27:49 140486165006144 [Note] InnoDB: Compressed tables use zlib 1.2.3 2017-10-19 8:27:49 140486165006144 [Note] InnoDB: Using Linux native AIO 2017-10-19 8:27:49 140486165006144 [Note] InnoDB: Using generic crc32 instructions 2017-10-19 8:27:49 140486165006144 [Note] InnoDB: Initializing buffer pool, size = 1.2G 2017-10-19 8:27:49 140486165006144 [Note] InnoDB: Completed initialization of buffer pool 2017-10-19 8:27:50 140486165006144 [Note] InnoDB: Highest supported file format is Barracuda. 2017-10-19 8:27:50 140486165006144 [Note] InnoDB: Starting crash recovery from checkpoint LSN=11717466889 2017-10-19 8:28:06 140486165006144 [Note] InnoDB: Processed 1529 .ibd/.isl files 2017-10-19 8:28:06 140486165006144 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer... 2017-10-19 8:28:07 140486165006144 [Note] InnoDB: Read redo log up to LSN=11717530624 2017-10-19 8:28:08 140486165006144 [Note] InnoDB: Starting final batch to recover 102 pages from redo log 2017-10-19 08:28:08 7fc57bf60740 InnoDB: Error: page 1 log sequence number 11717753894 InnoDB: is in the future! Current system log sequence number 11717530202. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: for more information. 2017-10-19 08:28:08 7fc52a48a700 InnoDB: Error: page 0 log sequence number 11717784380 InnoDB: is in the future! Current system log sequence number 11717530202. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: for more information. On October 11th, the database was still started normally. Then, at some point between October 11th and 19th, it was killed (with SIGKILL, apparently), and after restart it was complaining 2017-10-19 08:28:08 7fc57bf60740 InnoDB: Error: page 1 log sequence number 11717753894 InnoDB: is in the future! Current system log sequence number 11717530202. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See Is it possible that between October 11th and 19th somebody indeed copied or moved around some of InnoDB files?
            pokrakos Pokraczny added a comment -

            No one changed anything. The only process that put mails was imapsync, and on the mysql side it was the zimbra that put data in the database and in mailboxes. The database itself is over 7 GB + 500 GB in mail files

            pokrakos Pokraczny added a comment - No one changed anything. The only process that put mails was imapsync, and on the mysql side it was the zimbra that put data in the database and in mailboxes. The database itself is over 7 GB + 500 GB in mail files
            pokrakos Pokraczny added a comment -

            Ok. I found solution. I run mysql in recovery mode - innodb_force_recovery = 3 . Next I get recovery databases from https://wiki.zimbra.com/wiki/Mysql_Crash_Recovery

            Propably problem was in this error:

            ERROR 1062 (23000) at line 303: Duplicate entry '413-cd30f5gVl01dYhExLdamvJex7mw=' for key 'PRIMARY'

            pokrakos Pokraczny added a comment - Ok. I found solution. I run mysql in recovery mode - innodb_force_recovery = 3 . Next I get recovery databases from https://wiki.zimbra.com/wiki/Mysql_Crash_Recovery Propably problem was in this error: ERROR 1062 (23000) at line 303: Duplicate entry '413-cd30f5gVl01dYhExLdamvJex7mw=' for key 'PRIMARY'

            People

              Unassigned Unassigned
              pokrakos Pokraczny
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.