[MDEV-14170] InnoDB: Failing assertion: sibling_mode == RW_NO_LATCH || btr_page_get_next(get_block->frame, mtr) == page_get_page_no(page) Created: 2017-10-27  Updated: 2022-06-07  Resolved: 2022-06-07

Status: Closed
Project: MariaDB Server
Component/s: Admin statements
Affects Version/s: 10.1.25
Fix Version/s: 10.6.9, 10.7.5, 10.8.4, 10.9.2

Type: Bug Priority: Major
Reporter: Pokraczny Assignee: Unassigned
Resolution: Duplicate Votes: 1
Labels: None
Environment:

Zimbra Release 8.8.3.GA.1872.UBUNTU16.64 UBUNTU16_64 FOSS edition.
Hyper-v virtual, 4 cores, 12 GB RAM, iscsi partition for zimbra

Attachments: Text File mysql_error.log    
Issue Links:
Duplicate
duplicates MDEV-13542 Crashing on a corrupted page is unhel... Closed
Relates
relates to MDEV-19711 Assertion failure during IMPORT TABLE... Closed

 Description   

Hello. I migrating Mails from groupwise tn zimbre. Some 500 users and 700 GB of content. Everything went well until the content was over 500 GB. Then mysql crash. I do not want to run every time.
I have an error:

171027 11:52:18 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
 
2017-10-27 11:52:19 139677805532992 [Note] /opt/zimbra/common/sbin/mysqld (mysqld 10.1.25-MariaDB) starting as process 17178 ...
 
2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Using mutexes to ref count buffer pool pages
 
2017-10-27 11:52:19 139677805532992 [Note] InnoDB: The InnoDB memory heap is disabled
 
2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
2017-10-27 11:52:19 139677805532992 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
 
2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Compressed tables use zlib 1.2.3
 
2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Using Linux native AIO
 
2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Using generic crc32 instructions
 
2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Initializing buffer pool, size = 2.0G
 
2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Completed initialization of buffer pool
 
2017-10-27 11:52:19 139677805532992 [Note] InnoDB: Highest supported file format is Barracuda.
 
2017-10-27 11:52:19 139677805532992 [Note] InnoDB: The log sequence number 17195140108 in ibdata file do not match the log sequence number 17195140630 in the ib_logfiles!
 
2017-10-27 11:52:20 139677805532992 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
 
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
 
InnoDB: in total 3 row operations to undo
 
InnoDB: Trx id counter is 14855936
 
2017-10-27 11:52:24 139677805532992 [Note] InnoDB: 128 rollback segment(s) are active.
 
2017-10-27 11:52:24 139675408623360 [Note] InnoDB: Starting in background the rollback of recovered transactions
 
2017-10-27 11:52:24 139677805532992 [Note] InnoDB: Waiting for purge to start
 
2017-10-27 11:52:24 7f08b71d6700  InnoDB: Rolling back trx with id 14853455, 3 rows to undo
 
2017-10-27 11:52:24 7f08b71d6700  InnoDB: Assertion failure in thread 139675408623360 in file btr0cur.cc line 294
 
InnoDB: Failing assertion: sibling_mode == RW_NO_LATCH || btr_page_get_next(get_block->frame, mtr) == page_get_page_no(page)
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
 
InnoDB: about forcing recovery.
 
171027 11:52:24 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,.
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.25-MariaDB
 
key_buffer_size=134217728
 
read_buffer_size=1048576
 
max_used_connections=0
 
max_threads=112
 
thread_count=0
 
It is possible that mysqld could use up to.
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 362733 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x48400
 
*** buffer overflow detected ***: /opt/zimbra/common/sbin/mysqld terminated
 
======= Backtrace: =========
 
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f0943db07e5]
 
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f0943e5211c]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x117120)[0x7f0943e50120]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x119067)[0x7f0943e52067]
 
/opt/zimbra/common/sbin/mysqld(my_addr_resolve+0x48)[0x562fdccbdca8]
 
/opt/zimbra/common/sbin/mysqld(my_print_stacktrace+0x1e2)[0x562fdcca7e12]
 
/opt/zimbra/common/sbin/mysqld(handle_fatal_signal+0x2f5)[0x562fdc8817d5]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f094479f390]
 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f0943d6e428]
 
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f0943d7002a]
 
/opt/zimbra/common/sbin/mysqld(+0x84c6d7)[0x562fdcb396d7]
 
/opt/zimbra/common/sbin/mysqld(+0x84d89a)[0x562fdcb3a89a]
 
/opt/zimbra/common/sbin/mysqld(+0x7e8305)[0x562fdcad5305]
 
/opt/zimbra/common/sbin/mysqld(+0x91059d)[0x562fdcbfd59d]
 
/opt/zimbra/common/sbin/mysqld(+0x9117a4)[0x562fdcbfe7a4]
 
/opt/zimbra/common/sbin/mysqld(+0x7f0a74)[0x562fdcadda74]
 
/opt/zimbra/common/sbin/mysqld(+0x7adcfe)[0x562fdca9acfe]
 
/opt/zimbra/common/sbin/mysqld(+0x81c80c)[0x562fdcb0980c]
 
/opt/zimbra/common/sbin/mysqld(+0x81cf41)[0x562fdcb09f41]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f09447956ba]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f0943e403dd]
 
======= Memory map: ========
 
562fdc2ed000-562fdd15b000 r-xp 00000000 08:11 7346587                    /opt/zimbra/common/sbin/mysqld
 
562fdd35b000-562fdd402000 r--p 00e6e000 08:11 7346587                    /opt/zimbra/common/sbin/mysqld
 
562fdd402000-562fdd4b2000 rw-p 00f15000 08:11 7346587                    /opt/zimbra/common/sbin/mysqld
 
562fdd4b2000-562fdd52b000 rw-p 00000000 00:00 0.
 
562fdf3cd000-562fef594000 rw-p 00000000 00:00 0                          [heap]
 
7f08b39d0000-7f08b39d1000 ---p 00000000 00:00 0.
 
7f08b39d1000-7f08b41d1000 rwxp 00000000 00:00 0.
 
7f08b41d1000-7f08b41d2000 ---p 00000000 00:00 0.
 
7f08b41d2000-7f08b49d2000 rwxp 00000000 00:00 0.
 
7f08b49d2000-7f08b49d3000 ---p 00000000 00:00 0.
 
7f08b49d3000-7f08b51d3000 rwxp 00000000 00:00 0.
 
7f08b51d3000-7f08b51d4000 ---p 00000000 00:00 0.
 
7f08b51d4000-7f08b59d4000 rwxp 00000000 00:00 0.
 
7f08b59d4000-7f08b59d5000 ---p 00000000 00:00 0.
 
7f08b59d5000-7f08b61d5000 rwxp 00000000 00:00 0.
 
7f08b61d5000-7f08b61d6000 ---p 00000000 00:00 0.
 
7f08b61d6000-7f08b69d6000 rwxp 00000000 00:00 0.
 
7f08b69d6000-7f08b69d7000 ---p 00000000 00:00 0.
 
7f08b69d7000-7f08b71d7000 rwxp 00000000 00:00 0.
 
7f08b71d7000-7f08b71d8000 ---p 00000000 00:00 0.
 
7f08b71d8000-7f08b79d8000 rwxp 00000000 00:00 0.
 
7f08b79d8000-7f08b79d9000 ---p 00000000 00:00 0.
 
7f08b79d9000-7f08b81d9000 rwxp 00000000 00:00 0.
 
7f08b81d9000-7f08b81da000 ---p 00000000 00:00 0.
 
7f08b81da000-7f08b89da000 rwxp 00000000 00:00 0.
 
7f08b89db000-7f08b91db000 rwxp 00000000 00:00 0.
 
7f08b91db000-7f08b91dc000 ---p 00000000 00:00 0.
 
7f08b91dc000-7f08b99dc000 rwxp 00000000 00:00 0.
 
7f08b99dc000-7f08b99dd000 ---p 00000000 00:00 0.
 
7f08b99dd000-7f08ba1dd000 rwxp 00000000 00:00 0.
 
7f08ba1dd000-7f08ba1de000 ---p 00000000 00:00 0.
 
7f08ba1de000-7f08ba9de000 rwxp 00000000 00:00 0.
 
7f08ba9de000-7f08ba9df000 ---p 00000000 00:00 0.
 
7f08ba9df000-7f08bb1df000 rwxp 00000000 00:00 0.
 
7f08bb1df000-7f08bb1e0000 ---p 00000000 00:00 0.
 
7f08bb1e0000-7f08bb9e0000 rwxp 00000000 00:00 0.
 
7f08bb9e0000-7f08bb9e1000 ---p 00000000 00:00 0.
 
7f08bb9e1000-7f08bc1e1000 rwxp 00000000 00:00 0.
 
7f08bc1e1000-7f08bc1e2000 ---p 00000000 00:00 0.
 
7f08bc1e2000-7f08bc9e2000 rwxp 00000000 00:00 0.
 
7f08bc9e2000-7f0942ae2000 rw-p 00000000 00:00 0.
 
7f0942ae2000-7f0942ae3000 ---p 00000000 00:00 0.
 
7f0942ae3000-7f09432e3000 rwxp 00000000 00:00 0.
 
7f09432e3000-7f09432ee000 r-xp 00000000 08:01 4063795                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
 
7f09432ee000-7f09434ed000 ---p 0000b000 08:01 4063795                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
 
7f09434ed000-7f09434ee000 r--p 0000a000 08:01 4063795                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
 
7f09434ee000-7f09434ef000 rw-p 0000b000 08:01 4063795                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
 
7f09434ef000-7f09434f5000 rw-p 00000000 00:00 0.
 
7f09434f5000-7f0943500000 r-xp 00000000 08:01 4063799                    /lib/x86_64-linux-gnu/libnss_nis-2.23.so
 
7f0943500000-7f09436ff000 ---p 0000b000 08:01 4063799                    /lib/x86_64-linux-gnu/libnss_nis-2.23.so
 
7f09436ff000-7f0943700000 r--p 0000a000 08:01 4063799                    /lib/x86_64-linux-gnu/libnss_nis-2.23.so
 
7f0943700000-7f0943701000 rw-p 0000b000 08:01 4063799                    /lib/x86_64-linux-gnu/libnss_nis-2.23.so
 
7f0943701000-7f0943717000 r-xp 00000000 08:01 4063789                    /lib/x86_64-linux-gnu/libnsl-2.23.so
 
7f0943717000-7f0943916000 ---p 00016000 08:01 4063789                    /lib/x86_64-linux-gnu/libnsl-2.23.so
 
7f0943916000-7f0943917000 r--p 00015000 08:01 4063789                    /lib/x86_64-linux-gnu/libnsl-2.23.so
 
7f0943917000-7f0943918000 rw-p 00016000 08:01 4063789                    /lib/x86_64-linux-gnu/libnsl-2.23.so
 
7f0943918000-7f094391a000 rw-p 00000000 00:00 0.
 
7f094391a000-7f0943922000 r-xp 00000000 08:01 4063791                    /lib/x86_64-linux-gnu/libnss_compat-2.23.so
 
7f0943922000-7f0943b21000 ---p 00008000 08:01 4063791                    /lib/x86_64-linux-gnu/libnss_compat-2.23.so
 
7f0943b21000-7f0943b22000 r--p 00007000 08:01 4063791                    /lib/x86_64-linux-gnu/libnss_compat-2.23.so
 
7f0943b22000-7f0943b23000 rw-p 00008000 08:01 4063791                    /lib/x86_64-linux-gnu/libnss_compat-2.23.so
 
7f0943b23000-7f0943b39000 r-xp 00000000 08:01 4063757                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7f0943b39000-7f0943d38000 ---p 00016000 08:01 4063757                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7f0943d38000-7f0943d39000 rw-p 00015000 08:01 4063757                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 
7f0943d39000-7f0943ef9000 r-xp 00000000 08:01 4063731                    /lib/x86_64-linux-gnu/libc-2.23.so
 
7f0943ef9000-7f09440f9000 ---p 001c0000 08:01 4063731                    /lib/x86_64-linux-gnu/libc-2.23.so
 
7f09440f9000-7f09440fd000 r--p 001c0000 08:01 4063731                    /lib/x86_64-linux-gnu/libc-2.23.so
 
7f09440fd000-7f09440ff000 rw-p 001c4000 08:01 4063731                    /lib/x86_64-linux-gnu/libc-2.23.so
 
7f09440ff000-7f0944103000 rw-p 00000000 00:00 0.
 
7f0944103000-7f094420b000 r-xp 00000000 08:01 4063772                    /lib/x86_64-linux-gnu/libm-2.23.so
 
7f094420b000-7f094440a000 ---p 00108000 08:01 4063772                    /lib/x86_64-linux-gnu/libm-2.23.so
 
7f094440a000-7f094440b000 r--p 00107000 08:01 4063772                    /lib/x86_64-linux-gnu/libm-2.23.so
 
7f094440b000-7f094440c000 rw-p 00108000 08:01 4063772                    /lib/x86_64-linux-gnu/libm-2.23.so
 
7f094440c000-7f094457e000 r-xp 00000000 08:01 1835470                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
 
7f094457e000-7f094477e000 ---p 00172000 08:01 1835470                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
 
7f094477e000-7f0944788000 r--p 00172000 08:01 1835470                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
 
7f0944788000-7f094478a000 rw-p 0017c000 08:01 1835470                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
 
7f094478a000-7f094478e000 rw-p 00000000 00:00 0.
 
7f094478e000-7f09447a6000 r-xp 00000000 08:01 4063818                    /lib/x86_64-linux-gnu/libpthread-2.23.so
 
7f09447a6000-7f09449a5000 ---p 00018000 08:01 4063818                    /lib/x86_64-linux-gnu/libpthread-2.23.so
 
7f09449a5000-7f09449a6000 r--p 00017000 08:01 4063818                    /lib/x86_64-linux-gnu/libpthread-2.23.so
 
7f09449a6000-7f09449a7000 rw-p 00018000 08:01 4063818                    /lib/x86_64-linux-gnu/libpthread-2.23.so
 
7f09449a7000-7f09449ab000 rw-p 00000000 00:00 0.
 
7f09449ab000-7f09449ae000 r-xp 00000000 08:01 4063745                    /lib/x86_64-linux-gnu/libdl-2.23.so
 
7f09449ae000-7f0944bad000 ---p 00003000 08:01 4063745                    /lib/x86_64-linux-gnu/libdl-2.23.so
 
7f0944bad000-7f0944bae000 r--p 00002000 08:01 4063745                    /lib/x86_64-linux-gnu/libdl-2.23.so
 
7f0944bae000-7f0944baf000 rw-p 00003000 08:01 4063745                    /lib/x86_64-linux-gnu/libdl-2.23.so
 
7f0944baf000-7f0944dfd000 r-xp 00000000 08:11 7340050                    /opt/zimbra/common/lib/libcrypto.so.1.0.0
 
7f0944dfd000-7f0944ffc000 ---p 0024e000 08:11 7340050                    /opt/zimbra/common/lib/libcrypto.so.1.0.0
 
7f0944ffc000-7f0945018000 r--p 0024d000 08:11 7340050                    /opt/zimbra/common/lib/libcrypto.so.1.0.0
 
7f0945018000-7f0945024000 rw-p 00269000 08:11 7340050                    /opt/zimbra/common/lib/libcrypto.so.1.0.0
 
7f0945024000-7f0945027000 rw-p 00000000 00:00 0.
 
7f0945027000-7f094508f000 r-xp 00000000 08:11 7340051                    /opt/zimbra/common/lib/libssl.so.1.0.0
 
7f094508f000-7f094528e000 ---p 00068000 08:11 7340051                    /opt/zimbra/common/lib/libssl.so.1.0.0
 
7f094528e000-7f0945292000 r--p 00067000 08:11 7340051                    /opt/zimbra/common/lib/libssl.so.1.0.0
 
7f0945292000-7f0945299000 rw-p 0006b000 08:11 7340051                    /opt/zimbra/common/lib/libssl.so.1.0.0
 
7f0945299000-7f09452a2000 r-xp 00000000 08:01 4063739                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
 
7f09452a2000-7f09454a1000 ---p 00009000 08:01 4063739                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
 
7f09454a1000-7f09454a2000 r--p 00008000 08:01 4063739                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
 
7f09454a2000-7f09454a3000 rw-p 00009000 08:01 4063739                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
 
7f09454a3000-7f09454d1000 rw-p 00000000 00:00 0.
 
7f09454d1000-7f094553f000 r-xp 00000000 08:01 4063811                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
 
7f094553f000-7f094573f000 ---p 0006e000 08:01 4063811                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
 
7f094573f000-7f0945740000 r--p 0006e000 08:01 4063811                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
 
7f0945740000-7f0945741000 rw-p 0006f000 08:01 4063811                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
 
7f0945741000-7f0945742000 r-xp 00000000 08:01 4063717                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7f0945742000-7f0945941000 ---p 00001000 08:01 4063717                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7f0945941000-7f0945942000 r--p 00000000 08:01 4063717                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7f0945942000-7f0945943000 rw-p 00001000 08:01 4063717                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
 
7f0945943000-7f0945952000 r-xp 00000000 08:01 4063730                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
 
7f0945952000-7f0945b51000 ---p 0000f000 08:01 4063730                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
 
7f0945b51000-7f0945b52000 r--p 0000e000 08:01 4063730                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
 
7f0945b52000-7f0945b53000 rw-p 0000f000 08:01 4063730                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
 
7f0945b53000-7f0945b7a000 r-xp 00000000 08:11 7340353                    /opt/zimbra/common/lib/libtcmalloc_minimal.so.4.2.6
 
7f0945b7a000-7f0945d79000 ---p 00027000 08:11 7340353                    /opt/zimbra/common/lib/libtcmalloc_minimal.so.4.2.6
 
7f0945d79000-7f0945d7a000 r--p 00026000 08:11 7340353                    /opt/zimbra/common/lib/libtcmalloc_minimal.so.4.2.6
 
7f0945d7a000-7f0945d7b000 rw-p 00027000 08:11 7340353                    /opt/zimbra/common/lib/libtcmalloc_minimal.so.4.2.6
 
7f0945d7b000-7f0945da0000 rw-p 00000000 00:00 0.
 
7f0945da0000-7f0945dc6000 r-xp 00000000 08:01 4063707                    /lib/x86_64-linux-gnu/ld-2.23.so
 
7f0945f7d000-7f0945f82000 rw-s 00000000 00:0d 134350                     /[aio] (deleted)
 
7f0945f82000-7f0945f87000 rw-s 00000000 00:0d 134349                     /[aio] (deleted)
 
7f0945f87000-7f0945f8c000 rw-s 00000000 00:0d 134348                     /[aio] (deleted)
 
7f0945f8c000-7f0945f91000 rw-s 00000000 00:0d 134347                     /[aio] (deleted)
 
7f0945f91000-7f0945f96000 rw-s 00000000 00:0d 134346                     /[aio] (deleted)
 
7f0945f96000-7f0945f9b000 rw-s 00000000 00:0d 134345                     /[aio] (deleted)
 
7f0945f9b000-7f0945fa0000 rw-s 00000000 00:0d 134344                     /[aio] (deleted)
 
7f0945fa0000-7f0945fa5000 rw-s 00000000 00:0d 134343                     /[aio] (deleted)
 
7f0945fa5000-7f0945faa000 rw-s 00000000 00:0d 134342                     /[aio] (deleted)
 
7f0945faa000-7f0945faf000 rw-s 00000000 00:0d 134341                     /[aio] (deleted)
 
7f0945faf000-7f0945fb4000 rw-s 00000000 00:0d 134340                     /[aio] (deleted)
 
7f0945fb4000-7f0945fbe000 rw-p 00000000 00:00 0.
 
7f0945fbe000-7f0945fc3000 rw-s 00000000 00:0d 134339                     /[aio] (deleted)
 
7f0945fc3000-7f0945fc5000 rw-p 00000000 00:00 0.
 
7f0945fc5000-7f0945fc6000 r--p 00025000 08:01 4063707                    /lib/x86_64-linux-gnu/ld-2.23.so
 
7f0945fc6000-7f0945fc7000 rw-p 00026000 08:01 4063707                    /lib/x86_64-linux-gnu/ld-2.23.so
 
7fff8fa4e000-7fff8fa6c000 rwxp 00000000 00:00 0                          [stack]
 
7fff8fa6c000-7fff8fa74000 rw-p 00000000 00:00 0.
 
7fff8fbc0000-7fff8fbc2000 r--p 00000000 00:00 0                          [vvar]
 
7fff8fbc2000-7fff8fbc4000 r-xp 00000000 00:00 0                          [vdso]
 
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 
171027 11:52:24 mysqld_safe mysqld from pid file /opt/zimbra/log/mysql.pid ended

in mysql_error.log:

2017-10-27 08:16:23 7fbe73281700  InnoDB: Assertion failure in thread 140455952520960 in file btr0cur.cc line 294
 
InnoDB: Failing assertion: sibling_mode == RW_NO_LATCH || btr_page_get_next(get_block->frame, mtr) == page_get_page_no(page)
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
 
InnoDB: about forcing recovery.
 
171027  8:16:23 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.25-MariaDB
 
key_buffer_size=134217728
 
read_buffer_size=1048576
 
max_used_connections=0
 
max_threads=112
 
thread_count=0
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 362733 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x48400
 
171027 08:16:23 mysqld_safe mysqld from pid file /opt/zimbra/log/mysql.pid ended

my my.cnf:

[mysqld]
 
 
 
basedir        = /opt/zimbra/common
 
datadir        = /opt/zimbra/db/data
 
socket         = /opt/zimbra/data/tmp/mysql/mysql.sock
 
pid-file       = /opt/zimbra/log/mysql.pid
 
bind-address   = 127.0.0.1
 
port           = 7306
 
user           = zimbra
 
tmpdir         = /opt/zimbra/data/tmp
 
 
 
external-locking
 
slow_query_log = 1
 
slow_query_log_file = /opt/zimbra/log/myslow.log
 
 
 
general_log_file = /opt/zimbra/log/mysql-mailboxd.log
 
 
 
long_query_time  = 1
 
log_queries_not_using_indexes
 
 
 
thread_cache_size = 110
 
max_connections   = 110
 
 
 
# We do a lot of writes, query cache turns out to be not useful.
 
query_cache_type = 0
 
 
 
key_buffer_size=134217728
 
read_buffer_size=1048576
 
max_used_connections=0
 
max_threads=112
 
thread_count=0
 
 
 
 
 
sort_buffer_size = 1048576
 
#read_buffer_size = 1048576
 
 
 
# (Num mailbox groups * Num tables in each group) + padding
 
table_open_cache = 1200
 
 
 
innodb_data_file_path          = ibdata1:10M:autoextend
 
innodb_buffer_pool_size        = 2G
 
innodb_log_file_size           = 524288000
 
innodb_log_buffer_size         = 8388608
 
#
 
innodb_file_per_table=1
 
 
 
# Value is: 200 + max_connections + 2 * table_open_cache
 
innodb_open_files              = 2710
 
 
 
innodb_max_dirty_pages_pct     = 30
 
innodb_flush_method            = O_DIRECT
 
innodb_flush_log_at_trx_commit = 0
 
max_allowed_packet             = 16777216
 
 
 
# sysctl vm.swappiness.
 
vm.swappiness = 60
 
 
 
# Skip reverse DNS lookup of clients
 
skip-name-resolve
 
 
 
[mysqld_safe]
 
 
 
log-error      = /opt/zimbra/log/mysqld.log
 
pid-file     = /opt/zimbra/log/mysql.pid



 Comments   
Comment by Elena Stepanova [ 2017-10-30 ]

pokrakos,

Do you know which version of MySQL/MariaDB you had before the migration? Maybe it is still in the error logs?

Does Zimbra offer a newer version of MariaDB than 10.1.25? There have been numerous fixes around the area in latest 10.1 releases, and while we didn't have this particular assertion failure, possibly it was fixed on the way.

Comment by Pokraczny [ 2017-11-06 ]

This is the latest (fresh) version zimbra. I want do backup groupwise mail on zimbra

Comment by Elena Stepanova [ 2017-11-06 ]

The first question still stands – do you know which version of MySQL/MariaDB you had before the migration? Maybe it is still in the error logs?

Comment by Pokraczny [ 2017-11-06 ]

I did not have. I have a Groupwise server. I installed a new Zimbra, which migrates with groupwise. So I installed the zimbra with the latest available packages.
MariaDB/MySql was included

Comment by Elena Stepanova [ 2017-11-06 ]

You did have MariaDB or MySQL before the pasted error log was produced. It's not the first start of the database server. Possibly, it's just not the beginning of the log from MariaDB which was installed with Zimbra; if so, please attach the whole log. Thanks.

Comment by Pokraczny [ 2017-11-07 ]

Full log in attach

Comment by Elena Stepanova [ 2017-11-11 ]

Thank you.

So, the problems started here:

2017-10-11  9:41:24 140521443088192 [Note] /opt/zimbra/common/sbin/mysqld: ready for connections.
 
Version: '10.1.25-MariaDB'  socket: '/opt/zimbra/data/tmp/mysql/mysql.sock'  port: 7306  Zimbra MariaDB binary distribution
 
171019 08:27:48 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
 
2017-10-19  8:27:49 140486165006144 [Note] /opt/zimbra/common/sbin/mysqld (mysqld 10.1.25-MariaDB) starting as process 3391 ...
 
2017-10-19  8:27:49 140486165006144 [Note] InnoDB: Using mutexes to ref count buffer pool pages
 
2017-10-19  8:27:49 140486165006144 [Note] InnoDB: The InnoDB memory heap is disabled
 
2017-10-19  8:27:49 140486165006144 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
2017-10-19  8:27:49 140486165006144 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
 
2017-10-19  8:27:49 140486165006144 [Note] InnoDB: Compressed tables use zlib 1.2.3
 
2017-10-19  8:27:49 140486165006144 [Note] InnoDB: Using Linux native AIO
 
2017-10-19  8:27:49 140486165006144 [Note] InnoDB: Using generic crc32 instructions
 
2017-10-19  8:27:49 140486165006144 [Note] InnoDB: Initializing buffer pool, size = 1.2G
 
2017-10-19  8:27:49 140486165006144 [Note] InnoDB: Completed initialization of buffer pool
 
2017-10-19  8:27:50 140486165006144 [Note] InnoDB: Highest supported file format is Barracuda.
 
2017-10-19  8:27:50 140486165006144 [Note] InnoDB: Starting crash recovery from checkpoint LSN=11717466889
 
2017-10-19  8:28:06 140486165006144 [Note] InnoDB: Processed 1529 .ibd/.isl files
 
2017-10-19  8:28:06 140486165006144 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
 
2017-10-19  8:28:07 140486165006144 [Note] InnoDB: Read redo log up to LSN=11717530624
 
2017-10-19  8:28:08 140486165006144 [Note] InnoDB: Starting final batch to recover 102 pages from redo log
 
2017-10-19 08:28:08 7fc57bf60740 InnoDB: Error: page 1 log sequence number 11717753894
 
InnoDB: is in the future! Current system log sequence number 11717530202.
 
InnoDB: Your database may be corrupt or you may have copied the InnoDB
 
InnoDB: tablespace but not the InnoDB log files. See
 
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
 
InnoDB: for more information.
 
2017-10-19 08:28:08 7fc52a48a700 InnoDB: Error: page 0 log sequence number 11717784380
 
InnoDB: is in the future! Current system log sequence number 11717530202.
 
InnoDB: Your database may be corrupt or you may have copied the InnoDB
 
InnoDB: tablespace but not the InnoDB log files. See
 
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
 
InnoDB: for more information.

On October 11th, the database was still started normally. Then, at some point between October 11th and 19th, it was killed (with SIGKILL, apparently), and after restart it was complaining

2017-10-19 08:28:08 7fc57bf60740 InnoDB: Error: page 1 log sequence number 11717753894
 
InnoDB: is in the future! Current system log sequence number 11717530202.
 
InnoDB: Your database may be corrupt or you may have copied the InnoDB
 
InnoDB: tablespace but not the InnoDB log files. See

Is it possible that between October 11th and 19th somebody indeed copied or moved around some of InnoDB files?

Comment by Pokraczny [ 2017-11-13 ]

No one changed anything. The only process that put mails was imapsync, and on the mysql side it was the zimbra that put data in the database and in mailboxes. The database itself is over 7 GB + 500 GB in mail files

Comment by Pokraczny [ 2017-11-15 ]

Ok. I found solution. I run mysql in recovery mode - innodb_force_recovery = 3 . Next I get recovery databases from https://wiki.zimbra.com/wiki/Mysql_Crash_Recovery

Propably problem was in this error:

ERROR 1062 (23000) at line 303: Duplicate entry '413-cd30f5gVl01dYhExLdamvJex7mw=' for key 'PRIMARY'

Generated at Thu Feb 08 08:11:28 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.