One of the nodes out of cluster of 3 MariaDB servers lost connection to 2 other nodes.
Then one node out of the 2 remaining nodes crashed because of connectivity issues.
It was restarted and the recovery failed with the below errors:
,2018-03-31 5:27:14 139632092739520 [Note] InnoDB: Using mutexes to ref count buffer pool pages
,2018-03-31 5:27:14 139632092739520 [Note] InnoDB: The InnoDB memory heap is disabled
,2018-03-31 5:27:14 139632092739520 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
,2018-03-31 5:27:14 139632092739520 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
,2018-03-31 5:27:14 139632092739520 [Note] InnoDB: Compressed tables use zlib 1.2.8
,2018-03-31 5:27:14 139632092739520 [Note] InnoDB: Using Linux native AIO
,2018-03-31 5:27:14 139632092739520 [Note] InnoDB: Using SSE crc32 instructions
,2018-03-31 5:27:14 139632092739520 [Note] InnoDB: Initializing buffer pool, size = 256.0M
,2018-03-31 5:27:14 139632092739520 [Note] InnoDB: Completed initialization of buffer pool
,2018-03-31 5:27:14 139632092739520 [Note] InnoDB: Highest supported file format is Barracuda.
,2018-03-31 5:27:14 139632092739520 [Note] InnoDB: Starting crash recovery from checkpoint LSN=185850334837
,2018-03-31 5:27:16 139632092739520 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
,2018-03-31 5:27:18 139632092739520 [Note] InnoDB: Starting final batch to recover 24 pages from redo log
,2018-03-31 05:27:18 7efe7bbfb700 InnoDB: Assertion failure in thread 139631462954752 in file page0cur.cc line 931
,InnoDB: We intentionally generate a memory trap.
,InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
,InnoDB: If you get repeated assertion failures or crashes, even
,InnoDB: immediately after the mysqld startup, there may be
,InnoDB: corruption in the InnoDB tablespace. Please refer to
,InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
,InnoDB: about forcing recovery.
,180331 5:27:18 [ERROR] mysqld got signal 6 ;
,This could be because you hit a bug. It is also possible that this binary
,or one of the libraries it was linked against is corrupt, improperly built,
,or misconfigured. This error can also be caused by malfunctioning hardware.
,
,To report this bug, see https://mariadb.com/kb/en/reporting-bugs
,
,We will try our best to scrape up some info that will hopefully help
,diagnose the problem, but since we have already crashed,
,something is definitely wrong and this may fail.
,
,Server version: 10.1.30-MariaDB-1~jessie
,key_buffer_size=134217728
,read_buffer_size=2097152
,max_used_connections=0
,max_threads=1026
,thread_count=0
,It is possible that mysqld could use up to
,key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6456140 K bytes of memory
,Hope that's ok; if not, decrease some variables in the equation.
,
,Thread pointer: 0x0
,Attempting backtrace. You can use the following information to find out
,where mysqld died. If you see no messages after this, something went
,terribly wrong...
,stack_bottom = 0x0 thread_stack 0x48400
,mysqld(my_print_stacktrace+0x2e)[0x7efea1f2824e]
,mysqld(handle_fatal_signal+0x2fd)[0x7efea1a5d55d]
,/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7efea107a890]
,/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7efe9f3d8067]
,/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7efe9f3d9448]
,mysqld(+0x78feb4)[0x7efea1c3aeb4]
,mysqld(+0x76d626)[0x7efea1c18626]
,mysqld(+0x76fef7)[0x7efea1c1aef7]
,mysqld(+0x87575c)[0x7efea1d2075c]
,mysqld(+0x8c925c)[0x7efea1d7425c]
,mysqld(+0x7fe560)[0x7efea1ca9560]
,/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7efea1073064]
,/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7efe9f48b62d]
,The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
,information that should help you find out what is causing the crash.