Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Cannot Reproduce
-
10.3.8
-
Ubuntu 16.04 LTS, installed from package mariadb.
Description
Hi there,
After a restart of one node in a galera cluster of 3 nodes (running inside a 1GB LAN) the node was no longer able to start with (see traceback below). During restart server was put under a load (mainly inserts and updates).
===
Jul 16 15:12:24 node2 systemd[1]: Starting MariaDB 10.3.8 database server...
Jul 16 15:12:45 node2 mysqld[14057]: WSREP: Failed to start mysqld for wsrep recovery: '2018-07-16 15:12:24 0 [Note] /usr/sbin/mysqld (mysqld 10.3.8-MariaDB-1:10.3.8+maria~xenial-log) starting as process 14230 ...
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: For Galera, using innodb_lock_schedule_algorithm=fcfs
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Using Linux native AIO
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Uses event mutexes
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Number of pools: 1
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Using generic crc32 instructions
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: Completed initialization of buffer pool
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:24 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:25 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=17920587334
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:36 0 [Note] InnoDB: 6 transaction(s) which must be rolled back or cleaned up in total 4200 row operations to undo
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:36 0 [Note] InnoDB: Trx id counter is 123430
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:36 0 [Note] InnoDB: Starting final batch to recover 1808 pages from redo log.
Jul 16 15:12:45 node2 mysqld[14057]: 2018-07-16 15:12:36 0x7f83c4f03700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.3.8/storage/innobase/rem/rem0rec.cc line 828
Jul 16 15:12:45 node2 mysqld[14057]: InnoDB: We intentionally generate a memory trap.
Jul 16 15:12:45 node2 mysqld[14057]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
Jul 16 15:12:45 node2 mysqld[14057]: InnoDB: If you get repeated assertion failures or crashes, even
Jul 16 15:12:45 node2 mysqld[14057]: InnoDB: immediately after the mysqld startup, there may be
Jul 16 15:12:45 node2 mysqld[14057]: InnoDB: corruption in the InnoDB tablespace. Please refer to
Jul 16 15:12:45 node2 mysqld[14057]: InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/
Jul 16 15:12:45 node2 mysqld[14057]: InnoDB: about forcing recovery.
Jul 16 15:12:45 node2 mysqld[14057]: 180716 15:12:36 [ERROR] mysqld got signal 6 ;
Jul 16 15:12:45 node2 mysqld[14057]: This could be because you hit a bug. It is also possible that this binary
Jul 16 15:12:45 node2 mysqld[14057]: or one of the libraries it was linked against is corrupt, improperly built,
Jul 16 15:12:45 node2 mysqld[14057]: or misconfigured. This error can also be caused by malfunctioning hardware.
Jul 16 15:12:45 node2 mysqld[14057]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Jul 16 15:12:45 node2 mysqld[14057]: We will try our best to scrape up some info that will hopefully help
Jul 16 15:12:45 node2 mysqld[14057]: diagnose the problem, but since we have already crashed,
Jul 16 15:12:45 node2 mysqld[14057]: something is definitely wrong and this may fail.
Jul 16 15:12:45 node2 mysqld[14057]: Server version: 10.3.8-MariaDB-1:10.3.8+maria~xenial-log
Jul 16 15:12:45 node2 mysqld[14057]: key_buffer_size=134217728
Jul 16 15:12:45 node2 mysqld[14057]: read_buffer_size=2097152
Jul 16 15:12:45 node2 mysqld[14057]: max_used_connections=0
Jul 16 15:12:45 node2 mysqld[14057]: max_threads=1002
Jul 16 15:12:45 node2 mysqld[14057]: thread_count=0
Jul 16 15:12:45 node2 mysqld[14057]: It is possible that mysqld could use up to
Jul 16 15:12:45 node2 mysqld[14057]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6309726 K bytes of memory
Jul 16 15:12:45 node2 mysqld[14057]: Hope that's ok; if not, decrease some variables in the equation.
Jul 16 15:12:45 node2 mysqld[14057]: Thread pointer: 0x0
Jul 16 15:12:45 node2 mysqld[14057]: Attempting backtrace. You can use the following information to find out
Jul 16 15:12:45 node2 mysqld[14057]: where mysqld died. If you see no messages after this, something went
Jul 16 15:12:45 node2 mysqld[14057]: terribly wrong...
Jul 16 15:12:45 node2 mysqld[14057]: stack_bottom = 0x0 thread_stack 0x49000
Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56420ac5c32e]
Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(handle_fatal_signal+0x347)[0x56420a707df7]
Jul 16 15:12:45 node2 mysqld[14057]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f846fd02390]
Jul 16 15:12:45 node2 mysqld[14057]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f846f2d1428]
Jul 16 15:12:45 node2 mysqld[14057]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f846f2d302a]
Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0x49f4e2)[0x56420a4544e2]
Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0xa373a3)[0x56420a9ec3a3]
Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0xa15a7e)[0x56420a9caa7e]
Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0x9f52e5)[0x56420a9aa2e5]
Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0x9f6bd9)[0x56420a9abbd9]
Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0xb2e847)[0x56420aae3847]
Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0xb99e69)[0x56420ab4ee69]
Jul 16 15:12:45 node2 mysqld[14057]: /usr/sbin/mysqld(+0xaa5dd8)[0x56420aa5add8]
Jul 16 15:12:45 node2 mysqld[14057]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f846fcf86ba]
Jul 16 15:12:45 node2 mysqld[14057]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f846f3a341d]
Jul 16 15:12:45 node2 mysqld[14057]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Jul 16 15:12:45 node2 mysqld[14057]: information that should help you find out what is causing the crash.
Jul 16 15:12:45 node2 mysqld[14057]: Aborted (core dumped)'
Jul 16 15:12:45 node2 systemd[1]: mariadb.service: Control process exited, code=exited status=1
Jul 16 15:12:45 node2 systemd[1]: Failed to start MariaDB 10.3.8 database server.
Jul 16 15:12:45 node2 systemd[1]: mariadb.service: Unit entered failed state.
Jul 16 15:12:45 node2 systemd[1]: mariadb.service: Failed with result 'exit-code'.
===