Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-19249

Slave MariaDB 10.2.23 crashing when replication is started

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Duplicate
    • 10.2.23
    • N/A
    • Replication
    • CentOS Linux release 7.6.1810 (Core)

      Linux db05.local 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linu

    Description

      Several databases with common size 3.2 TB

      Master - 10.1.37-MariaDB (binlog_format=ROW)
      Slave1 - 10.1.37-MariaDB
      Slave2 - 10.2.23-MariaDB (same crashing problem was with 10.3.13-MariaDB then we revert back to 10.1.37 and restore replication and upgrade to 10.2.23)

      Slave2 after update from 10.1.37-MariaDB to 10.2.23-MariaDB and running mysql_upgrade
      staring without any problem.
      replication also started ok and synced with master.

      Then after several hours slave suddenly catching signal 11 and crashing.
      Auto-recovery wasn't success (server doing recovery) and controld stops server by timeout.

      After restarting with --skip_replication server working ok.
      Any selects from db are ok. Seems DB is working fine.
      After running 'start slave;' replication is resumed and workign fine just not so long.
      After random period 30sec-30min from start replication problem is coming back. Server falling down.

      Below is a partial output from /var/log/messages:
      BEGIN
      mysqld: 190412 10:25:05 [ERROR] mysqld got signal 11 ;
      mysqld: This could be because you hit a bug. It is also possible that this binary
      mysqld: or one of the libraries it was linked against is corrupt, improperly built,
      mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
      mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      mysqld: We will try our best to scrape up some info that will hopefully help
      mysqld: diagnose the problem, but since we have already crashed,
      mysqld: something is definitely wrong and this may fail.
      mysqld: Server version: 10.2.23-MariaDB-log
      mysqld: key_buffer_size=268435456
      mysqld: read_buffer_size=131072
      mysqld: max_used_connections=1
      mysqld: max_threads=102
      mysqld: thread_count=10
      mysqld: It is possible that mysqld could use up to
      mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 486249 K bytes of memory
      mysqld: Hope that's ok; if not, decrease some variables in the equation.
      mysqld: Thread pointer: 0x7ef808001298
      mysqld: Attempting backtrace. You can use the following information to find out
      mysqld: where mysqld died. If you see no messages after this, something went
      mysqld: terribly wrong...
      mysqld: stack_bottom = 0x7ef8a0168180 thread_stack 0x49000
      mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x559e1f63ecae]
      mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x355)[0x559e1f0c5f35]
      mysqld: /lib64/libpthread.so.0(+0xf5d0)[0x7f097b10c5d0]
      mysqld: /usr/sbin/mysqld(ZN16Item_func_or_sum4walkEM4ItemFbPvEbS1+0x60)[0x559e1eef1a60]
      mysqld: /usr/sbin/mysqld(_Z21fix_session_vcol_exprP3THDP19Virtual_column_info+0x3c)[0x559e1efd36dc]
      mysqld: /usr/sbin/mysqld(_Z11lock_tablesP3THDP10TABLE_LISTjj+0x3ff)[0x559e1eeeae6f]
      mysqld: /usr/sbin/mysqld(_Z20open_and_lock_tablesP3THDRK14DDL_options_stP10TABLE_LISTbjP19Prelocking_strategy+0x8a)[0x559e1eeec2ea]
      mysqld: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0xa3c)[0x559e1f1b8e2c]
      mysqld: /usr/sbin/mysqld(+0x46648b)[0x559e1eeb448b]
      mysqld: /usr/sbin/mysqld(handle_slave_sql+0x1833)[0x559e1eebceb3]
      mysqld: /lib64/libpthread.so.0(+0x7dd5)[0x7f097b104dd5]
      mysqld: /lib64/libc.so.6(clone+0x6d)[0x7f09794a6ead]
      mysqld: Trying to get some variables.
      mysqld: Some pointers may be invalid and cause the dump to abort.
      mysqld: Query (0x0):
      mysqld: Connection ID (thread ID): 11
      mysqld: Status: NOT_KILLED
      kernel: mysqld[41013]: segfault at 2d9 ip 0000559e1eef1a60 sp 00007ef8a01675e0 error 4 in mysqld[559e1ea4e000+1125000]
      mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
      mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      mysqld: information that should help you find out what is causing the crash.
      systemd: mariadb.service: main process exited, code=killed, status=11/SEGV
      systemd: Unit mariadb.service entered failed state.
      systemd: mariadb.service failed.
      systemd: mariadb.service holdoff time over, scheduling restart.
      systemd: Stopped MariaDB 10.2.23 database server.
      END

      Thank you very much

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              vitaly vitaly.t
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.