Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9560

Mariadb 10.1 Crashes when replicating from 10.0

    XMLWordPrintable

Details

    Description

      When replicating from our 10.0 server, MariaDB 10.1.11 crashes. After the crash, it will not startup again unless all relay logs are deleted.

      Servers running Ubuntu, both with 200G ram.

      2016-02-15 15:12:47 140328625293056 [Warning] Slave SQL: Could not execute Write_rows_v1 event on table Voorraadbeheer.InklaarLog; Unknown error, Error_code: 1105; handler error No Error!; the event's master log mysql-bin.056373, end_log_pos 217054452, Gtid 0-765-1299574704, Internal MariaDB error code: 1105
      2016-02-15 15:12:47 140328625293056 [ERROR] Slave SQL: Could not read field 'Valuta' of table 'Voorraadbeheer.InklaarPrijs', Gtid 0-765-1299574704, Internal MariaDB error code: 1610
      160215 15:12:47 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.1.11-MariaDB-1~trusty-log
      key_buffer_size=16777216
      read_buffer_size=131072
      max_used_connections=0
      max_threads=502
      thread_count=1
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1118986 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0x7f95a6c2b008
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7fa0cddcb590 thread_stack 0x80000
      2016-02-15 15:12:47 140328649512704 [Note] Slave I/O thread: connected to master 'slave@jen.managementboek.nl:3306',replication started in log 'mysql-bin.056376' at position 692280882
      addr2line: 'mysqld': No such file
      mysqld(my_print_stacktrace+0x2e)[0x559e045013fe]
      mysqld(handle_fatal_signal+0x38d)[0x559e0402ef9d]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fa0d02e5340]
      mysqld(_Z10unpack_rowP14rpl_group_infoP5TABLEjPKhPK9st_bitmapPS4_PmS4_+0x89)[0x559e0410f0b9]
      mysqld(_ZN21Update_rows_log_event11do_exec_rowEP14rpl_group_info+0x220)[0x559e0410be80]
      mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x31c)[0x559e040fe2cc]
      mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x1e1)[0x559e03e3a2e1]
      mysqld(handle_slave_sql+0x261b)[0x559e03e3dd1b]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fa0d02dd182]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fa0cfa0047d]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x0): is an invalid pointer
      Connection ID (thread ID): 5
      Status: NOT_KILLED

      gdb shows the following:

      (gdb) bt
      #0  0x0000555555be30b9 in unpack_row(rpl_group_info*, TABLE*, unsigned int, unsigned char const*, st_bitmap const*, unsigned char const**, unsigned long*, unsigned char const*) ()
      #1  0x0000555555bdfe80 in Update_rows_log_event::do_exec_row(rpl_group_info*) ()
      #2  0x0000555555bd22cc in Rows_log_event::do_apply_event(rpl_group_info*) ()
      #3  0x000055555590e2e1 in apply_event_and_update_pos(Log_event*, THD*, rpl_group_info*, rpl_parallel_thread*) ()
      #4  0x0000555555911d1b in handle_slave_sql ()
      #5  0x00007ffff6afe182 in start_thread (arg=0x7ffff5c85700) at pthread_create.c:312
      #6  0x00007ffff622147d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

      Slave my.cnf

      [client]
      port            = 3306
      socket          = /var/run/mysqld/mysqld.sock
      back_log=1500
       
      # # UTF-8
      character_set_server = utf8
      collation_server = utf8_general_ci
       
      #
      # * Logging and Replication
      #
      #log_slow_queries       = /var/log/mysql/mysql-slow.log
      long_query_time         = 1
      log-queries-not-using-indexes
      log-output              = TABLE
      log_warnings            = 2
       
      sync_binlog             = 0
      binlog_checksum         = crc32
      binlog_commit_wait_usec = 50000
      binlog_commit_wait_count= 20
      server-id               = 345 
      log_bin                 = /data/bigmomma/mysql/log/mysql-bin.log
      binlog_format           = MIXED
      expire_logs_days        = 7
      max_binlog_size         = 1G
      log-slave-updates
      slave_compressed_protocol = 1
      relay-log               = /data/bigmomma/mysql/log/mysql-relay.log
       
       
      # INNODB #
      innodb_log_files_in_group      = 2
      innodb_flush_log_at_trx_commit = 0
      innodb_buffer_pool_size        = 40G
      innodb_file_per_table
      innodb_log_file_size=1G
      innodb_flush_method=O_DIRECT
      innodb_file_format=barracuda
      innodb_io_capacity = 20000
      innodb_buffer_pool_instances=10
      innodb_log_buffer_size=512M
      #innodb_adaptive_checkpoint = keep_average
      #innodb_flush_neighbor_pages = 0
      group_concat_max_len = 8096
       
      [mysqldump]
      quick
      quote-names
      max_allowed_packet      = 16M
      ignore-table            = telefoongids.white_subscriber
      ignore-table            = telefoongids.pink_subscriber

      Master my.cnf

      [client]
      port            = 3306
      socket          = /var/run/mysqld/mysqld.sock
       
      # Here is entries for some specific programs
      # The following values assume you have at least 32M ram
       
      # This was formally known as [safe_mysqld]. Both versions are currently parsed.
      [mysqld_safe]
      socket          = /var/run/mysqld/mysqld.sock
      nice            = 0
       
      [mysqld]
      replicate-wild-ignore-table = %.Voorafgaand%_Klanten
      replicate-wild-ignore-table = tmp.%
      transaction-isolation =  READ-COMMITTED
       
      user            = mysql
      pid-file        = /var/run/mysqld/mysqld.pid
      socket          = /var/run/mysqld/mysqld.sock
      port            = 3306
      basedir         = /usr
      datadir         = /var/lib/mysql
      tmpdir          = /tmp
      lc-messages-dir = /usr/share/mysql
      skip-external-locking
       
      log-error = /data/bigmomma/mysql/log/error.log
      innodb_stats_on_metadata = 0
      lock_wait_timeout = 20
      innodb_lock_wait_timeout = 20
       
      #
      # Instead of skip-networking the default is now to listen only on
      # localhost which is more compatible and is not less secure.
      bind-address            = 0.0.0.0
       
      # GTIDs
      master_info_repository = table
      relay_log_info_repository = table
      master-verify-checksum = on
      slave-sql-verify-checksum = on
       
       
      #
      # * Fine Tuning
      #
      key_buffer              = 16M
      #max_allowed_packet     = 16M
      thread_stack            = 512K
      thread_cache_size       = 8
      myisam-recover         = BACKUP
       
      # SAFETY #
      max_allowed_packet             = 64M
      max_connect_errors             = 1000000
      sysdate_is_now                 = 1
       
      max_connections=500
      event_scheduler=ON
      table_cache     = 3000
       
      tmp_table_size                 = 64M
      max_heap_table_size            = 64M
      query_cache_type               = 1
      query_cache_size               = 8M
      max_connections                = 500
      thread_cache_size              = 50
      open_files_limit               = 65535
      table_definition_cache         = 4096
      table_open_cache               = 4096
       
      max_sp_recursion_depth         = 2
       
      back_log=1500
       
       
      # UTF-8
      character_set_server = utf8
      collation_server = utf8_general_ci
       
       
      #
      # * Logging and Replication
      #
      #log_slow_queries       = /var/log/mysql/mysql-slow.log
      long_query_time         = 1
      log-queries-not-using-indexes
      log-output              = TABLE
      log_warnings            = 2
       
      sync_binlog             = 0
      binlog_checksum         = crc32
      binlog_commit_wait_usec = 50000
      binlog_commit_wait_count= 20
      server-id               = 765
      log_bin                 = /data/bigmomma/mysql/log/mysql-bin.log
      binlog_format           = MIXED
      expire_logs_days        = 7
      max_binlog_size         = 1G
      log-slave-updates
      slave_compressed_protocol = 1
      relay-log               = /data/bigmomma/mysql/log/mysql-relay.log
       
       
      # INNODB #
      innodb_log_files_in_group      = 2
      innodb_flush_log_at_trx_commit = 0
      innodb_buffer_pool_size        = 40G
      innodb_file_per_table
      innodb_log_file_size=1G
      innodb_flush_method=O_DIRECT
      innodb_file_format=barracuda
      innodb_io_capacity = 20000
      innodb_buffer_pool_instances=10
      innodb_log_buffer_size=32M
       
      group_concat_max_len = 8096
       
      [mysqldump]
      quick
      quote-names
      max_allowed_packet      = 16M
      gtid                    = ON
      ignore-table            = telefoongids.white_subscriber
      ignore-table            = telefoongids.pink_subscriber
       
      [mysql]
      #no-auto-rehash # faster start of mysql but no tab completition
       
      [isamchk]
      key_buffer              = 16M

      Attachments

        Issue Links

          Activity

            People

              serg Sergei Golubchik
              CrewOne Sander Pilon
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.