Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-11894

Galera crashes after a few minutes - unusable

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 10.2.3
    • Fix Version/s: 10.2.5
    • Component/s: Galera
    • Labels:
      None
    • Environment:
      Ubuntu 16.04.1 LTS
    • Sprint:
      10.0.30, 10.3.1-1

      Description

      I've restarted completely from scratch multiple times (e.g. wipe /var/lib/mysql, mysql_install_db, restore dump). I just can't get the second node to run. Below I quote first my mysql.cnf, then two variations of the errors I get.

      [mysqld]
       
      #Recommended by Drupal
      max_allowed_packet=512M
       
      #Try to prevent crashes
      skip-log-bin
       
      # Replicate faster :-)
      wsrep_slave_threads = 32
       
      #Query Cache for faster performance
      query_cache_type = 1
      query_cache_size = 256M
       
      #mysql settings mandatory for galera
      binlog_format=ROW
      binlog_row_image=MINIMAL
      default_storage_engine=InnoDB
      innodb_autoinc_lock_mode=2
      bind-address=0.0.0.0
      innodb_flush_log_at_trx_commit=0
       
      max_connections = 300
       
      #galera settings
      wsrep_on=ON
      wsrep_provider=/usr/lib/galera/libgalera_smm.so
      wsrep_cluster_name="uhu_cluster"
      wsrep_cluster_address="gcomm://node1,node2"
      wsrep_node_address=192.168.0.1
       
      enforce_storage_engine=InnoDB
      sql_mode=NO_AUTO_CREATE_USER
      

      Jan 24 00:44:21 uhu1 mysqld[14620]: 2017-01-24  0:44:21 140172607566592 [ERROR] [FATAL] InnoDB: SYS_COLUMNS.TABLE_ID mismatch
      Jan 24 00:44:21 uhu1 mysqld[14620]: 2017-01-24 00:44:21 0x7f7c7a7b3300  InnoDB: Assertion failure in thread 140172607566592 in file ut0ut.cc line 949
      Jan 24 00:44:21 uhu1 mysqld[14620]: InnoDB: We intentionally generate a memory trap.
      Jan 24 00:44:21 uhu1 mysqld[14620]: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      Jan 24 00:44:21 uhu1 mysqld[14620]: InnoDB: If you get repeated assertion failures or crashes, even
      Jan 24 00:44:21 uhu1 mysqld[14620]: InnoDB: immediately after the mysqld startup, there may be
      Jan 24 00:44:21 uhu1 mysqld[14620]: InnoDB: corruption in the InnoDB tablespace. Please refer to
      Jan 24 00:44:21 uhu1 mysqld[14620]: InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
      Jan 24 00:44:21 uhu1 mysqld[14620]: InnoDB: about forcing recovery.
      Jan 24 00:44:21 uhu1 mysqld[14620]: 170124  0:44:21 [ERROR] mysqld got signal 6 ;
      Jan 24 00:44:21 uhu1 mysqld[14620]: This could be because you hit a bug. It is also possible that this binary
      Jan 24 00:44:21 uhu1 mysqld[14620]: or one of the libraries it was linked against is corrupt, improperly built,
      Jan 24 00:44:21 uhu1 mysqld[14620]: or misconfigured. This error can also be caused by malfunctioning hardware.
      Jan 24 00:44:21 uhu1 mysqld[14620]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      Jan 24 00:44:21 uhu1 mysqld[14620]: We will try our best to scrape up some info that will hopefully help
      Jan 24 00:44:21 uhu1 mysqld[14620]: diagnose the problem, but since we have already crashed,
      Jan 24 00:44:21 uhu1 mysqld[14620]: something is definitely wrong and this may fail.
      Jan 24 00:44:21 uhu1 mysqld[14620]: Server version: 10.2.3-MariaDB-10.2.3+maria~xenial
      Jan 24 00:44:21 uhu1 mysqld[14620]: key_buffer_size=134217728
      Jan 24 00:44:21 uhu1 mysqld[14620]: read_buffer_size=2097152
      Jan 24 00:44:21 uhu1 mysqld[14620]: max_used_connections=10
      Jan 24 00:44:21 uhu1 mysqld[14620]: max_threads=302
      Jan 24 00:44:21 uhu1 mysqld[14620]: thread_count=49
      Jan 24 00:44:21 uhu1 mysqld[14620]: It is possible that mysqld could use up to
      Jan 24 00:44:21 uhu1 mysqld[14620]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1992895 K  bytes of memory
      Jan 24 00:44:21 uhu1 mysqld[14620]: Hope that's ok; if not, decrease some variables in the equation.
      Jan 24 00:44:21 uhu1 mysqld[14620]: Thread pointer: 0x0x7f7c2b072008
      Jan 24 00:44:21 uhu1 mysqld[14620]: Attempting backtrace. You can use the following information to find out
      Jan 24 00:44:21 uhu1 mysqld[14620]: where mysqld died. If you see no messages after this, something went
      Jan 24 00:44:21 uhu1 mysqld[14620]: terribly wrong...
      Jan 24 00:44:21 uhu1 mysqld[14620]: stack_bottom = 0x7f7c7a7b28f8 thread_stack 0x48c00
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55eb89297dae]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(handle_fatal_signal+0x2dd)[0x55eb88cedb1d]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f7c9a6eb390]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f7c99cbb428]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f7c99cbd02a]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(+0x3fb37a)[0x55eb88aa837a]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(+0xa0f896)[0x55eb890bc896]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(+0xaaa4b6)[0x55eb891574b6]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(+0xaabfd7)[0x55eb89158fd7]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(+0xa9c680)[0x55eb89149680]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(+0x87e8d9)[0x55eb88f2b8d9]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(+0x891a47)[0x55eb88f3ea47]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcij+0x33)[0x55eb88cf1e33]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPKcjjjP5TABLEb+0x6c3)[0x55eb88c01883]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP18Open_table_context+0x9c3)[0x55eb88b10bc3]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(_Z11open_tablesP3THDRK14DDL_options_stPP10TABLE_LISTPjjP19Prelocking_strategy+0xec6)[0x55eb88b139f6]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(_Z20open_and_lock_tablesP3THDRK14DDL_options_stP10TABLE_LISTbjP19Prelocking_strategy+0x33)[0x55eb88b13e43]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(_ZN22Sql_cmd_truncate_table16handler_truncateEP3THDP10TABLE_LISTb+0x80)[0x55eb88e3b800]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(_ZN22Sql_cmd_truncate_table14truncate_tableEP3THDP10TABLE_LIST+0x1d8)[0x55eb88e3c108]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(_ZN22Sql_cmd_truncate_table7executeEP3THD+0x50)[0x55eb88e3c1b0]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x12f7)[0x55eb88b57d77]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x28a)[0x55eb88b5faea]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(+0x4b32c7)[0x55eb88b602c7]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x11d1)[0x55eb88b61a91]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(_Z10do_commandP3THD+0x17c)[0x55eb88b6302c]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x24a)[0x55eb88c260ea]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55eb88c2627d]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f7c9a6e16ba]
      Jan 24 00:44:21 uhu1 mysqld[14620]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f7c99d8c82d]
      Jan 24 00:44:21 uhu1 mysqld[14620]: Trying to get some variables.
      Jan 24 00:44:21 uhu1 mysqld[14620]: Some pointers may be invalid and cause the dump to abort.
      Jan 24 00:44:21 uhu1 mysqld[14620]: Query (0x7f7c2b09d020): TRUNCATE simple_sitemap
      Jan 24 00:44:21 uhu1 mysqld[14620]: Connection ID (thread ID): 358
      Jan 24 00:44:21 uhu1 mysqld[14620]: Status: NOT_KILLED
      Jan 24 00:44:21 uhu1 mysqld[14620]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
      Jan 24 00:44:21 uhu1 mysqld[14620]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      Jan 24 00:44:21 uhu1 mysqld[14620]: information that should help you find out what is causing the crash.
      

      Jan 24 00:51:58 uhu1 mysqld[28280]: 2017-01-24 00:51:58 0x7fb7df3fc700  InnoDB: Assertion failure in thread 140427701241600 in file trx0purge.cc line 168
      Jan 24 00:51:58 uhu1 mysqld[28280]: InnoDB: Failing assertion: purge_sys->iter.trx_no <= purge_sys->rseg->last_trx_no
      Jan 24 00:51:58 uhu1 mysqld[28280]: InnoDB: We intentionally generate a memory trap.
      Jan 24 00:51:58 uhu1 mysqld[28280]: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      Jan 24 00:51:58 uhu1 mysqld[28280]: InnoDB: If you get repeated assertion failures or crashes, even
      Jan 24 00:51:58 uhu1 mysqld[28280]: InnoDB: immediately after the mysqld startup, there may be
      Jan 24 00:51:58 uhu1 mysqld[28280]: InnoDB: corruption in the InnoDB tablespace. Please refer to
      Jan 24 00:51:58 uhu1 mysqld[28280]: InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
      Jan 24 00:51:58 uhu1 mysqld[28280]: InnoDB: about forcing recovery.
      Jan 24 00:51:58 uhu1 mysqld[28280]: 170124  0:51:58 [ERROR] mysqld got signal 6 ;
      Jan 24 00:51:58 uhu1 mysqld[28280]: This could be because you hit a bug. It is also possible that this binary
      Jan 24 00:51:58 uhu1 mysqld[28280]: or one of the libraries it was linked against is corrupt, improperly built,
      Jan 24 00:51:58 uhu1 mysqld[28280]: or misconfigured. This error can also be caused by malfunctioning hardware.
      Jan 24 00:51:58 uhu1 mysqld[28280]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      Jan 24 00:51:58 uhu1 mysqld[28280]: We will try our best to scrape up some info that will hopefully help
      Jan 24 00:51:58 uhu1 mysqld[28280]: diagnose the problem, but since we have already crashed,
      Jan 24 00:51:58 uhu1 mysqld[28280]: something is definitely wrong and this may fail.
      Jan 24 00:51:58 uhu1 mysqld[28280]: Server version: 10.2.3-MariaDB-10.2.3+maria~xenial
      Jan 24 00:51:58 uhu1 mysqld[28280]: key_buffer_size=134217728
      Jan 24 00:51:58 uhu1 mysqld[28280]: read_buffer_size=2097152
      Jan 24 00:51:58 uhu1 mysqld[28280]: max_used_connections=4
      Jan 24 00:51:58 uhu1 mysqld[28280]: max_threads=302
      Jan 24 00:51:58 uhu1 mysqld[28280]: thread_count=43
      Jan 24 00:51:58 uhu1 mysqld[28280]: It is possible that mysqld could use up to
      Jan 24 00:51:58 uhu1 mysqld[28280]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1992895 K  bytes of memory
      Jan 24 00:51:58 uhu1 mysqld[28280]: Hope that's ok; if not, decrease some variables in the equation.
      Jan 24 00:51:58 uhu1 mysqld[28280]: Thread pointer: 0x0x7fb7e8412008
      Jan 24 00:51:58 uhu1 mysqld[28280]: Attempting backtrace. You can use the following information to find out
      Jan 24 00:51:58 uhu1 mysqld[28280]: where mysqld died. If you see no messages after this, something went
      Jan 24 00:51:58 uhu1 mysqld[28280]: terribly wrong...
      Jan 24 00:51:58 uhu1 mysqld[28280]: stack_bottom = 0x7fb7df3fbb68 thread_stack 0x48c00
      Jan 24 00:51:58 uhu1 mysqld[28280]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e8162d0dae]
      Jan 24 00:51:58 uhu1 mysqld[28280]: /usr/sbin/mysqld(handle_fatal_signal+0x2dd)[0x55e815d26b1d]
      Jan 24 00:51:58 uhu1 mysqld[28280]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fb845168390]
      Jan 24 00:51:58 uhu1 mysqld[28280]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7fb844738428]
      Jan 24 00:51:58 uhu1 mysqld[28280]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fb84473a02a]
      Jan 24 00:51:58 uhu1 mysqld[28280]: /usr/sbin/mysqld(+0x3fb37a)[0x55e815ae137a]
      Jan 24 00:51:58 uhu1 mysqld[28280]: /usr/sbin/mysqld(+0x9d0403)[0x55e8160b6403]
      Jan 24 00:51:58 uhu1 mysqld[28280]: /usr/sbin/mysqld(+0x9d5630)[0x55e8160bb630]
      Jan 24 00:51:58 uhu1 mysqld[28280]: /usr/sbin/mysqld(+0x9d618c)[0x55e8160bc18c]
      Jan 24 00:51:58 uhu1 mysqld[28280]: /usr/sbin/mysqld(+0x9affa8)[0x55e816095fa8]
      Jan 24 00:51:58 uhu1 mysqld[28280]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fb84515e6ba]
      Jan 24 00:51:58 uhu1 mysqld[28280]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fb84480982d]
      Jan 24 00:51:58 uhu1 mysqld[28280]: Trying to get some variables.
      Jan 24 00:51:58 uhu1 mysqld[28280]: Some pointers may be invalid and cause the dump to abort.
      Jan 24 00:51:58 uhu1 mysqld[28280]: Query (0x0):
      Jan 24 00:51:58 uhu1 mysqld[28280]: Connection ID (thread ID): 4
      Jan 24 00:51:58 uhu1 mysqld[28280]: Status: NOT_KILLED
      Jan 24 00:51:58 uhu1 mysqld[28280]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
      Jan 24 00:51:58 uhu1 mysqld[28280]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      Jan 24 00:51:58 uhu1 mysqld[28280]: information that should help you find out what is causing the crash.
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jplindst Jan Lindström
              Reporter:
              rgpublic Ranjan Ghosh
              Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.