Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29229

Server crashes every time the slave SQL thread encounters an error

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Duplicate
    • 10.6.7
    • N/A
    • Replication, Server
    • None
    • Ubuntu 22.04 LTS (GNU/Linux 5.15.0-1013-aws x86_64)

    Description

      Server crashes every time the slave SQL thread encounters an error.

      2022-08-02 11:22:36 6 [ERROR] Slave SQL: Error 'Table 'superset.ab_user' doesn't exist' on query. Default database: 'superset'. Query: 'UPD
      ATE ab_user SET last_login='2022-08-02 02:22:36.802940', fail_login_count=1 WHERE ab_user.id = 3', Internal MariaDB error code: 1146
      2022-08-02 11:22:36 6 [Warning] Slave: Table 'superset.ab_user' doesn't exist Error_code: 1146
      2022-08-02 11:22:36 6 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE 
      START". We stopped at log 'mysql-bin-changelog.117203' position 30457705
      2022-08-02 11:22:36 6 [Note] Slave SQL thread exiting, replication stopped in log 'mysql-bin-changelog.117203' at position 30457705, master
      : pod101-aurora-globaldb-instance-1.ci5uha4gpirp.us-east-1.rds.amazonaws.com:3306
      220802 11:22:36 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.6.7-MariaDB-2ubuntu1
      key_buffer_size=4294967296
      read_buffer_size=33554432
      max_used_connections=10
      max_threads=502
      thread_count=5
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 37106421 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x49000
      ??:0(my_print_stacktrace)[0x5561595d7cf0]
      ??:0(handle_fatal_signal)[0x55615917f118]
      ??:0(__sigaction)[0x7f4af3ec1520]
      ??:0(pthread_kill)[0x7f4af3f15a7c]
      ??:0(raise)[0x7f4af3ec1476]
      ??:0(abort)[0x7f4af3ea77f3]
      ??:0(psi_prlock_wrlock)[0x556158dc31dc]
      ??:0(wsrep::wsrep_provider_v26::status() const)[0x5561596af8af]
      ??:0(_Unwind_GetTextRelBase)[0x7f4af3e75de6]
      ??:0(_Unwind_ForcedUnwind)[0x7f4af3e764e0]
      ??:0(sem_trywait)[0x7f4af3f1c4c6]
      ??:0(pthread_exit)[0x7f4af3f14d3a]
      ??:0(handle_slave_sql)[0x556158e75c45]
      ??:0(aria_get_capabilities)[0x55615937f83a]
      ??:0(pthread_condattr_setpshared)[0x7f4af3f13b43]
      ??:0(__xmknodat)[0x7f4af3fa5a00]
      
      

      Attachments

        Issue Links

          Activity

            danblack Daniel Black added a comment -

            Most likely a duplicate of MDEV-25633

            danblack Daniel Black added a comment - Most likely a duplicate of MDEV-25633

            the stack trace makes no sense. May be it's corrupted

            where did you get the binary from?

            serg Sergei Golubchik added a comment - the stack trace makes no sense. May be it's corrupted where did you get the binary from?
            danblack Daniel Black added a comment -

            Stack traces have been corrupted in the same way for innodb for quite a while. Cause still unknown.

            By the version this is the original ubuntu released MariaDB-10.6.

            Install the mariadb from the jammy updates and it will have the LTO fix (http://changelogs.ubuntu.com/changelogs/pool/universe/m/mariadb-10.6/mariadb-10.6_10.6.7-2ubuntu1.1/changelog) which should resolve this issue.

            This upstream bug https://bugs.launchpad.net/ubuntu/+source/mariadb-10.6/+bug/1979695 is the same. While your slave thread was stopped due to errors rather than manually, but the end interaction of stopping the SQL slave thread is the same. This as best we've looked so far is some interaction between LTO causing an assert in the pthread_exit (MDEV-25633).

            danblack Daniel Black added a comment - Stack traces have been corrupted in the same way for innodb for quite a while. Cause still unknown. By the version this is the original ubuntu released MariaDB-10.6. Install the mariadb from the jammy updates and it will have the LTO fix ( http://changelogs.ubuntu.com/changelogs/pool/universe/m/mariadb-10.6/mariadb-10.6_10.6.7-2ubuntu1.1/changelog ) which should resolve this issue. This upstream bug https://bugs.launchpad.net/ubuntu/+source/mariadb-10.6/+bug/1979695 is the same. While your slave thread was stopped due to errors rather than manually, but the end interaction of stopping the SQL slave thread is the same. This as best we've looked so far is some interaction between LTO causing an assert in the pthread_exit ( MDEV-25633 ).

            People

              Unassigned Unassigned
              marvin Marvin Hoffmann
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.