Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-23627

Error in /usr/sbin/mysqld: corrupted size vs. prev_size

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • None
    • N/A, 10.6.9

    Description

      MariaDB seems to crash and restart itself, giving a massive dump of error logging. This has happened to us on three separate occasions. I unfortunately don't know much else about what's going on. Please let me know what further information I need to provide.

      Since the error logging was too long to post in the description here I have dumped it to a Pastebin link that can be found here: https://pastebin.com/Yv4fHNF3, I have also attached it to this ticket.

      Attachments

        Activity

          AdamStark, it looks like there is a deadlock between InnoDB threads that leads to an InnoDB watchdog to kill the process.

          To debug the hang, we would need the stack traces of the process. Please see https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ and especially the section on getting stack traces while the process is running. I would like to see some samples of stack traces already when the server shows the first symptoms of hanging (the first "long semaphore wait" message appears).

          marko Marko Mäkelä added a comment - AdamStark , it looks like there is a deadlock between InnoDB threads that leads to an InnoDB watchdog to kill the process. To debug the hang, we would need the stack traces of the process. Please see https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ and especially the section on getting stack traces while the process is running. I would like to see some samples of stack traces already when the server shows the first symptoms of hanging (the first "long semaphore wait" message appears).
          AdamStark Adam Stark (Inactive) added a comment - - edited

          At the moment it appears that setting innodb_adaptive_hash_index to 0 has resolved the semaphore issue. Since these DB's are crashing in production I am hesitant to install additional debugging and profiling tools. If we experience this again I will install these tools and generate a full stack trace to supply additional details.

          AdamStark Adam Stark (Inactive) added a comment - - edited At the moment it appears that setting innodb_adaptive_hash_index to 0 has resolved the semaphore issue. Since these DB's are crashing in production I am hesitant to install additional debugging and profiling tools. If we experience this again I will install these tools and generate a full stack trace to supply additional details.

          Feel free to add more information when you have it — we'll reopen the issue then

          serg Sergei Golubchik added a comment - Feel free to add more information when you have it — we'll reopen the issue then
          glucz Geza Lucz added a comment - - edited

          I got a very similar looking crash on 10.3.28-MariaDB
          server thread appeared to be up but had to be killed to be able to restart.
          err.txt

          glucz Geza Lucz added a comment - - edited I got a very similar looking crash on 10.3.28-MariaDB server thread appeared to be up but had to be killed to be able to restart. err.txt
          pramod.mahto@mariadb.com Pramod Mahto added a comment -

          Similar crash happen for MariaDB 10.6.9

           
          corrupted size vs. prev_size while consolidating
          230620  6:51:45 [ERROR] mysqld got signal 6 ;
          This could be because you hit a bug. It is also possible that this binary
          or one of the libraries it was linked against is corrupt, improperly built,
          or misconfigured. This error can also be caused by malfunctioning hardware.
           
          To report this bug, see https://mariadb.com/kb/en/reporting-bugs
           
          We will try our best to scrape up some info that will hopefully help
          diagnose the problem, but since we have already crashed, 
          something is definitely wrong and this may fail.
           
          Server version: 10.6.9
          key_buffer_size=67108864
          read_buffer_size=131072
          max_used_connections=212
          max_threads=3002
          thread_count=80
          It is possible that mysqld could use up to 
          key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6676289 K  bytes of memory
          Hope that's ok; if not, decrease some variables in the equation.
           
          Thread pointer: 0x7f352401b5d8
          Attempting backtrace. You can use the following information to find out
          where mysqld died. If you see no messages after this, something went
          terribly wrong...
          stack_bottom = 0x7f353c0ecc18 thread_stack 0x40000
          2023-06-20  7:35:20 0 [Note] mariadbd: Aria engine: starting recovery
          recovered pages: 0% 100% (0.0 seconds); tables to flush: 1 0
           (0.0 seconds); 
          
          

          pramod.mahto@mariadb.com Pramod Mahto added a comment - Similar crash happen for MariaDB 10.6.9   corrupted size vs. prev_size while consolidating 230620 6:51:45 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.6.9 key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=212 max_threads=3002 thread_count=80 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6676289 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x7f352401b5d8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f353c0ecc18 thread_stack 0x40000 2023-06-20 7:35:20 0 [Note] mariadbd: Aria engine: starting recovery recovered pages: 0% 100% (0.0 seconds); tables to flush: 1 0 (0.0 seconds);

          People

            marko Marko Mäkelä
            AdamStark Adam Stark (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.