Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12972

Random and Frequent Segfault (SIG 11) During Runtime

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • 10.1.23, 10.0(EOL), 10.1(EOL), 10.2(EOL)
    • 10.1.26, 10.0.32, 10.2.8
    • Optimizer
    • None
    • CentOS7, Ubuntu 16.04

    Description

      Over the past few weeks I've had three different servers with MariaDB 10.1.20 through 10.1.23 inclusive all crash seemingly at random with a Signal 11.

      No common time of day, no common operation other than operating on data stored in InnoDB tables.

      Attached is the GDB trace from the core dump I managed to obtain as well as configs.

      I can make the core dump available if needs be however it is over 200GB uncompressed, approx 50GB with gzip.

      The overall configuration is a single read/write master with slaves attached. One of the slaves is used for SELECT statements by the same application while the other is a dormant stand by.

      The crash happens on three separate servers of identical hardware. CentOS7 and Ubuntu 16.04 have both been the running OS while the crash has occurred. The crash has only occurred while the server has been operating as the master with slaves attached, as yet we've not seen an active read slave or dormant slave exhibit the same crash. Potential concurrency issue?

      We DO have a vast number of tables in the DB, close to 700k, approx 400k of those active during a normal business day. table_open_cache is currently set at 524288 because we've found if we don't set it/set it low, we get locked up waiting for mysql to look through the table cache for an evict-able table before it gives up and adds a new table_cache_entry anyway.

      Also attached is the my.cnf and the query/query plan for the query in the stack trace for the thread that segfaulted, though we've seen it happen on optimize table statements before now.

      On the current active write master I've set the optimizer_switch to default as I see MRR involved in the thread that died, something we turned on at some point and I note in MariaDB defaults, it's switched off.

      Attachments

        1. _usr_sbin_mysqld.112.crash
          55 kB
        2. example-query.sql
          3 kB
        3. mariadb-debug.out
          329 kB
        4. my.cnf
          5 kB
        5. show-create-index.txt
          17 kB
        6. stack1
          111 kB

        Issue Links

          Activity

            Ninpo Alex Boag-Munroe created issue -
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            Labels need_feedback
            elenst Elena Stepanova made changes -
            Component/s Optimizer [ 10200 ]
            Component/s Storage Engine - XtraDB [ 10135 ]
            Fix Version/s 10.0 [ 16000 ]
            Fix Version/s 10.1 [ 16100 ]
            Fix Version/s 10.2 [ 14601 ]
            Affects Version/s 10.0 [ 16000 ]
            Affects Version/s 10.1 [ 16100 ]
            Affects Version/s 10.2 [ 14601 ]
            Assignee Varun Gupta [ varun ]
            elenst Elena Stepanova made changes -
            Status Open [ 1 ] Confirmed [ 10101 ]
            Ninpo Alex Boag-Munroe made changes -
            Attachment _usr_sbin_mysqld.112.crash [ 43857 ]
            Ninpo Alex Boag-Munroe made changes -
            Attachment show-create-index.txt [ 43858 ]
            elenst Elena Stepanova made changes -
            Comment [ A comment with security level 'Users' was removed. ]
            elenst Elena Stepanova made changes -
            Labels need_feedback
            jplindst Jan Lindström (Inactive) made changes -
            Attachment stack1 [ 43918 ]
            jplindst Jan Lindström (Inactive) made changes -
            monty Michael Widenius made changes -
            Assignee Varun Gupta [ varun ] Michael Widenius [ monty ]
            monty Michael Widenius made changes -
            Status Confirmed [ 10101 ] In Progress [ 3 ]
            monty Michael Widenius made changes -
            issue.field.resolutiondate 2017-08-03 16:29:44.0 2017-08-03 16:29:44.103
            monty Michael Widenius made changes -
            Fix Version/s 10.0.32 [ 22504 ]
            Fix Version/s 10.1.26 [ 22553 ]
            Fix Version/s 10.2.8 [ 22544 ]
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.0 [ 16000 ]
            Fix Version/s 10.1 [ 16100 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            dbart Daniel Bartholomew made changes -
            Fix Version/s 10.0.33 [ 22552 ]
            Fix Version/s 10.0.32 [ 22504 ]
            dbart Daniel Bartholomew made changes -
            Fix Version/s 10.0.32 [ 22504 ]
            Fix Version/s 10.0.33 [ 22552 ]
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 81055 ] MariaDB v4 [ 152258 ]
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 135460

            People

              monty Michael Widenius
              Ninpo Alex Boag-Munroe
              Votes:
              2 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.