Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-19138

MariaDB becomes unresponsive and often crashes

    Details

      Description

      Since upgrading form 5.5 to 10.2.17, we have started seeing stalls on our servers, which on two occasions have caused the servers to crash after the stalls lasted apex 600 seconds.

      It starts like this and repeats

      2019-03-31 13:33:24 140115816994560 [Warning] InnoDB: A long semaphore wait:
      --Thread 140353328404224 has waited at ha_innodb.cc line 15332 for 241.00 seconds the semaphore:
      Mutex at 0x7fa77a9f06c0, Mutex DICT_SYS created dict0dict.cc:1097, lock var 2
       
      2019-03-31 13:33:24 140115816994560 [Note] InnoDB: A semaphore wait:
      --Thread 140355166058240 has waited at ha_innodb.cc line 16119 for 232.00 seconds the semaphore:
      Mutex at 0x56150b6aa040, Mutex SRV_MONITOR_FILE created srv0start.cc:1726, lock var 2
       
      2019-03-31 13:33:24 140115816994560 [Note] InnoDB: A semaphore wait:
      --Thread 140344273450752 has waited at dict0dict.cc line 1161 for 39.00 seconds the semaphore:
      Mutex at 0x7fa77a9f06c0, Mutex DICT_SYS created dict0dict.cc:1097, lock var 2
       
      2019-03-31 13:33:24 140115816994560 [Note] InnoDB: A semaphore wait:
      --Thread 140349093312256 has waited at dict0dict.cc line 1161 for 34.00 seconds the semaphore:
      Mutex at 0x7fa77a9f06c0, Mutex DICT_SYS created dict0dict.cc:1097, lock var 2
       
      2019-03-31 13:33:24 140115816994560 [Note] InnoDB: A semaphore wait:
      --Thread 140343849826048 has waited at ha_innodb.cc line 16119 for 203.00 seconds the semaphore:
      Mutex at 0x56150b6aa040, Mutex SRV_MONITOR_FILE created srv0start.cc:1726, lock var 2
       
      2019-03-31 13:33:24 140115816994560 [Note] InnoDB: A semaphore wait:
      --Thread 140344697378560 has waited at dict0dict.cc line 1161 for 97.00 seconds the semaphore:
      Mutex at 0x7fa77a9f06c0, Mutex DICT_SYS created dict0dict.cc:1097, lock var 2
       
      2019-03-31 13:33:24 140115816994560 [Note] InnoDB: A semaphore wait:
      --Thread 140348086073088 has waited at dict0dict.cc line 1161 for 108.00 seconds the semaphore:
      Mutex at 0x7fa77a9f06c0, Mutex DICT_SYS created dict0dict.cc:1097, lock var 2
       
      2019-03-31 13:33:24 140115816994560 [Note] InnoDB: A semaphore wait:
      --Thread 140351210829568 has waited at dict0dict.cc line 1161 for 113.00 seconds the semaphore:
      Mutex at 0x7fa77a9f06c0, Mutex DICT_SYS created dict0dict.cc:1097, lock var 2
       
      2019-03-31 13:33:24 140115816994560 [Note] InnoDB: A semaphore wait:
      --Thread 140352057472768 has waited at dict0dict.cc line 1161 for 218.00 seconds the semaphore:
      Mutex at 0x7fa77a9f06c0, Mutex DICT_SYS created dict0dict.cc:1097, lock var 2
       
      2019-03-31 13:33:24 140115816994560 [Note] InnoDB: A semaphore wait:
      --Thread 140337616541440 has waited at dict0dict.cc line 1161 for 69.00 seconds the semaphore:
      Mutex at 0x7fa77a9f06c0, Mutex DICT_SYS created dict0dict.cc:1097, lock var 2
       
      2019-03-31 13:33:24 140115816994560 [Note] InnoDB: A semaphore wait:
      --Thread 140339030267648 has waited at row0ins.cc line 1938 for 130.00 seconds the semaphore:
      S-lock on RW-latch at 0x7f70e33ba500 created in file dict0dict.cc line 1108
      a writer (thread id 140115800209152) has reserved it in mode  exclusive
      number of readers 0, waiters flag 1, lock_word: 0
      Last time read locked in file row0ins.cc line 1938
      Last time write locked in file dict0stats.cc line 2454
      2019-03-31 13:33:24 140115816994560 [Note] InnoDB: A semaphore wait:
      --Thread 140343283046144 has waited at dict0dict.cc line 1161 for 34.00 seconds the semaphore:
      Mutex at 0x7fa77a9f06c0, Mutex DICT_SYS created dict0dict.cc:1097, lock var 2
       
      2019-03-31 13:33:24 140115816994560 [Note] InnoDB: A semaphore wait:
      --Thread 140344977544960 has waited at dict0dict.cc line 1161 for 139.00 seconds the semaphore:
      Mutex at 0x7fa77a9f06c0, Mutex DICT_SYS created dict0dict.cc:1097, lock var 2
      

      Attached are full logs from one of the stalls and one of the stalls that resulted in a crash. The stall also has a 'show engine innodb status' dump.

        Attachments

        1. crash.log
          946 kB
        2. stall.log
          630 kB

          Activity

            People

            • Assignee:
              marko Marko Mäkelä
              Reporter:
              Whitehouse Steve
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: