Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-34642

Shutdown take indefinitely when /tmp is full.

Details

    • Bug
    • Status: Stalled (View Workflow)
    • Critical
    • Resolution: Unresolved
    • 10.6.16
    • 10.5, 10.6
    • None

    Description

      The customer rebooted the VM due to /tmp reached 100%. But MariaDB server shutdown took a very long time(refer to the error log attached) . It doesn't have any info about when the shutdown completed. There is no info InnoDB: Starting Shutdown and [Note] /usr/sbin/mariadbd: Shutdown complete was logged in the error log. 

      2024-07-18 10:33:06 553 [ERROR] mariadbd: Disk full (/mariadb/tmp/#sql-temptable-b73-229-2349f1.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
      		 
      2024-07-18 10:33:07 973590463 [ERROR] mariadbd: Disk full (/mariadb/tmp/#sql-temptable-b73-3a07cfbf-2700d.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
      		 
      2024-07-18 10:33:07 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
      		 
       
      		 
      2024-07-18 10:33:08 1 [Note] Stopping ack receiver thread
      		 
      2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973538290 (user : 'monuser') did not exit
      		 
      2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973327102 (user : 'monuser') did not exit
      		 
       
      		 
      2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973315593 (user : 'monuser') did not exit
      		 
       
      		 
      2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973260285 (user : 'monuser') did not exit
      		 
       
      		 
      2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973250296 (user : 'casadb_user') did not exit
      		 
       
      		 
      2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973230937 (user : 'casadb_user') did not exit
      		 
      2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973230894 (user : 'casadb_user') did not exit
      		 
       
      		 
      .......
      		 
      .......
      		 
       
      		 
      2024-07-18 10:34:02 973214855 [ERROR] mariadbd: Disk full (/mariadb/tmp/#sql-temptable-b73-3a021487-3426d.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
      		 
      2024-07-18 10:34:02 973214855 [Warning] Sort aborted, host: 10.219.245.102, user: casadb_user, thread: 973214855, query: Select func.FUNCTION_NAME as function, txn.product_type as productType, PRODUCT_ID as productId, REQ_DATE as requestDate,REQ_DATE_TIME as requestDateTimeStr, req_time as requestTime, COUNTRYCODE as country, DOC_STAGE as stage, txn.ID as txnRefNumber, txn.sub_stage as subStage, channel as channel, customer_name as customerName, coalesce(nullif(txn.cif_id,''), txn.customerid) as cifId, branch_code as branchCode, originals as originals, bu as bu, txn.priority  as priority, branch_name as branchName, txn.status as status, sub_status as subStatus, assignee as assignee, txn.sub_product_type  as subProductType , txn.CASE_NUM as caseNum, txn.DOC_REF_NUM as docRefNum, IF (st.stage_name LIKE '%RPA%', 'RPA', st.stage_name) as stageDesc, txn.skill as skill, txn.segment as segment , txn.biz_field1 as bizfield1, txn.biz_field2 as bizfield2, txn.biz_field3 as bizfield3, txn.biz_field4 as bizfield4,txn.acted_by as actedBy, txn.modified_date as 
      2024-07-18 10:48:42 937893309 [Note] Stop semi-sync binlog_dump to slave (server_id: 1)
      		 
      2024-07-18 10:48:42 937893309 [Warning] Aborted connection 937893309 to db: 'unconnected' user: 'repuser' host: '10.219.235.178' (Got an error writing communication packets)

      Attachments

        1. error_log_mariadb.log
          1.51 MB
          Pandikrishnan Gurusamy

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MDEV-33813 ERROR 1021 (HY000): Disk full (./org/test1.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-33877 Disk full with transactional Aria table can lead to a hang

          • Major - Major loss of function.
          • Open

          Activity

            Ascending order - Click to sort in descending order
            pandi.gurusamy Pandikrishnan Gurusamy created issue -
            pandi.gurusamy Pandikrishnan Gurusamy made changes -
            Field Original Value New Value
            Description The customer rebooted the VM due to /tmp reached 100%. But MariaDB server shutdown took a very long time(refer to the error log attached). There is no info {{InnoDB: Starting Shutdown}} was logged in the error log.

            {code}
            2024-07-18 10:33:06 553 [ERROR] mariadbd: Disk full (/mariadb/tmp/#sql-temptable-b73-229-2349f1.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
            2024-07-18 10:33:07 973590463 [ERROR] mariadbd: Disk full (/mariadb/tmp/#sql-temptable-b73-3a07cfbf-2700d.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
            2024-07-18 10:33:07 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown

            2024-07-18 10:33:08 1 [Note] Stopping ack receiver thread
            2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973538290 (user : 'monuser') did not exit
            2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973327102 (user : 'monuser') did not exit

            2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973315593 (user : 'monuser') did not exit

            2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973260285 (user : 'monuser') did not exit

            2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973250296 (user : 'casadb_user') did not exit

            2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973230937 (user : 'casadb_user') did not exit
            2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973230894 (user : 'casadb_user') did not exit

            .......
            .......

            2024-07-18 10:34:02 973214855 [ERROR] mariadbd: Disk full (/mariadb/tmp/#sql-temptable-b73-3a021487-3426d.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
            2024-07-18 10:34:02 973214855 [Warning] Sort aborted, host: 10.219.245.102, user: casadb_user, thread: 973214855, query: Select func.FUNCTION_NAME as function, txn.product_type as productType, PRODUCT_ID as productId, REQ_DATE as requestDate,REQ_DATE_TIME as requestDateTimeStr, req_time as requestTime, COUNTRYCODE as country, DOC_STAGE as stage, txn.ID as txnRefNumber, txn.sub_stage as subStage, channel as channel, customer_name as customerName, coalesce(nullif(txn.cif_id,''), txn.customerid) as cifId, branch_code as branchCode, originals as originals, bu as bu, txn.priority as priority, branch_name as branchName, txn.status as status, sub_status as subStatus, assignee as assignee, txn.sub_product_type as subProductType , txn.CASE_NUM as caseNum, txn.DOC_REF_NUM as docRefNum, IF (st.stage_name LIKE '%RPA%', 'RPA', st.stage_name) as stageDesc, txn.skill as skill, txn.segment as segment , txn.biz_field1 as bizfield1, txn.biz_field2 as bizfield2, txn.biz_field3 as bizfield3, txn.biz_field4 as bizfield4,txn.acted_by as actedBy, txn.modified_date as
            2024-07-18 10:48:42 937893309 [Note] Stop semi-sync binlog_dump to slave (server_id: 1)
            2024-07-18 10:48:42 937893309 [Warning] Aborted connection 937893309 to db: 'unconnected' user: 'repuser' host: '10.219.235.178' (Got an error writing communication packets)

            {code}             		The customer rebooted the VM due to /tmp reached 100%. But MariaDB server shutdown took a very long time(refer to the error log attached) . It doesn't have any info about when the shutdown completed. There is no info {{InnoDB: Starting Shutdown}} and {{[Note] /usr/sbin/mariadbd: Shutdown complete}} was logged in the error log.

            {code}
            2024-07-18 10:33:06 553 [ERROR] mariadbd: Disk full (/mariadb/tmp/#sql-temptable-b73-229-2349f1.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
            2024-07-18 10:33:07 973590463 [ERROR] mariadbd: Disk full (/mariadb/tmp/#sql-temptable-b73-3a07cfbf-2700d.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
            2024-07-18 10:33:07 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown

            2024-07-18 10:33:08 1 [Note] Stopping ack receiver thread
            2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973538290 (user : 'monuser') did not exit
            2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973327102 (user : 'monuser') did not exit

            2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973315593 (user : 'monuser') did not exit

            2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973260285 (user : 'monuser') did not exit

            2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973250296 (user : 'casadb_user') did not exit

            2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973230937 (user : 'casadb_user') did not exit
            2024-07-18 10:33:28 0 [Warning] /usr/sbin/mariadbd: Thread 973230894 (user : 'casadb_user') did not exit

            .......
            .......

            2024-07-18 10:34:02 973214855 [ERROR] mariadbd: Disk full (/mariadb/tmp/#sql-temptable-b73-3a021487-3426d.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
            2024-07-18 10:34:02 973214855 [Warning] Sort aborted, host: 10.219.245.102, user: casadb_user, thread: 973214855, query: Select func.FUNCTION_NAME as function, txn.product_type as productType, PRODUCT_ID as productId, REQ_DATE as requestDate,REQ_DATE_TIME as requestDateTimeStr, req_time as requestTime, COUNTRYCODE as country, DOC_STAGE as stage, txn.ID as txnRefNumber, txn.sub_stage as subStage, channel as channel, customer_name as customerName, coalesce(nullif(txn.cif_id,''), txn.customerid) as cifId, branch_code as branchCode, originals as originals, bu as bu, txn.priority as priority, branch_name as branchName, txn.status as status, sub_status as subStatus, assignee as assignee, txn.sub_product_type as subProductType , txn.CASE_NUM as caseNum, txn.DOC_REF_NUM as docRefNum, IF (st.stage_name LIKE '%RPA%', 'RPA', st.stage_name) as stageDesc, txn.skill as skill, txn.segment as segment , txn.biz_field1 as bizfield1, txn.biz_field2 as bizfield2, txn.biz_field3 as bizfield3, txn.biz_field4 as bizfield4,txn.acted_by as actedBy, txn.modified_date as
            2024-07-18 10:48:42 937893309 [Note] Stop semi-sync binlog_dump to slave (server_id: 1)
            2024-07-18 10:48:42 937893309 [Warning] Aborted connection 937893309 to db: 'unconnected' user: 'repuser' host: '10.219.235.178' (Got an error writing communication packets)

            {code}
            pandi.gurusamy Pandikrishnan Gurusamy made changes -
            Affects Version/s 10.6.16 [ 29014 ]
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 203678
            Zendesk active tickets 203678
            danblack Daniel Black made changes -
            Assignee Daniel Black [ danblack ]
            danblack Daniel Black made changes -
            Fix Version/s 10.5 [ 23123 ]
            Fix Version/s 10.6 [ 24028 ]
            danblack Daniel Black made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            danblack Daniel Black added a comment -

            wlad can I get a review on https://github.com/MariaDB/server/pull/3419 please.

            danblack Daniel Black added a comment - wlad can I get a review on https://github.com/MariaDB/server/pull/3419 please.
            danblack Daniel Black made changes -
            Assignee Daniel Black [ danblack ] Vladislav Vaintroub [ wlad ]
            Status In Progress [ 3 ] In Review [ 10002 ]
            danblack Daniel Black made changes -
            danblack Daniel Black added a comment -

            probably duplicates MDEV-33813 that is now fixed.

            This adds a test case.

            danblack Daniel Black added a comment - probably duplicates MDEV-33813 that is now fixed. This adds a test case.
            danblack Daniel Black made changes -
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk active tickets 203678
            wlad Vladislav Vaintroub made changes -
            Assignee Vladislav Vaintroub [ wlad ] Daniel Black [ danblack ]
            wlad Vladislav Vaintroub added a comment - - edited

            I reviewed and said "no" to adding a test case, that would take a minute doing nothing.
            https://github.com/MariaDB/server/pull/3419#issuecomment-2249919951

            Either the test should be added to version where it is fixed, or fix should be downported, I think

            wlad Vladislav Vaintroub added a comment - - edited I reviewed and said "no" to adding a test case, that would take a minute doing nothing. https://github.com/MariaDB/server/pull/3419#issuecomment-2249919951 Either the test should be added to version where it is fixed, or fix should be downported, I think
            julien.fritsch Julien Fritsch made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            serg Sergei Golubchik made changes -
            Status In Review [ 10002 ] Stalled [ 10000 ]
            vlad.radu Vlad Radu made changes -
            Labels foundation

            People

              danblack Daniel Black
              pandi.gurusamy Pandikrishnan Gurusamy
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.