Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-31595

Using MariaDB with Nextcloud on docker gives fatal IO error

Details

    Description

      Sorry,I known nothing about databases, but MariaDB has stopped working for a few days, and showing the attached logs, with some alarming messages. I could find no similar logs in the reported issues (I tried to follow the instructions here). It was running great for at least a year, and started crashing without any updates, afaicr. Also attached is my docker-compose.yaml file.

      Attachments

        Issue Links

          Activity

            jds Jay Schieber added a comment -

            Sorry, but I don't understand all of this. I currently have no 'docker-compose.yaml' files that runs successfully. When I ran one using the image 'quay.io/mariadb-foundation/mariadb-devel:11.0', it originally worked, but when what (I think) was the final fix, it crashed with different complaints (see my last log above). Now, that dev image no longer works either. Could I get more specific instructions of the docker-compose setting necessary to get the version to test. Apologies for my ignorance here. If it is relevant, I am not aware of any multiple instances running here.

            jds Jay Schieber added a comment - Sorry, but I don't understand all of this. I currently have no 'docker-compose.yaml' files that runs successfully. When I ran one using the image 'quay.io/mariadb-foundation/mariadb-devel:11.0', it originally worked, but when what (I think) was the final fix, it crashed with different complaints (see my last log above). Now, that dev image no longer works either. Could I get more specific instructions of the docker-compose setting necessary to get the version to test. Apologies for my ignorance here. If it is relevant, I am not aware of any multiple instances running here.
            jds Jay Schieber added a comment -

            I have been without the database (and nextcloud) for nearly a month now, and really stuck about how to fix it. I have the latest of both, but MariaDB still won't run in docker. It currently throws the error below. Is there any way to salvage the situation?

            2023-07-26 15:19:31+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started.
             
            2023-07-26 15:19:31+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
             
            2023-07-26 15:19:31+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started.
             
            2023-07-26 15:19:32+00:00 [Note] [Entrypoint]: MariaDB upgrade information missing, assuming required
             
            2023-07-26 15:19:32+00:00 [Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade) required, but skipped due to $MARIADB_AUTO_UPGRADE setting
             
            2023-07-26 15:19:32 0 [Note] Starting MariaDB 11.0.2-MariaDB-1:11.0.2+maria~ubu2204 source revision 0005f2f06c8e1aea4915887decad67885108a929 as process 1
             
            2023-07-26 15:19:32 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
             
            2023-07-26 15:19:32 0 [Note] InnoDB: Number of transaction pools: 1
             
            2023-07-26 15:19:32 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
             
            2023-07-26 15:19:32 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
             
            2023-07-26 15:19:32 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
             
            2023-07-26 15:19:32 0 [Note] InnoDB: Completed initialization of buffer pool
             
            2023-07-26 15:19:32 0 [ERROR] InnoDB: Inconsistent tablespace ID in file .//undo001
             
            2023-07-26 15:19:32 0 [ERROR] InnoDB: Unable to read first page of file .//undo001
             
            2023-07-26 15:19:32 0 [ERROR] InnoDB: Inconsistent tablespace ID in file .//undo002
             
            2023-07-26 15:19:32 0 [ERROR] InnoDB: Unable to read first page of file .//undo002
             
            2023-07-26 15:19:32 0x7f315470d880  InnoDB: Assertion failure in file ./storage/innobase/srv/srv0start.cc line 804
             
            InnoDB: Failing assertion: !i || prev_id + 1 == space_id
             
            InnoDB: We intentionally generate a memory trap.
             
            InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
             
            InnoDB: If you get repeated assertion failures or crashes, even
             
            InnoDB: immediately after the mariadbd startup, there may be
             
            InnoDB: corruption in the InnoDB tablespace. Please refer to
             
            InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
             
            InnoDB: about forcing recovery.
             
            230726 15:19:32 [ERROR] mysqld got signal 6 ;
             
            This could be because you hit a bug. It is also possible that this binary
             
            or one of the libraries it was linked against is corrupt, improperly built,
             
            or misconfigured. This error can also be caused by malfunctioning hardware.
             
            To report this bug, see https://mariadb.com/kb/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
             
            diagnose the problem, but since we have already crashed, 
             
            something is definitely wrong and this may fail.
             
            Server version: 11.0.2-MariaDB-1:11.0.2+maria~ubu2204 source revision: 0005f2f06c8e1aea4915887decad67885108a929
             
            key_buffer_size=134217728
             
            read_buffer_size=131072
             
            max_used_connections=0
             
            max_threads=153
             
            thread_count=0
             
            It is possible that mysqld could use up to 
             
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468023 K  bytes of memory
             
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x0
             
            Attempting backtrace. You can use the following information to find out
             
            where mysqld died. If you see no messages after this, something went
             
            terribly wrong...
             
            stack_bottom = 0x0 thread_stack 0x49000
             
            Printing to addr2line failed
             
            mariadbd(my_print_stacktrace+0x32)[0x5597984d7a62]
             
            mariadbd(handle_fatal_signal+0x488)[0x559797faae28]
             
            /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f3154a86520]
             
            /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f3154adaa7c]
             
            /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f3154a86476]
             
            /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f3154a6c7f3]
             
            mariadbd(+0x6970f3)[0x559797bbf0f3]
             
            mariadbd(+0x68e841)[0x559797bb6841]
             
            mariadbd(+0x690e34)[0x559797bb8e34]
             
            mariadbd(+0xd7aad9)[0x5597982a2ad9]
             
            mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x86)[0x559797fae0b6]
             
            mariadbd(+0x83d686)[0x559797d65686]
             
            mariadbd(_Z11plugin_initPiPPci+0x91d)[0x559797d6684d]
             
            mariadbd(+0x70bb91)[0x559797c33b91]
             
            mariadbd(_Z11mysqld_mainiPPc+0x424)[0x559797c39324]
             
            /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f3154a6dd90]
             
            /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f3154a6de40]
             
            mariadbd(_start+0x25)[0x559797c2db05]
             
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
             
            information that should help you find out what is causing the crash.
             
            Writing a core file...
             
            Working directory at /var/lib/mysql
             
            Resource Limits:
             
            Limit                     Soft Limit           Hard Limit           Units     
             
            Max cpu time              unlimited            unlimited            seconds   
             
            Max file size             unlimited            unlimited            bytes     
             
            Max data size             unlimited            unlimited            bytes     
             
            Max stack size            8388608              unlimited            bytes     
             
            Max core file size        unlimited            unlimited            bytes     
             
            Max resident set          unlimited            unlimited            bytes     
             
            Max processes             unlimited            unlimited            processes 
             
            Max open files            1048576              1048576              files     
             
            Max locked memory         65536                65536                bytes     
             
            Max address space         unlimited            unlimited            bytes     
             
            Max file locks            unlimited            unlimited            locks     
             
            Max pending signals       127584               127584               signals   
             
            Max msgqueue size         819200               819200               bytes     
             
            Max nice priority         0                    0                    
             
            Max realtime priority     0                    0                    
             
            Max realtime timeout      unlimited            unlimited            us        
             
            Core pattern: |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E
             
            Kernel version: Linux version 5.15.0-76-generic (buildd@lcy02-amd64-028) (gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023
             
            Fatal signal 11 while backtracing
            

            jds Jay Schieber added a comment - I have been without the database (and nextcloud) for nearly a month now, and really stuck about how to fix it. I have the latest of both, but MariaDB still won't run in docker. It currently throws the error below. Is there any way to salvage the situation? 2023 - 07 - 26 15 : 19 : 31 + 00 : 00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1 : 11.0 . 2 +maria~ubu2204 started.   2023 - 07 - 26 15 : 19 : 31 + 00 : 00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'   2023 - 07 - 26 15 : 19 : 31 + 00 : 00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1 : 11.0 . 2 +maria~ubu2204 started.   2023 - 07 - 26 15 : 19 : 32 + 00 : 00 [Note] [Entrypoint]: MariaDB upgrade information missing, assuming required   2023 - 07 - 26 15 : 19 : 32 + 00 : 00 [Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade) required, but skipped due to $MARIADB_AUTO_UPGRADE setting   2023 - 07 - 26 15 : 19 : 32 0 [Note] Starting MariaDB 11.0 . 2 -MariaDB- 1 : 11.0 . 2 +maria~ubu2204 source revision 0005f2f06c8e1aea4915887decad67885108a929 as process 1   2023 - 07 - 26 15 : 19 : 32 0 [Note] InnoDB: Compressed tables use zlib 1.2 . 11   2023 - 07 - 26 15 : 19 : 32 0 [Note] InnoDB: Number of transaction pools: 1   2023 - 07 - 26 15 : 19 : 32 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions   2023 - 07 - 26 15 : 19 : 32 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)   2023 - 07 - 26 15 : 19 : 32 0 [Note] InnoDB: Initializing buffer pool, total size = 128 .000MiB, chunk size = 2 .000MiB   2023 - 07 - 26 15 : 19 : 32 0 [Note] InnoDB: Completed initialization of buffer pool   2023 - 07 - 26 15 : 19 : 32 0 [ERROR] InnoDB: Inconsistent tablespace ID in file . //undo001   2023 - 07 - 26 15 : 19 : 32 0 [ERROR] InnoDB: Unable to read first page of file . //undo001   2023 - 07 - 26 15 : 19 : 32 0 [ERROR] InnoDB: Inconsistent tablespace ID in file . //undo002   2023 - 07 - 26 15 : 19 : 32 0 [ERROR] InnoDB: Unable to read first page of file . //undo002   2023 - 07 - 26 15 : 19 : 32 0x7f315470d880 InnoDB: Assertion failure in file ./storage/innobase/srv/srv0start.cc line 804   InnoDB: Failing assertion: !i || prev_id + 1 == space_id   InnoDB: We intentionally generate a memory trap.   InnoDB: Submit a detailed bug report to https: //jira.mariadb.org/   InnoDB: If you get repeated assertion failures or crashes, even   InnoDB: immediately after the mariadbd startup, there may be   InnoDB: corruption in the InnoDB tablespace. Please refer to   InnoDB: https: //mariadb.com/kb/en/library/innodb-recovery-modes/   InnoDB: about forcing recovery.   230726 15 : 19 : 32 [ERROR] mysqld got signal 6 ;   This could be because you hit a bug. It is also possible that this binary   or one of the libraries it was linked against is corrupt, improperly built,   or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https: //mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help   diagnose the problem, but since we have already crashed,   something is definitely wrong and this may fail.   Server version: 11.0 . 2 -MariaDB- 1 : 11.0 . 2 +maria~ubu2204 source revision: 0005f2f06c8e1aea4915887decad67885108a929   key_buffer_size= 134217728   read_buffer_size= 131072   max_used_connections= 0   max_threads= 153   thread_count= 0   It is possible that mysqld could use up to   key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468023 K bytes of memory   Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x0   Attempting backtrace. You can use the following information to find out   where mysqld died. If you see no messages after this , something went   terribly wrong...   stack_bottom = 0x0 thread_stack 0x49000   Printing to addr2line failed   mariadbd(my_print_stacktrace+ 0x32 )[ 0x5597984d7a62 ]   mariadbd(handle_fatal_signal+ 0x488 )[ 0x559797faae28 ]   /lib/x86_64-linux-gnu/libc.so. 6 (+ 0x42520 )[ 0x7f3154a86520 ]   /lib/x86_64-linux-gnu/libc.so. 6 (pthread_kill+ 0x12c )[ 0x7f3154adaa7c ]   /lib/x86_64-linux-gnu/libc.so. 6 (raise+ 0x16 )[ 0x7f3154a86476 ]   /lib/x86_64-linux-gnu/libc.so. 6 (abort+ 0xd3 )[ 0x7f3154a6c7f3 ]   mariadbd(+ 0x6970f3 )[ 0x559797bbf0f3 ]   mariadbd(+ 0x68e841 )[ 0x559797bb6841 ]   mariadbd(+ 0x690e34 )[ 0x559797bb8e34 ]   mariadbd(+ 0xd7aad9 )[ 0x5597982a2ad9 ]   mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+ 0x86 )[ 0x559797fae0b6 ]   mariadbd(+ 0x83d686 )[ 0x559797d65686 ]   mariadbd(_Z11plugin_initPiPPci+ 0x91d )[ 0x559797d6684d ]   mariadbd(+ 0x70bb91 )[ 0x559797c33b91 ]   mariadbd(_Z11mysqld_mainiPPc+ 0x424 )[ 0x559797c39324 ]   /lib/x86_64-linux-gnu/libc.so. 6 (+ 0x29d90 )[ 0x7f3154a6dd90 ]   /lib/x86_64-linux-gnu/libc.so. 6 (__libc_start_main+ 0x80 )[ 0x7f3154a6de40 ]   mariadbd(_start+ 0x25 )[ 0x559797c2db05 ]   The manual page at https: //mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains   information that should help you find out what is causing the crash.   Writing a core file...   Working directory at /var/lib/mysql   Resource Limits:   Limit Soft Limit Hard Limit Units   Max cpu time unlimited unlimited seconds   Max file size unlimited unlimited bytes   Max data size unlimited unlimited bytes   Max stack size 8388608 unlimited bytes   Max core file size unlimited unlimited bytes   Max resident set unlimited unlimited bytes   Max processes unlimited unlimited processes   Max open files 1048576 1048576 files   Max locked memory 65536 65536 bytes   Max address space unlimited unlimited bytes   Max file locks unlimited unlimited locks   Max pending signals 127584 127584 signals   Max msgqueue size 819200 819200 bytes   Max nice priority 0 0   Max realtime priority 0 0   Max realtime timeout unlimited unlimited us   Core pattern: |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E   Kernel version: Linux version 5.15 . 0 - 76 -generic (buildd @lcy02 -amd64- 028 ) (gcc (Ubuntu 11.3 . 0 -1ubuntu1~ 22.04 . 1 ) 11.3 . 0 , GNU ld (GNU Binutils for Ubuntu) 2.38 ) # 83 -Ubuntu SMP Thu Jun 15 19 : 16 : 32 UTC 2023   Fatal signal 11 while backtracing

            jds, the intentional crash following the message

            mariadb-11.0.2

            InnoDB: Failing assertion: !i || prev_id + 1 == space_id
            

            should have been fixed in MDEV-31487, which was included in the MariaDB Server 11.0.3 release.

            If you are unable to start up the database after upgrading to 11.0.3, you could try setting innodb_force_recovery=5 to prevent any undo logs from being accessed. That should allow you to use a tool like mariadb-dump to create a logical backup of the database contents. This will be using the transaction isolation level READ UNCOMMITTED. Then you could load the dump into a newly initialized database.

            marko Marko Mäkelä added a comment - jds , the intentional crash following the message mariadb-11.0.2 InnoDB: Failing assertion: !i || prev_id + 1 == space_id should have been fixed in MDEV-31487 , which was included in the MariaDB Server 11.0.3 release. If you are unable to start up the database after upgrading to 11.0.3, you could try setting innodb_force_recovery=5 to prevent any undo logs from being accessed. That should allow you to use a tool like mariadb-dump to create a logical backup of the database contents. This will be using the transaction isolation level READ UNCOMMITTED . Then you could load the dump into a newly initialized database.

            Another cause for that assertion failure is MDEV-31851.

            marko Marko Mäkelä added a comment - Another cause for that assertion failure is MDEV-31851 .

            Is this reproducible with a more recent version of MariaDB Server?

            marko Marko Mäkelä added a comment - Is this reproducible with a more recent version of MariaDB Server?

            People

              Unassigned Unassigned
              jds Jay Schieber
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.