[MDEV-28132] NCP MariaDB Service down Created: 2022-03-20  Updated: 2022-03-26  Resolved: 2022-03-23

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.3.31
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Holger Skarba Assignee: Unassigned
Resolution: Won't Fix Votes: 0
Labels: None

Attachments: PNG File NCP Status.png    

 Description   

Hi there,
since a week i have this status of my NCP (NextcloudPi) on Raspberry PI 3B like in the screenshot.

I don't know what happened. Since a week I'm trying to get it work. I hope my data I give you is enough to help me. If not feel free to answer.

When I try to start mysql or mariadb I get:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)

error.log of mysql gives:

2022-03-19 14:46:23 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
2022-03-19 14:46:23 0 [Note] InnoDB: Using Linux native AIO
2022-03-19 14:46:23 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-03-19 14:46:23 0 [Note] InnoDB: Uses event mutexes
2022-03-19 14:46:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-03-19 14:46:23 0 [Note] InnoDB: Number of pools: 1
2022-03-19 14:46:23 0 [Note] InnoDB: Using generic crc32 instructions
2022-03-19 14:46:24 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2022-03-19 14:46:24 0 [Note] InnoDB: Completed initialization of buffer pool
2022-03-19 14:46:24 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2022-03-19 14:46:24 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=88502137593
2022-03-19 14:46:24 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2022-03-19 14:46:24 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2022-03-19 14:46:24 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-03-19 14:46:24 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-03-19 14:46:24 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-03-19 14:46:24 0 [Note] InnoDB: 10.3.31 started; log sequence number 88502137665; transaction id 55565404
2022-03-19 14:46:24 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-03-19 14:46:24 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-03-19 14:46:24 0 [Note] Recovering after a crash using tc.log
2022-03-19 14:46:24 0 [Note] Starting crash recovery...
2022-03-19 14:46:24 0 [Note] Crash recovery finished.
2022-03-19 14:46:24 0 [ERROR] InnoDB: Database page corruption on disk or a failed file read of tablespace innodb_system page [page id: space=0, page number=527]. You may have to recover from a backup.
2022-03-19 14:46:24 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):

InnoDB: End of page dump
2022-03-19 14:46:24 0 [Note] InnoDB: Uncompressed page, stored checksum in field1 1509796343, calculated checksums for field1: crc32 667289808, innodb 947024553, page type 17855 == INDEX.none 3735928559, stored checksum in field2 1509796343, calculated checksums for field2: crc32 667289808, innodb 1044635519, none 3735928559, page LSN 10 2871273055, low 4 bytes of LSN at page end 2871273055, page number (if stored to page already) 527, space id (if created with >= MySQL-4.1.1 and stored already) 0
2022-03-19 14:46:24 0 [Note] InnoDB: Page may be an index page where index id is 18446744069414584320
2022-03-19 14:46:24 0 [Note] InnoDB: It is also possible that your operating system has corrupted its own file cache and rebooting your computer removes the error. If the corrupt page is an index page. You can also try to fix the corruption by dumping, dropping, and reimporting the corrupt table. You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
2022-03-19 14:46:24 0 [ERROR] [FATAL] InnoDB: Aborting because of a corrupt database page.
220319 14:46:24 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.3.31-MariaDB-0+deb10u1
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=6
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 466215 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 6865 6865 processes
Max open files 32768 32768 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 6865 6865 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us t

I can't restore the database either because I get the same issue like before mentioned.

I hope you can help me and save me from reinstalling the whole Nextcloud.



 Comments   
Comment by Vicențiu Ciorbaru [ 2022-03-20 ]

Hi trombonefix!

It seems like you have disk corruption. Ideally you should restore from a backup if you have one.

As this is running on a raspberry pi, I suspect you are storing the database on an SD Card. It is possible the SD Card has gone bad.
Take a backup of the data directory with the server stopped. It should be /var/lib/mysql. Then have a look at https://mariadb.com/kb/en/innodb-recovery-modes/ and try going through them one by one in increasing order and try to dump the database contents into a text file.

For that you can try:
a. mariadb-dump (mysqldump)
b. SELECT * FROM table INTO OUTFILE

Hopefully you can recover your data like this. It's possible that only a few pages were corrupted and you can recover most of it.

Comment by Holger Skarba [ 2022-03-20 ]

Thank you for your effort!
I just don't know if it's worth the effort. I think that the too small SD card (16gb) caused the problem. It only has 5gb of free space left. I observed that the swap file is also quite large. I think it makes more sense to completely reinstall the Raspbian and Nextcloud with a new larger SD card.
When I started the NextcloudPi project, I didn't think that it would eventually even replace my Onedrive. Back then, the 16GB SD card was enough...
Do you understand?

Comment by Sergei Golubchik [ 2022-03-23 ]

okay, let's close it then, as it's appears to be a hardware issue, not a bug as such.

trombonefix, please, comment if you disagree and we'll reopen.

Comment by Holger Skarba [ 2022-03-26 ]

Yes, thanks for the help!

Generated at Thu Feb 08 09:58:19 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.