I ran mysqlcheck -o --databases dewiki and now it works! 
Thank you so much for the suggestion, billynoah from Database Administrators on Stack Exchange!
Thanks, cheers and all the best!
Till
Can you please try to produce a stack trace of the crash?
Unfortunately, our instructions on producing a stack trace do not specifically cover OpenBSD. It could be easiest to start up the 10.6.4 server, then attach GDB to it, and then issue the command that would cause the crash, and finally execute thread apply all backtrace and quit in GDB.
Hi Marko, thank you for your reply!
Yesterday, I noticed that Zend OPcache wasn't working anymore even though it was enabled via opcache.enable=1 and used to work with the older OpenBSD versions. I fixed the OPcache problem by adding zend_extension=/usr/local/lib/php-7.4/modules/opcache.so to /etc/php-7.4.ini.
Today, mysqld didn't even start. I had to add innodb_force_recovery=1 to my.cnf to get it running. I then performed TRUNCATE TABLE objectcache; on all databases and ran mysqlcheck -o --all-databases. Then the mysqld worked again but the error log was still flooded with things like:
[...]
|
2021-11-23 18:03:13 0 [ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=3376, page number=49]
|
[...]
|
2021-11-23 18:03:13 0 [ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=3376, page number=52]
|
[...]
|
And I also had the good old signal 11 again:
211123 18:51:08 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.6.4-MariaDB-log
|
key_buffer_size=16777216
|
read_buffer_size=262144
|
max_used_connections=27
|
max_threads=153
|
thread_count=28
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 137754 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x5430d16d018
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x542b70858c8 thread_stack 0x49000
|
??:0(my_print_stacktrace)[0x5402ff824bb]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x54245332030): REPLACE /* SqlBagOStuff::modifyTableSpecificBlobsForSet */ INTO `objectcache` (keyname,value,exptime) VALUES ('global:resourceloader-filter:minify-css:8:1623de2344aa928628c5a5957af75962','\x9d\x8d\xcd\x8a\xc20^P\xc7_E<)X\x83uq\xd7\xf4\xd0g^Y\x93i^XZ\'a&UQ|w[^Q\xe9\xe2\xcd\xcb\xf0^[\xfe_j\xb7\
|
xe5\xce\xcea\xdd \xfa\x8e\xb8\xbd^]\xc0\xb5Ab\xcf\xbe\xa0#^D\xb4\xbdt^Ks6\x82\Z{q\xa8F\xc5\x99#z\x823\xb5\xf4^N\x9a\xa7[\xcd\xf8^W\xe4\"\xaf^S\x87\xba\xfc\xdd\xeeqY}\x94^N ^D)\x82\x80\'\xe4\xbc\xc8^B\xac d\xe0\xd5\x84\x97\xab\xaf\xd7\xf5^Tj\xd7\xfc\xb9\x9f^?\xeb)*e\x8al\xdd\xd0\x8e2\xeb\xb0\xc9S]0!d\xcb\xf1ESM\x
|
e9\x8avS\xa6\xcbl<U^B\xef\x89C1V\xd8\xcd.]\xee\xf3\xea^A','20211124175108')
|
|
|
Connection ID (thread ID): 32304
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,parti
|
al_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderb
|
y_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
|
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
2021-11-23 18:51:08 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
|
2021-11-23 18:51:08 0 [Note] InnoDB: Number of pools: 1
|
2021-11-23 18:51:08 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
|
2021-11-23 18:51:08 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
|
2021-11-23 18:51:08 0 [Note] InnoDB: Completed initialization of buffer pool
|
2021-11-23 18:51:08 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1259552822921,1259552822921
|
2021-11-23 18:51:08 0 [Note] InnoDB: Transaction 4833000596 was in the XA prepared state.
|
2021-11-23 18:51:08 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo
|
2021-11-23 18:51:08 0 [Note] InnoDB: Trx id counter is 4833000597
|
2021-11-23 18:51:08 0 [Note] InnoDB: Starting final batch to recover 1842 pages from redo log.
|
2021-11-23 18:51:09 0 [Note] InnoDB: Last binlog file './mysql-bin.000353', position 14902961
|
2021-11-23 18:51:09 0 [Note] InnoDB: 128 rollback segments are active.
|
2021-11-23 18:51:09 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
|
2021-11-23 18:51:09 0 [Note] InnoDB: Rollback of non-prepared transactions completed
|
2021-11-23 18:51:09 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
|
2021-11-23 18:51:09 0 [Note] InnoDB: Creating shared tablespace for temporary tables
|
2021-11-23 18:51:09 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
|
2021-11-23 18:51:09 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
|
2021-11-23 18:51:09 0 [Note] InnoDB: 10.6.4 started; log sequence number 1259577892320; transaction id 4833000598
|
2021-11-23 18:51:09 0 [Note] InnoDB: Loading buffer pool(s) from /var/mysql/ib_buffer_pool
|
2021-11-23 18:51:09 0 [Note] Plugin 'FEEDBACK' is disabled.
|
2021-11-23 18:51:09 0 [Note] Recovering after a crash using mysql-bin
|
2021-11-23 18:51:09 0 [Note] Starting table crash recovery...
|
2021-11-23 18:51:09 0 [Note] InnoDB: Starting recovery for XA transactions...
|
2021-11-23 18:51:09 0 [Note] InnoDB: Transaction 4833000596 in prepared state after recovery
|
2021-11-23 18:51:09 0 [Note] InnoDB: Transaction contains changes to 1 rows
|
2021-11-23 18:51:09 0 [Note] InnoDB: 1 transactions in prepared state after recovery
|
2021-11-23 18:51:09 0 [Note] Found 1 prepared transaction(s) in InnoDB
|
2021-11-23 18:51:09 0 [Note] Crash table recovery finished.
|
2021-11-23 18:51:09 0 [Note] InnoDB: Buffer pool(s) load completed at 211123 18:51:09
|
2021-11-23 18:51:09 0 [Note] Server socket created on IP: '127.0.0.1'.
|
2021-11-23 18:51:09 0 [Note] /usr/local/libexec/mariadbd: ready for connections.
|
I tried to get the stack trace but whenever I attached GDB to the mysql process, MariaDB kinda froze without adding info to the log file. Sorry, maybe I did it wrong. Anyway, since it is a production server and I wanted it to work again as soon as possible, I uninstalled MariaDB, deleted /var/mysql, reinstalled MariaDB and imported the databases. So far it works fine and the log file is not getting bigger.
Thanks and cheers,
Till
Now MariaDB won't even run properly. The log file is flooded with entries like this:
211124 13:54:02 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.6.4-MariaDB-log
|
key_buffer_size=16777216
|
read_buffer_size=262144
|
max_used_connections=0
|
max_threads=153
|
thread_count=1
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 137754 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x7dc5420d018
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7dbc0967008 thread_stack 0x49000
|
??:0(my_print_stacktrace)[0x7d9b83944bb]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x0): (null)
|
Connection ID (thread ID): 0
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
|
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
|
|
We think the query pointer is invalid, but we will try to print it anyway.
|
Query:
|
|
|
2021-11-24 13:54:02 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
|
2021-11-24 13:54:02 0 [Note] InnoDB: Number of pools: 1
|
2021-11-24 13:54:02 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
|
2021-11-24 13:54:02 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
|
2021-11-24 13:54:02 0 [Note] InnoDB: Completed initialization of buffer pool
|
2021-11-24 13:54:02 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1631071874,1631071874
|
2021-11-24 13:54:02 0 [Note] InnoDB: Starting final batch to recover 1394 pages from redo log.
|
2021-11-24 13:54:02 0 [Note] InnoDB: Last binlog file './mysql-bin.000008', position 7063240
|
2021-11-24 13:54:02 0 [Note] InnoDB: 128 rollback segments are active.
|
2021-11-24 13:54:03 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
|
2021-11-24 13:54:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables
|
2021-11-24 13:54:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
|
2021-11-24 13:54:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
|
2021-11-24 13:54:03 0 [Note] InnoDB: 10.6.4 started; log sequence number 1647953151; transaction id 1068614
|
2021-11-24 13:54:03 0 [Note] InnoDB: Loading buffer pool(s) from /var/mysql/ib_buffer_pool
|
2021-11-24 13:54:03 0 [Note] Plugin 'FEEDBACK' is disabled.
|
2021-11-24 13:54:03 0 [Note] InnoDB: Buffer pool(s) load completed at 211124 13:54:03
|
2021-11-24 13:54:03 0 [Note] Recovering after a crash using mysql-bin
|
2021-11-24 13:54:03 0 [Note] Starting table crash recovery...
|
2021-11-24 13:54:03 0 [Note] Crash table recovery finished.
|
2021-11-24 13:54:03 0 [Note] Server socket created on IP: '127.0.0.1'.
|
2021-11-24 13:54:03 0 [Note] /usr/local/libexec/mariadbd: ready for connections.
|
Version: '10.6.4-MariaDB-log' socket: '/var/run/mysql/mysql.sock' port: 3306 OpenBSD port: mariadb-server-10.6.4p2v1
|
2021-11-24 13:54:03 0x37d05c35c40 InnoDB: Assertion failure in file /usr/obj/ports/mariadb-10.6.4/mariadb-10.6.4/storage/innobase/btr/btr0cur.cc line 335
|
InnoDB: Failing assertion: page_is_comp(get_block->frame) == page_is_comp(block->frame)
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mariadbd startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
211124 13:54:03 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.6.4-MariaDB-log
|
key_buffer_size=16777216
|
read_buffer_size=262144
|
max_used_connections=0
|
max_threads=153
|
thread_count=1
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 137754 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x37ca7e44018
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x37c69543598 thread_stack 0x49000
|
??:0(my_print_stacktrace)[0x37a4ff154bb]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x0): (null)
|
Connection ID (thread ID): 0
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
|
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
|
|
We think the query pointer is invalid, but we will try to print it anyway.
|
Query:
|
|
|
2021-11-24 13:54:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
|
2021-11-24 13:54:03 0 [Note] InnoDB: Number of pools: 1
|
2021-11-24 13:54:03 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
|
2021-11-24 13:54:03 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
|
2021-11-24 13:54:03 0 [Note] InnoDB: Completed initialization of buffer pool
|
2021-11-24 13:54:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1631071874,1631071874
|
mysql -u root -p gives me an "ERROR 2002 (HY000): Can't connect to local server through socket '/var/run/mysql/mysql.sock' (61)".
I opened mysql-bin.000008 but I can't really go to position 20068278. Notepad++ tells me I can't go further than 7210138.
Any help is greatly appreciated!
Thanks and cheers,
Till
I just upgraded to OpenBSD 7.0-current. Now I'm running mariadb-client-10.6.5v1 and mariadb-server-10.6.5v1. MariaDB keeps crashing and I can't attach GDB to the mysql process since the process ID changes all the time.
I tried this:
mariabackup --backup \
|
--target-dir=/home/myproject/backup/mariadb \
|
--user=root --password=mypassword
|
Segmentation fault (core dumped)
|
It created the file mariadb-backup.core. I performed gdb python -c mariadb-backup.core:
GNU gdb 6.3
|
Copyright 2004 Free Software Foundation, Inc.
|
GDB is free software, covered by the GNU General Public License, and you are
|
welcome to change it and/or distribute copies of it under certain conditions.
|
Type "show copying" to see the conditions.
|
There is absolutely no warranty for GDB. Type "show warranty" for details.
|
This GDB was configured as "amd64-unknown-openbsd7.0"...(no debugging symbols found)
|
|
|
Core was generated by `mariadb-backup'.
|
Program terminated with signal 11, Segmentation fault.
|
#0 0x000002822059551e in ?? ()
|
(gdb) thread apply all backtrace
|
|
|
Thread 1 (process 248700):
|
#0 0x000002822059551e in ?? ()
|
#1 0x000002842ce94c80 in ?? ()
|
#2 0x000002843fcf8320 in ?? ()
|
#3 0x000002842ce94c80 in ?? ()
|
#4 0x0000000000000000 in ?? ()
|
I'm so tired. I'm thinking about moving to a shared host 
I restored all databases from a backup before the update. So far MariaDB is running fine again.