[MDEV-29654] mysqld got signal 11 Created: 2022-09-28  Updated: 2023-03-16

Status: Open
Project: MariaDB Server
Component/s: None
Affects Version/s: 10.6.10, 10.9.4, 10.11.2
Fix Version/s: None

Type: Bug Priority: Major
Reporter: zEvilz Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Debian 11 amd64



 Description   

The problem started after updating to 10.6.10 from 10.6.9.

error log:

220928  5:59:55 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.6.10-MariaDB-1:10.6.10+maria~deb11-log
key_buffer_size=8589934592
read_buffer_size=131072
max_used_connections=34
max_threads=1002
thread_count=34
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 12647128 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f478c000c58
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f48bcc44d58 thread_stack 0x49000
??:0(my_print_stacktrace)[0x55b5efee0cae]
??:0(handle_fatal_signal)[0x55b5ef9afeb5]
??:0(__restore_rt)[0x7f4be3e7a140]
??:0(main)[0x55b5ef69c79e]
??:0(main)[0x55b5ef69b5a8]
??:0(void std::this_thread::sleep_for<long, std::ratio<1l, 1l> >(std::chrono::duration<long, std::ratio<1l, 1l> > const&))[0x55b5efd6c42b]
??:0(void std::this_thread::sleep_for<long, std::ratio<1l, 1l> >(std::chrono::duration<long, std::ratio<1l, 1l> > const&))[0x55b5efd6ec02]
??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55b5efc9f9a5]
??:0(handler::ha_index_prev(unsigned char*))[0x55b5ef9b69ab]
??:0(report_error(TABLE*, int))[0x55b5ef7ccfb4]
??:0(sub_select(JOIN*, st_join_table*, bool))[0x55b5ef7bf8f5]
??:0(JOIN::exec_inner())[0x55b5ef7ed4e7]
??:0(JOIN::exec())[0x55b5ef7ed885]
??:0(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x55b5ef7eb9f6]
??:0(handle_select(THD*, LEX*, select_result*, unsigned long))[0x55b5ef7ec1af]
??:0(comp_ne_creator(bool))[0x55b5ef774ea1]
??:0(mysql_execute_command(THD*, bool))[0x55b5ef783be2]
??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x55b5ef785187]
??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x55b5ef787a7d]
??:0(do_command(THD*, bool))[0x55b5ef789748]
??:0(do_handle_one_connection(CONNECT*, bool))[0x55b5ef89466f]
??:0(handle_one_connection)[0x55b5ef8949bd]
??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x55b5efbee192]
??:0(start_thread)[0x7f4be3e6eea7]
??:0(clone)[0x7f4be3a75aef]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f478c010970): SELECT SQL_CALC_FOUND_ROWS o.`id_order`, o.`id_customer`, o.`total_paid`, o.`id_currency`, o.`date_upd`, c.`firstname`, c.`lastname`
					FROM `ps_orders` as o
					LEFT JOIN `ps_customer` as c ON (c.`id_customer` = o.`id_customer`)
					WHERE `id_order` > 131328 AND o.id_shop IN (1) 
					ORDER BY `id_order` DESC
					LIMIT 5
 
Connection ID (thread ID): 4474
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             514786               514786               processes 
Max open files            1048576              1048576              files     
Max locked memory         524288               524288               bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       514786               514786               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: core
 
Kernel version: Linux version 5.10.0-18-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.140-1 (2022-09-02)
 
2022-09-28  6:00:00 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-09-28  6:00:00 0 [Note] InnoDB: Number of pools: 1
2022-09-28  6:00:00 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-09-28  6:00:00 0 [Note] InnoDB: Using liburing
2022-09-28  6:00:00 0 [Note] InnoDB: Initializing buffer pool, total size = 4294967296, chunk size = 134217728
2022-09-28  6:00:00 0 [Note] InnoDB: Completed initialization of buffer pool
2022-09-28  6:00:00 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=4850606667198,4850701936883
2022-09-28  6:00:01 0 [Note] InnoDB: Starting final batch to recover 14284 pages from redo log.
2022-09-28  6:00:02 0 [Note] InnoDB: 128 rollback segments are active.
2022-09-28  6:00:02 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2022-09-28  6:00:02 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-09-28  6:00:02 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-09-28  6:00:02 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-09-28  6:00:02 0 [Note] InnoDB: 10.6.10 started; log sequence number 4850776853701; transaction id 3062451519
2022-09-28  6:00:02 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-09-28  6:00:02 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-09-28  6:00:02 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2022-09-28  6:00:02 0 [Note] Server socket created on IP: '127.0.0.1'.
2022-09-28  6:00:02 0 [Note] /usr/sbin/mariadbd: ready for connections.
Version: '10.6.10-MariaDB-1:10.6.10+maria~deb11-log'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2022-09-28  6:00:02 0 [Note] InnoDB: Buffer pool(s) load completed at 220928  6:00:02
2022-09-28  6:00:03 272 [ERROR] mariadbd: Table './livea_s_db/ps_marketplace_product_action' is marked as crashed and should be repaired
2022-09-28  6:00:03 272 [Warning] Checking table:   './livea_s_db/ps_marketplace_product_action'

With every fail one or more random tables marked as crashed and repaired. Requests are too random in fail. I tried repare and optimize all tables in all databases, but it not helped.



 Comments   
Comment by Marko Mäkelä [ 2022-09-28 ]

Can you please install the dbgsym package for 10.6.10 and produce a full stack trace from a debugger? The built-in stack trace output above sub_select() looks corrupted.

Which storage engine are you using for your tables? According to the output, at least one of the tables may be created in MyISAM, or possibly Aria. MyISAM tables are not crash-safe.

Comment by Daniel Black [ 2022-09-28 ]

note the soft limit of `Max core file size` needs to be raised to get a stack trace. (systemd LimitCORE).

Comment by Sergei Golubchik [ 2022-10-30 ]

Do you have more logs with crashes? This stack trace, indeed, look corrupted.
Is it always the same query? Can you provide SHOW CREATE TABLE output for ps_orders and ps_customer?

Comment by zEvilz [ 2022-11-08 ]

I can't test on production server. MariaDB already downgraded to 10.6.9 after problems.

Comment by Daniel Black [ 2022-11-08 ]

I appreciate you don't want the production server to crash. Is SHOW CREATE TABLE output for ps_orders and ps_customer possible?

Is/was it always the same query crashing? Is EXPLAIN FORMAT=JSON query possible, even on the 10.6.9 will give us a possible ability to construct a test case to correct this.

Comment by zEvilz [ 2022-11-09 ]

ps_orders

CREATE TABLE `ps_orders` (
  `id_order` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `reference` varchar(255) DEFAULT NULL,
  `id_shop_group` int(11) unsigned NOT NULL DEFAULT 1,
  `id_shop` int(11) unsigned NOT NULL DEFAULT 1,
  `id_carrier` int(10) unsigned NOT NULL,
  `id_lang` int(10) unsigned NOT NULL,
  `id_customer` int(10) unsigned NOT NULL,
  `id_cart` int(10) unsigned NOT NULL,
  `id_currency` int(10) unsigned NOT NULL,
  `id_address_delivery` int(10) unsigned NOT NULL,
  `id_address_invoice` int(10) unsigned NOT NULL,
  `zoho_desk_id` varchar(45) DEFAULT NULL,
  `zoho_products_id` varchar(255) DEFAULT NULL,
  `zoho_order_products_id` varchar(45) DEFAULT NULL,
  `current_state` int(10) unsigned NOT NULL,
  `secure_key` varchar(32) NOT NULL DEFAULT '-1',
  `payment` varchar(255) NOT NULL,
  `conversion_rate` decimal(13,6) NOT NULL DEFAULT 1.000000,
  `module` varchar(255) DEFAULT NULL,
  `recyclable` tinyint(1) unsigned NOT NULL DEFAULT 0,
  `gift` tinyint(1) unsigned NOT NULL DEFAULT 0,
  `gift_message` text DEFAULT NULL,
  `mobile_theme` tinyint(1) NOT NULL DEFAULT 0,
  `shipping_number` varchar(64) DEFAULT NULL,
  `total_discounts` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `total_discounts_tax_incl` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `total_discounts_tax_excl` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `total_paid` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `total_paid_tax_incl` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `total_paid_tax_excl` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `total_paid_real` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `total_products` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `total_products_wt` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `total_shipping` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `total_shipping_tax_incl` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `total_shipping_tax_excl` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `carrier_tax_rate` decimal(10,3) NOT NULL DEFAULT 0.000,
  `total_wrapping` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `total_wrapping_tax_incl` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `total_wrapping_tax_excl` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `round_mode` tinyint(1) NOT NULL DEFAULT 2,
  `round_type` tinyint(1) NOT NULL DEFAULT 1,
  `invoice_number` int(10) unsigned NOT NULL DEFAULT 0,
  `delivery_number` int(10) unsigned NOT NULL DEFAULT 0,
  `invoice_date` datetime NOT NULL,
  `delivery_date` datetime NOT NULL,
  `valid` int(1) unsigned NOT NULL DEFAULT 0,
  `date_add` datetime NOT NULL,
  `date_upd` datetime NOT NULL,
  `mp_order_id` varchar(255) DEFAULT '',
  `mp_status` int(11) DEFAULT NULL,
  `ups_price_total` decimal(20,2) NOT NULL DEFAULT 0.00,
  `geodb_shipping_price` decimal(20,2) NOT NULL DEFAULT 0.00,
  `total_profit` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `total_profit_percent` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `email_sent` tinyint(4) NOT NULL DEFAULT 0,
  `email_send` tinyint(4) NOT NULL DEFAULT 0,
  `sms_sent2` tinyint(4) NOT NULL DEFAULT 0,
  `sms_sent1` tinyint(4) NOT NULL DEFAULT 0,
  PRIMARY KEY (`id_order`),
  KEY `reference` (`reference`),
  KEY `id_customer` (`id_customer`),
  KEY `id_cart` (`id_cart`),
  KEY `invoice_number` (`invoice_number`),
  KEY `id_carrier` (`id_carrier`),
  KEY `id_lang` (`id_lang`),
  KEY `id_currency` (`id_currency`),
  KEY `id_address_delivery` (`id_address_delivery`),
  KEY `id_address_invoice` (`id_address_invoice`),
  KEY `id_shop_group` (`id_shop_group`),
  KEY `current_state` (`current_state`),
  KEY `id_shop` (`id_shop`),
  KEY `date_add` (`date_add`),
  KEY `mp_order_id` (`mp_order_id`)
) ENGINE=InnoDB AUTO_INCREMENT=338414 DEFAULT CHARSET=utf8mb3 ROW_FORMAT=COMPACT

ps_customer

CREATE TABLE `ps_customer` (
  `id_customer` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `id_shop_group` int(11) unsigned NOT NULL DEFAULT 1,
  `id_shop` int(11) unsigned NOT NULL DEFAULT 1,
  `id_gender` int(10) unsigned NOT NULL,
  `id_default_group` int(10) unsigned NOT NULL DEFAULT 1,
  `id_lang` int(10) unsigned DEFAULT NULL,
  `id_risk` int(10) unsigned NOT NULL DEFAULT 1,
  `zoho_crm_id` varchar(45) DEFAULT NULL,
  `zoho_contact_id` varchar(45) DEFAULT NULL,
  `company` varchar(64) DEFAULT NULL,
  `siret` varchar(14) DEFAULT NULL,
  `ape` varchar(5) DEFAULT NULL,
  `firstname` varchar(32) NOT NULL,
  `lastname` varchar(32) NOT NULL,
  `email` varchar(128) NOT NULL,
  `passwd` varchar(32) NOT NULL,
  `last_passwd_gen` timestamp NOT NULL DEFAULT current_timestamp(),
  `birthday` date DEFAULT NULL,
  `newsletter` tinyint(1) unsigned NOT NULL DEFAULT 0,
  `ip_registration_newsletter` varchar(15) DEFAULT NULL,
  `newsletter_date_add` datetime DEFAULT NULL,
  `optin` tinyint(1) unsigned NOT NULL DEFAULT 0,
  `website` varchar(128) DEFAULT NULL,
  `outstanding_allow_amount` decimal(20,6) NOT NULL DEFAULT 0.000000,
  `show_public_prices` tinyint(1) unsigned NOT NULL DEFAULT 0,
  `max_payment_days` int(10) unsigned NOT NULL DEFAULT 60,
  `secure_key` varchar(32) NOT NULL DEFAULT '-1',
  `note` text DEFAULT NULL,
  `active` tinyint(1) unsigned NOT NULL DEFAULT 0,
  `is_guest` tinyint(1) NOT NULL DEFAULT 0,
  `deleted` tinyint(1) NOT NULL DEFAULT 0,
  `date_add` datetime NOT NULL,
  `date_upd` datetime NOT NULL,
  PRIMARY KEY (`id_customer`),
  KEY `customer_email` (`email`),
  KEY `customer_login` (`email`,`passwd`),
  KEY `id_customer_passwd` (`id_customer`,`passwd`),
  KEY `id_gender` (`id_gender`),
  KEY `id_shop_group` (`id_shop_group`),
  KEY `id_shop` (`id_shop`,`date_add`)
) ENGINE=InnoDB AUTO_INCREMENT=333272 DEFAULT CHARSET=utf8mb3

Comment by zEvilz [ 2022-11-09 ]

All queries was random in error log.

Comment by zEvilz [ 2022-11-09 ]

{
  "query_block": {
    "select_id": 1,
    "table": {
      "table_name": "o",
      "access_type": "ref",
      "possible_keys": ["PRIMARY", "id_shop"],
      "key": "id_shop",
      "key_length": "4",
      "used_key_parts": ["id_shop"],
      "ref": ["const"],
      "rows": 158533,
      "filtered": 49.9998436,
      "attached_condition": "o.id_shop <=> 1 and o.id_order > 131328"
    },
    "table": {
      "table_name": "c",
      "access_type": "eq_ref",
      "possible_keys": ["PRIMARY", "id_customer_passwd"],
      "key": "PRIMARY",
      "key_length": "4",
      "used_key_parts": ["id_customer"],
      "ref": ["livea_db.o.id_customer"],
      "rows": 1,
      "filtered": 100
    }
  }
}

Comment by Marko Mäkelä [ 2022-11-09 ]

zEvilz, please be advised that 10.6.9 is not safe to use because of MDEV-29383 and MDEV-29374. It is possible that your data was already corrupted due to those bugs before you upgraded to 10.6.10 in the first place. At the end of MDEV-29383 you can see an example of a test where preparing a backup or recovering from a crash appeared to be fine, but the database was actually corrupted.

If you need to downgrade from 10.6.10, then you should downgrade to 10.6.8. I would also recommend to run CHECK TABLE on all InnoDB tables, or even better, rebuild the database from a SQL dump.

Comment by zEvilz [ 2022-11-10 ]

@marko, my problem with 10.6.10 could be related to the fact that the base was broken while it was working on 10.6.9?

Is it enough to downgrade to 10.6.8, than create database dumps and restore databases from dumps?

Comment by zEvilz [ 2022-11-15 ]

I have same error on 10.9.4.

221115 13:09:35 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.9.4-MariaDB-1:10.9.4+maria~deb11-log
key_buffer_size=8589934592
read_buffer_size=131072
max_used_connections=57
max_threads=502
thread_count=57
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10522285 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7efa60000f38
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7efad8c31d18 thread_stack 0x49000
??:0(my_print_stacktrace)[0x5648fcede7ee]
??:0(handle_fatal_signal)[0x5648fc9bcd55]
??:0(__restore_rt)[0x7f0948489140]
??:0(main)[0x5648fc65c03d]
??:0(main)[0x5648fc65ac40]
??:0(void std::this_thread::sleep_for<long, std::ratio<1l, 1l> >(std::chrono::duration<long, std::ratio<1l, 1l> > const&))[0x5648fcd63621]
??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x5648fcc995b2]
??:0(handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function))[0x5648fc9c2ecd]
??:0(cp_buffer_from_ref(THD*, TABLE*, st_table_ref*))[0x5648fc7d1d30]
??:0(sub_select(JOIN*, st_join_table*, bool))[0x5648fc7bd213]
??:0(Item::remove_eq_conds(THD*, Item::cond_result*, bool))[0x5648fc7aa63c]
??:0(sub_select(JOIN*, st_join_table*, bool))[0x5648fc7bd228]
??:0(Item::remove_eq_conds(THD*, Item::cond_result*, bool))[0x5648fc7aa63c]
??:0(sub_select(JOIN*, st_join_table*, bool))[0x5648fc7bd228]
??:0(Item::remove_eq_conds(THD*, Item::cond_result*, bool))[0x5648fc7aa63c]
??:0(sub_select(JOIN*, st_join_table*, bool))[0x5648fc7bd228]
??:0(Item::remove_eq_conds(THD*, Item::cond_result*, bool))[0x5648fc7aa63c]
??:0(sub_select(JOIN*, st_join_table*, bool))[0x5648fc7bd2ab]
??:0(JOIN::exec_inner())[0x5648fc7eb487]
??:0(JOIN::exec())[0x5648fc7eb825]
??:0(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5648fc7e999d]
??:0(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5648fc7ea14f]
??:0(comp_ne_creator(bool))[0x5648fc763e61]
??:0(mysql_execute_command(THD*, bool))[0x5648fc772e75]
??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x5648fc774467]
??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x5648fc776d2d]
??:0(do_command(THD*, bool))[0x5648fc778898]
??:0(do_handle_one_connection(CONNECT*, bool))[0x5648fc89544f]
??:0(handle_one_connection)[0x5648fc89579d]
??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x5648fcbdef32]
??:0(start_thread)[0x7f094847dea7]
??:0(clone)[0x7f0948084a2f]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7efa60010960): SELECT 
                SUM(d.product_quantity * IFNULL(pp.quantity,1)) as "quantity",
                CASE
                    WHEN o.mp_order_id LIKE "%|FR" AND (a.id_country = 8 OR a.id_country = 3) THEN "mano btc"
                    WHEN o.mp_order_id LIKE "%|FR_B2B" THEN "mano btb"
                    WHEN o.mp_order_id LIKE "%|FR_MF_B2B" THEN "mano fr mf btb"
                    WHEN o.mp_order_id LIKE "%|FR_MF" THEN "mano fr mf"
                    WHEN o.mp_order_id LIKE "%|GB" THEN "mano gb"
                    WHEN o.mp_order_id LIKE "%|darty" THEN "darty"
                    WHEN o.mp_order_id LIKE "%_VP" THEN "VP"
                    WHEN o.mp_order_id LIKE "%|bricomarche" THEN "bricomarche"
                    WHEN o.mp_order_id LIKE "%|showroomprive" THEN "SP"
                    WHEN o.mp_order_id LIKE "%|DE" THEN "mano de"
                    WHEN o.mp_order_id LIKE "%|ES" THEN "mano es"
                    WHEN o.mp_order_id LIKE "%|IT" THEN "mano it"
                    WHEN o.mp_order_id LIKE "%|ES_MF" THEN "mano es mf"
                    WHEN o.mp_order_id LIKE "%|IT_MF" THEN "mano it mf"
                    WHEN o.mp_order_id LIKE "%|DE_MF" THEN "mano de mf"
 		            WHEN o.mp_order_id LIKE "%|brico" THEN "brico"
                    WHEN o.mp_order_id LIKE "%|leroy" THEN "leroy"
                    WHEN o.mp_order_id LIKE "%|bricomarche" THEN "bricomarche"
                    WHEN o.module LIKE "cdiscount" THEN "cd"
                    WHEN o.module LIKE "priceminister" THEN "rakuten"
                ELSE "livea"
                END as "marketplace"
            
FROM `ps_orders` o
LEFT JOIN `ps_address` `a` ON a.id_address = o.id_address_delivery
LEFT JOIN `ps_order_detail` `d` ON d.id_order = o.id_order
LEFT JOIN `ps_customer` `pcu` ON o.id_customer = pcu.id_customer AND pcu.id_shop = 1
LEFT JOIN `ps_group` `pgr` ON pcu.id_default_group = pgr.id_group
LEFT JOIN `ps_group_lang` `pgrl` ON pcu.id_default_group = pgrl.id_group AND pgrl.id_lang = 2
LEFT JOIN `ps_pack` `pp` ON pp.id_product_pack = d.product_id AND pp.id_product_item = 47242
WHERE (pp.id_product_item = 47242 OR d.product_id = 47242) AND (o.date_add BETWEEN '2021-11-15 00:00:00' AND '2022-11-16 00:00:00') AND (o.current_state IN (2,3,4,5,9,17,18,22,23)) AND (pgrl.name <> 'AKAAZ')
GROUP BY marketplace
 
Connection ID (thread ID): 531641
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             514786               514786               processes 
Max open files            1048576              1048576              files     
Max locked memory         524288               524288               bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       514786               514786               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: core
 
Kernel version: Linux version 5.10.0-19-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.149-2 (2022-10-21)
 
2022-11-15 13:09:41 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-11-15 13:09:41 0 [Note] InnoDB: Number of transaction pools: 1
2022-11-15 13:09:41 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-11-15 13:09:41 0 [Note] InnoDB: Using liburing
2022-11-15 13:09:41 0 [Note] InnoDB: Initializing buffer pool, total size = 48.000GiB, chunk size = 0.750GiB
2022-11-15 13:09:41 0 [Note] InnoDB: Completed initialization of buffer pool
2022-11-15 13:09:41 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
2022-11-15 13:09:41 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=65282963377
2022-11-15 13:09:56 0 [Note] InnoDB: Read redo log up to LSN=67430008820
2022-11-15 13:10:11 0 [Note] InnoDB: Read redo log up to LSN=69340130806
2022-11-15 13:10:24 0 [Note] InnoDB: Starting final batch to recover 134010 pages from redo log.
2022-11-15 13:10:26 0 [Note] InnoDB: To recover: 115780 pages from log
2022-11-15 13:10:31 0 [Note] InnoDB: 128 rollback segments are active.
2022-11-15 13:10:31 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2022-11-15 13:10:31 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2022-11-15 13:10:31 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2022-11-15 13:10:31 0 [Note] InnoDB: log sequence number 71018658886; transaction id 22693525
2022-11-15 13:10:31 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-11-15 13:10:31 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-11-15 13:10:31 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2022-11-15 13:10:31 0 [Note] Server socket created on IP: '127.0.0.1'.
2022-11-15 13:10:31 0 [Note] /usr/sbin/mariadbd: ready for connections.
Version: '10.9.4-MariaDB-1:10.9.4+maria~deb11-log'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2022-11-15 13:10:32 94 [ERROR] mariadbd: Table './livea_d_db/ps_ba_importer_config' is marked as crashed and should be repaired
2022-11-15 13:10:32 94 [Warning] Checking table:   './livea_d_db/ps_ba_importer_config'
2022-11-15 13:10:32 95 [ERROR] mariadbd: Table './livea_d_db/ps_marketplace_product_action' is marked as crashed and should be repaired
2022-11-15 13:10:32 95 [Warning] Checking table:   './livea_d_db/ps_marketplace_product_action'
2022-11-15 13:10:32 112 [ERROR] mariadbd: Table './livea_d_db/ps_ba_cronjobs_importer' is marked as crashed and should be repaired
2022-11-15 13:10:32 112 [Warning] Checking table:   './livea_d_db/ps_ba_cronjobs_importer'
2022-11-15 13:10:32 157 [ERROR] mariadbd: Table './livea_db/ps_ba_importer_config' is marked as crashed and should be repaired
2022-11-15 13:10:32 157 [Warning] Checking table:   './livea_db/ps_ba_importer_config'
2022-11-15 13:10:32 158 [ERROR] mariadbd: Table './livea_db/ps_marketplace_product_action' is marked as crashed and should be repaired
2022-11-15 13:10:32 158 [Warning] Checking table:   './livea_db/ps_marketplace_product_action'
2022-11-15 13:10:32 165 [ERROR] mariadbd: Table './livea_db/ps_cdiscount_product_option' is marked as crashed and should be repaired
2022-11-15 13:10:32 165 [Warning] Checking table:   './livea_db/ps_cdiscount_product_option'
2022-11-15 13:10:32 175 [ERROR] mariadbd: Table './livea_db/ps_ba_cronjobs_importer' is marked as crashed and should be repaired
2022-11-15 13:10:32 175 [Warning] Checking table:   './livea_db/ps_ba_cronjobs_importer'
2022-11-15 13:10:32 181 [ERROR] mariadbd: Table './livea_db/ps_cdiscount_orders' is marked as crashed and should be repaired
2022-11-15 13:10:32 181 [Warning] Checking table:   './livea_db/ps_cdiscount_orders'
2022-11-15 13:10:35 0 [Note] InnoDB: Buffer pool(s) load completed at 221115 13:10:35

But this is the first fail in 2 days. It was much more frequent on 10.6.10. I am created database dumps on 10.6.9, completelly removed mariadb packages and directories. Then installed 10.9.4 and restored databases from dumps.

Comment by zEvilz [ 2022-11-15 ]

One more:

221115 14:21:33 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.9.4-MariaDB-1:10.9.4+maria~deb11-log
key_buffer_size=8589934592
read_buffer_size=131072
max_used_connections=35
max_threads=502
thread_count=35
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10522285 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7fc8cc000c58
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fc99c1a5d18 thread_stack 0x49000
??:0(my_print_stacktrace)[0x5651f022f7ee]
??:0(handle_fatal_signal)[0x5651efd0dd55]
??:0(__restore_rt)[0x7fd80cc01140]
??:0(main)[0x5651ef9ad03d]
??:0(main)[0x5651ef9abc40]
??:0(void std::this_thread::sleep_for<long, std::ratio<1l, 1l> >(std::chrono::duration<long, std::ratio<1l, 1l> > const&))[0x5651f00b5f3d]
??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x5651effea5b2]
??:0(handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function))[0x5651efd13ecd]
??:0(join_read_key2(THD*, st_join_table*, TABLE*, st_table_ref*))[0x5651efb23060]
??:0(sub_select(JOIN*, st_join_table*, bool))[0x5651efb0e213]
??:0(Item::remove_eq_conds(THD*, Item::cond_result*, bool))[0x5651efafb63c]
??:0(sub_select(JOIN*, st_join_table*, bool))[0x5651efb0e2ab]
??:0(JOIN::exec_inner())[0x5651efb3c487]
??:0(JOIN::exec())[0x5651efb3c825]
??:0(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5651efb3a99d]
??:0(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5651efb3b14f]
??:0(comp_ne_creator(bool))[0x5651efab4e61]
??:0(mysql_execute_command(THD*, bool))[0x5651efac3e75]
??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x5651efac5467]
??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x5651efac7d2d]
??:0(do_command(THD*, bool))[0x5651efac9898]
??:0(do_handle_one_connection(CONNECT*, bool))[0x5651efbe644f]
??:0(handle_one_connection)[0x5651efbe679d]
??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x5651eff2ff32]
??:0(start_thread)[0x7fd80cbf5ea7]
??:0(clone)[0x7fd80c7fca2f]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7fc8cc010960): SELECT SQL_CALC_FOUND_ROWS o.`id_order`, o.`id_customer`, o.`total_paid`, o.`id_currency`, o.`date_upd`, c.`firstname`, c.`lastname`
					FROM `ps_orders` as o
					LEFT JOIN `ps_customer` as c ON (c.`id_customer` = o.`id_customer`)
					WHERE `id_order` > 120900 AND o.id_shop IN (1) 
					ORDER BY `id_order` DESC
					LIMIT 5
 
Connection ID (thread ID): 16974
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             514786               514786               processes 
Max open files            1048576              1048576              files     
Max locked memory         524288               524288               bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       514786               514786               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: core
 
Kernel version: Linux version 5.10.0-19-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.149-2 (2022-10-21)
 
2022-11-15 14:21:39 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-11-15 14:21:39 0 [Note] InnoDB: Number of transaction pools: 1
2022-11-15 14:21:39 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-11-15 14:21:39 0 [Note] InnoDB: Using liburing
2022-11-15 14:21:39 0 [Note] InnoDB: Initializing buffer pool, total size = 48.000GiB, chunk size = 0.750GiB
2022-11-15 14:21:39 0 [Note] InnoDB: Completed initialization of buffer pool
2022-11-15 14:21:39 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
2022-11-15 14:21:39 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=71392260924
2022-11-15 14:21:48 0 [Note] InnoDB: Starting final batch to recover 47803 pages from redo log.
2022-11-15 14:21:49 0 [Note] InnoDB: 128 rollback segments are active.
2022-11-15 14:21:49 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2022-11-15 14:21:49 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2022-11-15 14:21:49 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2022-11-15 14:21:49 0 [Note] InnoDB: log sequence number 72672576859; transaction id 23252907
2022-11-15 14:21:49 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-11-15 14:21:49 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-11-15 14:21:49 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2022-11-15 14:21:49 0 [Note] Server socket created on IP: '127.0.0.1'.
2022-11-15 14:21:49 0 [Note] /usr/sbin/mariadbd: ready for connections.
Version: '10.9.4-MariaDB-1:10.9.4+maria~deb11-log'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2022-11-15 14:21:50 156 [ERROR] mariadbd: Table './livea_db/ps_marketplace_product_action' is marked as crashed and should be repaired
2022-11-15 14:21:50 156 [Warning] Checking table:   './livea_db/ps_marketplace_product_action'
2022-11-15 14:21:50 173 [ERROR] mariadbd: Table './livea_db/ps_ba_cronjobs_importer' is marked as crashed and should be repaired
2022-11-15 14:21:50 173 [Warning] Checking table:   './livea_db/ps_ba_cronjobs_importer'
2022-11-15 14:21:54 0 [Note] InnoDB: Buffer pool(s) load completed at 221115 14:21:54

Most likely it also started to fail with random queries.

Comment by Vicente Rossello Jaume [ 2023-03-14 ]

Does this affect version 10.11.2? We upgraded last week and had a similar error with corruption.

2023-03-14  5:30:46 0 [Warning] InnoDB: Record in index `FK_tripnode_OUTBOUND_ID` of table `travelc`.`tripnode` was not found on rollback, trying to insert: TUPLE (info_bits=0, 2 fields): {[14]IDEA-2072373-3(0x494445412D323037323337332D33),[14]IDEA-2072373-3(0x494445412D323037323337332D33)} at: COMPACT RECORD(info_bits=0, 3 fields): {NULL,[0](0x),[4]    (0x00000000)}
2023-03-14  5:30:46 0 [Warning] InnoDB: Record in index `FK_tripnode_OUTBOUND_ID` of table `travelc`.`tripnode` was not found on rollback, trying to insert: TUPLE (info_bits=0, 2 fields): {[14]IDEA-2072373-3(0x494445412D323037323337332D33),[14]IDEA-2072373-3(0x494445412D323037323337332D33)} at: COMPACT RECORD(info_bits=0, 3 fields): {NULL,[0](0x),[4]    (0x00000000)}
230314  5:30:46 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.11.2-MariaDB-1:10.11.2+maria~ubu2204-log source revision: cafba8761af55ae16cc69c9b53a341340a845b36
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=5002
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11147677 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went

Comment by Daniel Black [ 2023-03-15 ]

Given your 10.11.2 crash was from fresh import, crash it does appear to affect that too. Can you try to get a backtrace from a core generated, with debug info installed. Can you also show SHOW CREATE TABLE information for the tables that appear in the travelc table an any other in the output. Obsecure table names if you want but please keep indexes and types identical. Any information you can provide about how these table updated might be of use. Also are you using any non-default settings my_print_defaults --mysqld?

Comment by Vicente Rossello Jaume [ 2023-03-15 ]

It wasn't a fresh import, we were coming from 10.9.4 and I already noticed a different index corruption in that table (different index), that I fixed just recreating that index. That time the corruption only affected queries using that index.

But this time there was a data corruption and the server could not boot. I had to fix rebooting with recovery=3, dumping the table and reimporting the table again.

I'll have to look at how to generate that backtrace, the core file is 1.1GB and I don't know how to have the debug info installed, we are using a docker container.

CREATE TABLE `tripnode` (
	`ID` VARCHAR(255) NOT NULL COLLATE 'utf8mb3_unicode_ci',
	`HOTELENABLED` TINYINT(1) NULL DEFAULT '0',
	`ARRIVALTRANSFERDISABLED` TINYINT(1) NOT NULL DEFAULT '0',
	`DEPARTURETRANSFERDISABLED` TINYINT(1) NOT NULL DEFAULT '0',
	`REQUESTEDARRIVALDATE` DATE NULL DEFAULT NULL,
	`STAYDAYS` INT(11) NULL DEFAULT NULL,
	`routing` TINYINT(1) NOT NULL DEFAULT '0',
	`transport_base_id` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`LOCATION_id` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`HOTEL_id` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`OUTBOUND_ID` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`SELECTEDHOTEL_ID` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`SELECTEDCRUISE_ID` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`MAGICBOX_ID` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`MAGICBOX_CLOSEDTOUR_ID` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`MAGICBOX_CARRENTAL_ID` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`SELECTEDCLOSEDTOUR_ID` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`tripid` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`nodeorder` INT(11) NULL DEFAULT NULL,
	`SELECTEDARRIVALTRANSFER_ID` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`SELECTEDDEPARTURETRANSFER_ID` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`SELECTEDCAR_ID` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`RIDEHAILINGSERVICE_ID` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb3_unicode_ci',
	`trip_spot_id` INT(11) NULL DEFAULT NULL,
	`stay_hour` INT(11) NULL DEFAULT NULL,
	`check_in` DATETIME NULL DEFAULT NULL,
	PRIMARY KEY (`ID`) USING BTREE,
	INDEX `FK_tripnode_OUTBOUND_ID` (`OUTBOUND_ID`) USING BTREE,
	INDEX `FK_tripnode_tripid` (`tripid`) USING BTREE,
	INDEX `FK_tripnode_SELECTEDHOTEL_ID` (`SELECTEDHOTEL_ID`) USING BTREE,
	INDEX `FK_tripnode_LOCATION_id` (`LOCATION_id`) USING BTREE,
	INDEX `FK_tripnode_SELECTEDDEPARTURETRANSFER_ID` (`SELECTEDDEPARTURETRANSFER_ID`) USING BTREE,
	INDEX `FK_tripnode_SELECTEDARRIVALTRANSFER_ID` (`SELECTEDARRIVALTRANSFER_ID`) USING BTREE,
	INDEX `FK_tripnode_SELECTEDCLOSEDTOUR_ID` (`SELECTEDCLOSEDTOUR_ID`) USING BTREE,
	INDEX `FK_tripnode_hotel` (`HOTEL_id`) USING BTREE,
	INDEX `FK_tripnode_transport_base_id` (`transport_base_id`) USING BTREE,
	INDEX `FK_tripnode_MAGICBOX_ID` (`MAGICBOX_ID`) USING BTREE,
	INDEX `FK_tripnode_MAGICBOX_CLOSEDTOUR_ID` (`MAGICBOX_CLOSEDTOUR_ID`) USING BTREE,
	INDEX `FK_tripnode_trip_spot_id` (`trip_spot_id`) USING BTREE,
	INDEX `SELECTEDCAR_ID` (`SELECTEDCAR_ID`) USING BTREE,
	INDEX `FK_tripnode_SELECTEDCRUISE_ID` (`SELECTEDCRUISE_ID`) USING BTREE,
	INDEX `FK_tripnode_RIDEHAILINGSERVICE_ID` (`RIDEHAILINGSERVICE_ID`) USING BTREE,
	INDEX `FK_tripnode_MAGICBOX_CARRENTAL_ID` (`MAGICBOX_CARRENTAL_ID`) USING BTREE,
	CONSTRAINT `FK_tripnode_LOCATION_id` FOREIGN KEY (`LOCATION_id`) REFERENCES `destination` (`id`) ON UPDATE RESTRICT ON DELETE RESTRICT,
	CONSTRAINT `FK_tripnode_MAGICBOX_CARRENTAL_ID` FOREIGN KEY (`MAGICBOX_CARRENTAL_ID`) REFERENCES `tripmagicboxcarrental` (`ID`) ON UPDATE NO ACTION ON DELETE NO ACTION,
	CONSTRAINT `FK_tripnode_MAGICBOX_CLOSEDTOUR_ID` FOREIGN KEY (`MAGICBOX_CLOSEDTOUR_ID`) REFERENCES `tripmagicboxclosedtourservice` (`ID`) ON UPDATE RESTRICT ON DELETE RESTRICT,
	CONSTRAINT `FK_tripnode_MAGICBOX_ID` FOREIGN KEY (`MAGICBOX_ID`) REFERENCES `tripmagicboxhotelservice` (`ID`) ON UPDATE RESTRICT ON DELETE RESTRICT,
	CONSTRAINT `FK_tripnode_OUTBOUND_ID` FOREIGN KEY (`OUTBOUND_ID`) REFERENCES `tripconnection` (`ID`) ON UPDATE RESTRICT ON DELETE RESTRICT,
	CONSTRAINT `FK_tripnode_RIDEHAILINGSERVICE_ID` FOREIGN KEY (`RIDEHAILINGSERVICE_ID`) REFERENCES `tripridehailingservice` (`ID`) ON UPDATE RESTRICT ON DELETE RESTRICT,
	CONSTRAINT `FK_tripnode_SELECTEDARRIVALTRANSFER_ID` FOREIGN KEY (`SELECTEDARRIVALTRANSFER_ID`) REFERENCES `triptransferservice` (`ID`) ON UPDATE RESTRICT ON DELETE RESTRICT,
	CONSTRAINT `FK_tripnode_SELECTEDCLOSEDTOUR_ID` FOREIGN KEY (`SELECTEDCLOSEDTOUR_ID`) REFERENCES `tripclosedtourservice` (`ID`) ON UPDATE RESTRICT ON DELETE RESTRICT,
	CONSTRAINT `FK_tripnode_SELECTEDCRUISE_ID` FOREIGN KEY (`SELECTEDCRUISE_ID`) REFERENCES `tripcruiseservice` (`ID`) ON UPDATE RESTRICT ON DELETE RESTRICT,
	CONSTRAINT `FK_tripnode_SELECTEDDEPARTURETRANSFER_ID` FOREIGN KEY (`SELECTEDDEPARTURETRANSFER_ID`) REFERENCES `triptransferservice` (`ID`) ON UPDATE RESTRICT ON DELETE RESTRICT,
	CONSTRAINT `FK_tripnode_SELECTEDHOTEL_ID` FOREIGN KEY (`SELECTEDHOTEL_ID`) REFERENCES `triphotelservice` (`ID`) ON UPDATE RESTRICT ON DELETE RESTRICT,
	CONSTRAINT `FK_tripnode_hotel` FOREIGN KEY (`HOTEL_id`) REFERENCES `hotel` (`ID`) ON UPDATE RESTRICT ON DELETE RESTRICT,
	CONSTRAINT `FK_tripnode_transport_base_id` FOREIGN KEY (`transport_base_id`) REFERENCES `transport_base` (`id`) ON UPDATE RESTRICT ON DELETE RESTRICT,
	CONSTRAINT `FK_tripnode_trip_spot_id` FOREIGN KEY (`trip_spot_id`) REFERENCES `trip_spot` (`ID`) ON UPDATE RESTRICT ON DELETE RESTRICT,
	CONSTRAINT `FK_tripnode_tripid` FOREIGN KEY (`tripid`) REFERENCES `trip` (`ID`) ON UPDATE RESTRICT ON DELETE RESTRICT,
	CONSTRAINT `tripnode_ibfk_1` FOREIGN KEY (`SELECTEDCAR_ID`) REFERENCES `tripcarrentalservice` (`ID`) ON UPDATE RESTRICT ON DELETE RESTRICT
)
COLLATE='utf8mb3_unicode_ci'
ENGINE=InnoDB
;

root@6259059155ba:/# my_print_defaults --mysqld
--socket=/run/mysqld/mysqld.sock
--host-cache-size=0
--skip-name-resolve
--pid-file=/run/mysqld/mysqld.pid
--basedir=/usr
--expire_logs_days=10
--character-set-server=utf8mb4
--collation-server=utf8mb4_general_ci
--server_id=1
--log-basename=master1
--user=mysql
--pid-file=/var/run/mysqld/mysqld.pid
--socket=/var/run/mysqld/mysqld.sock
--port=3306
--basedir=/usr
--tmpdir=/tmp
--lc-messages-dir=/usr/share/mysql
--log_warnings=2
--lower_case_table_names=0
--innodb_use_native_aio=OFF
--performance_schema=on
--userstat=1
--skip-external-locking
--init_connect=SET collation_connection = utf8mb4_unicode_ci
--init_connect=SET NAMES utf8mb4
--character-set-server=utf8mb4
--collation-server=utf8mb4_unicode_ci
--skip-character-set-client-handshake
--key_buffer_size=128M
--max_allowed_packet=64M
--thread_stack=192K
--table_open_cache_instances=48
--table_open_cache=6000
--table_definition_cache=2000
--max_heap_table_size=256M
--max_connections=5000
--query_cache_limit=0
--query_cache_size=0
--log_slow_query=1
--log_slow_query_time=2
--binlog-format=mixed
--binlog_expire_logs_seconds=0
--max_binlog_size=200M
--innodb_buffer_pool_size=1024M
--innodb_log_file_size=512M
--innodb_log_buffer_size=256M
--innodb_io_capacity=1000
--innodb_io_capacity_max=4000
--innodb_strict_mode=1
--innodb_file_per_table=1
--innodb_ft_min_token_size=2
--innodb_ft_enable_stopword=OFF
--innodb_write_io_threads=32
--innodb_read_io_threads=32
--innodb_flush_method=O_DIRECT
--innodb_buffer_pool_load_at_startup=ON
--innodb_buffer_pool_dump_at_shutdown=ON
--innodb_buffer_pool_dump_pct=50
--innodb_flush_neighbors=0
--innodb_monitor_enable=1
--innodb_adaptive_hash_index=0
--innodb_print_all_deadlocks=1
--join_buffer_size=256k
--max_prepared_stmt_count=100000
--thread_pool_size=100
--skip_name_resolve
--server_id=1
--log-basename=master1
--user=mysql
--pid-file=/var/run/mysqld/mysqld.pid
--socket=/var/run/mysqld/mysqld.sock
--port=3306
--basedir=/usr
--tmpdir=/tmp
--lc-messages-dir=/usr/share/mysql
--log_warnings=2
--lower_case_table_names=0
--innodb_use_native_aio=OFF
--performance_schema=on
--userstat=1
--skip-external-locking
--init_connect=SET collation_connection = utf8mb4_unicode_ci
--init_connect=SET NAMES utf8mb4
--character-set-server=utf8mb4
--collation-server=utf8mb4_unicode_ci
--skip-character-set-client-handshake
--key_buffer_size=128M
--max_allowed_packet=64M
--thread_stack=192K
--table_open_cache_instances=48
--table_open_cache=6000
--table_definition_cache=2000
--max_heap_table_size=256M
--max_connections=5000
--query_cache_limit=0
--query_cache_size=0
--log_slow_query=1
--log_slow_query_time=2
--binlog-format=mixed
--binlog_expire_logs_seconds=0
--max_binlog_size=200M
--innodb_buffer_pool_size=1024M
--innodb_log_file_size=512M
--innodb_log_buffer_size=256M
--innodb_io_capacity=1000
--innodb_io_capacity_max=4000
--innodb_strict_mode=1
--innodb_file_per_table=1
--innodb_ft_min_token_size=2
--innodb_ft_enable_stopword=OFF
--innodb_write_io_threads=32
--innodb_read_io_threads=32
--innodb_flush_method=O_DIRECT
--innodb_buffer_pool_load_at_startup=ON
--innodb_buffer_pool_dump_at_shutdown=ON
--innodb_buffer_pool_dump_pct=50
--innodb_flush_neighbors=0
--innodb_monitor_enable=1
--innodb_adaptive_hash_index=0
--innodb_print_all_deadlocks=1
--join_buffer_size=256k
--max_prepared_stmt_count=100000
--thread_pool_size=100
--skip_name_resolve
--max_statement_time=20
--innodb_buffer_pool_size=18432M
--innodb_force_recovery=3
--server_id=1
--log-basename=master1

Comment by Marko Mäkelä [ 2023-03-15 ]

That the built-in stack traces are misleading is the subject of MDEV-20738. Unfortunately, the stack trace is not helpful at all. Someone would have to resolve the addresses properly to generate a usable stack trace.

If the data files are corrupted, the corruption will not disappear when you upgrade or downgrade to a different version. One very nasty source of corruption was caused by MDEV-24096. You should only be affected by that if you ever used MariaDB 10.5.7 or 10.4.16. A more recent one was MDEV-29383, affecting 10.6.9, 10.7.5, 10.8.4, 10.9.2, 10.10.1. If you used mariadb-backup or crash recovery with any of those releases, you could have a corrupted database.

Comment by Vicente Rossello Jaume [ 2023-03-15 ]

We used 10.8.3 and 10.7.3 so yes, we could have the problem. How can we detect that corruption? The mysqlcheck did not throw anything.

Comment by Marko Mäkelä [ 2023-03-15 ]

cocorossello, if you used 10.8.3 and 10.7.3 but not 10.7.5 or 10.8.4, you should not be affected by MDEV-29383. Possibly, the MDEV-24402 CHECK TABLE…EXTENDED could catch the corruption more likely. Did you try the --extended flag (MDEV-30129)?

Comment by Marko Mäkelä [ 2023-03-15 ]

In any case, I would like to fix the SIGSEGV, if someone could identify the crashing piece of code, and possibly provide a copy of the corrupted page. Any bug related to a crash on corruption should have no excuse to remain ignored, now that MDEV-13542 and related bugs have been fixed.

Comment by Vicente Rossello Jaume [ 2023-03-16 ]

Marko, sorry for stealing hijacking this thread but we found some corruptions with the extended, not sure about what they mean:
travelc.SEQUENCE
Warning : InnoDB: Clustered index record with stale history in table `travelc`.`SEQUENCE`: COMPACT RECORD(info_bits=0, 4 fields):

{[13]SEQ_GEN_TABLE(0x5345515F47454E5F5441424C45),[6] l A(0x0000126C0441),[7]B $ (0x4200002497040B),[17] (0x80000000000000000000000000000000C8)}

status : OK
....
(it happens with quite a lot of tables). mysqlcheck without extended does not report anything

Generated at Thu Feb 08 10:10:16 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.