[MDEV-32598] crash in maria/ma_rt_split.c:210(_pcre_xclass) with Aria geospatial index page split Created: 2023-10-27  Updated: 2023-10-31

Status: Open
Project: MariaDB Server
Component/s: Data Manipulation - Insert
Affects Version/s: 10.4.31
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Jukka Santala Assignee: Alexey Botchkov
Resolution: Unresolved Votes: 0
Labels: crash
Environment:

RHEL 8.8 server
MariaDB [avl]> SHOW VARIABLES LIKE "aria%";
-------------------------------------------------------------+

Variable_name Value

-------------------------------------------------------------+

aria_block_size 8192
aria_checkpoint_interval 30
aria_checkpoint_log_activity 1048576
aria_encrypt_tables OFF
aria_force_start_after_recovery_failures 0
aria_group_commit none
aria_group_commit_interval 0
aria_log_dir_path /var/lib/mysql/
aria_log_file_size 1073741824
aria_log_purge_type immediate
aria_max_sort_file_size 9223372036853727232
aria_page_checksum ON
aria_pagecache_age_threshold 300
aria_pagecache_buffer_size 134217728
aria_pagecache_division_limit 100
aria_pagecache_file_hash_size 512
aria_recover_options BACKUP,QUICK
aria_repair_threads 1
aria_sort_buffer_size 268434432
aria_stats_method nulls_unequal
aria_sync_log_dir NEWFILE
aria_used_for_temp_tables ON

-------------------------------------------------------------+
22 rows in set (0.001 sec)


 Description   

First attempted to do CREATE LOCATIONS LIKE source.locations; INSERT INTO locations ( SELECT * FROM source.locations ); and it crashed. After that I tried REPAIR TABLE source.locations and it crashed again, then in recovery, and finally hung indefinitely. Moving source.locations out and killing process got it to start again.

(gdb) where
 
#0  0x0000000001a16afd in split_maria_rtree_node (node=0x7fff7a398320, n_entries=221, all_size=8196, key_size=37, min_size=2729, size1=2, size2=2, d_buffer=0x7fff7a39a050, n_dim=2) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_split.c:210
 
#1  0x0000000001a182e9 in maria_rtree_split_page (key=0x7fff7a3a1c40, page=0x7fff7a39c910, new_page_offs=0x7fff7a3a1870) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_split.c:425
 
#2  0x00000000019e7aa8 in maria_rtree_add_key (key=0x7fff7a3a1c40, page=0x7fff7a39c910, new_page=0x7fff7a3a1870) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_key.c:67
 
#3  0x00000000019e1b67 in maria_rtree_insert_req (info=0x62a00003c270, key=0x7fff7a3a1c40, page_pos=43089920, new_page=0x7fff7a3a1870, ins_level=-1, level=2) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_index.c:690
 
#4  0x00000000019e12fd in maria_rtree_insert_req (info=0x62a00003c270, key=0x7fff7a3a1c40, page_pos=27238400, new_page=0x7fff7a3a1870, ins_level=-1, level=1) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_index.c:631
 
#5  0x00000000019e12fd in maria_rtree_insert_req (info=0x62a00003c270, key=0x7fff7a3a1c40, page_pos=5521408, new_page=0x7fff7a3a1870, ins_level=-1, level=0) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_index.c:631
 
#6  0x00000000019e2694 in maria_rtree_insert_level (info=0x62a00003c270, key=0x7fff7a3a1c40, ins_level=-1, root=0x7fff7a3a1b20) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_index.c:758
 
#7  0x00000000019e34ab in maria_rtree_insert (info=0x62a00003c270, key=0x7fff7a3a1c40) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_index.c:862
 
#8  0x00000000019ada61 in writekeys (sort_param=0x7fff7a3a2230) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_check.c:2960
 
#9  0x00000000019ab660 in maria_repair (param=0x7fff79cef890, info=0x7fff7a398280, name=0x7fff7a3a3a50 "./avl/clientLocations", rep_quick=0 '\000') at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_check.c:2727
 
#10 0x00000000018069f3 in ha_maria::repair (this=0x61d0001a2310, thd=0x62b00009a270, param=0x7fff79cef890, do_optimize=false) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ha_maria.cc:1673
 
#11 0x000000000180490e in ha_maria::repair (this=0x61d0001a2310, thd=0x62b00009a270, check_opt=0x7fff7a3a3e30) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ha_maria.cc:1453
 
#12 0x000000000180b704 in ha_maria::check_and_repair (this=0x61d0001a2310, thd=0x62b00009a270) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ha_maria.cc:2336
 
#13 0x00000000011fbfac in handler::ha_check_and_repair (this=0x61d0001a2310, thd=0x62b00009a270) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/handler.cc:4610
 
#14 0x00000000008ad8a3 in auto_repair_table (thd=0x62b00009a270, table_list=0x62b0000a12e8) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_base.cc:3121
 
#15 0x00000000008ae81b in Open_table_context::recover_from_failed_open (this=0x7fff7a3a4170) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_base.cc:3352
 
#16 0x00000000008b3ce4 in open_tables (thd=0x62b00009a270, options=..., start=0x7fff7a3a42d0, counter=0x7fff7a3a42a0, flags=1024, prelocking_strategy=0x7fff7a3a42b0) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_base.cc:4425
 
#17 0x000000000089c6f2 in open_tables (thd=0x62b00009a270, tables=0x7fff7a3a42d0, counter=0x7fff7a3a42a0, flags=1024, prelocking_strategy=0x7fff7a3a42b0) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_base.h:257
 
#18 0x00000000008b931b in open_normal_and_derived_tables (thd=0x62b00009a270, tables=0x7fff7a3a4fe0, flags=1024, dt_phases=3) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_base.cc:5404
 
#19 0x0000000000bee094 in mysqld_list_fields (thd=0x62b00009a270, table_list=0x7fff7a3a4fe0, wild=0x62b0000a12d8 "") at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_show.cc:1585
 
#20 0x0000000000a2dd52 in dispatch_command (command=COM_FIELD_LIST, thd=0x62b00009a270, packet=0x62500005f181 "", packet_length=16, is_com_multi=false, is_next_command=false) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_parse.cc:2062
 
#21 0x0000000000a29201 in do_command (thd=0x62b00009a270) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_parse.cc:1378
 
#22 0x0000000000e2953b in do_handle_one_connection (connect=0x611000041830) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_connect.cc:1420
 
#23 0x0000000000e28da9 in handle_one_connection (arg=0x611000041830) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_connect.cc:1324
 
#24 0x0000000001b14640 in pfs_spawn_thread (arg=0x61600000fcf0) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/perfschema/pfs.cc:1869
 
#25 0x00007ffff5eed1da in start_thread (arg=<optimized out>) at pthread_create.c:479
 
#26 0x00007ffff5025e73 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
(gdb) print a
 
$37 = (SplitStruct *) 0x7fefffffffffffff
 
(gdb) print b
 
$38 = (SplitStruct *) 0x19f0ea5 <maria_rtree_d_mbr>

This table is read & written extensively, with about 300k rows of about 160 bytes each, but otherwise no idea how to reproduce the issue. I'm seeing MDEV-31766 is similar, though another function in the code.

It looks like in our case "square" of all the SplitStruct it's handling is INFINITY (I don't know why though), then the d in pick_seeds() becomes negative infinity so no combination passes, and pick_seeds() passes a & b (which are defined UNINIT_VAR) back unassigned.

 Comments   
Comment by Sergei Golubchik [ 2023-10-27 ]

we need a way to reproduce the issue to be able to fix it.
one idea that might help — instead of inserting all rows at once, you could try to insert them in batches, like, for example

INSERT INTO locations SELECT * FROM source.locations LIMIT 100;
 
INSERT INTO locations SELECT * FROM source.locations LIMIT 100,100;
 
INSERT INTO locations SELECT * FROM source.locations LIMIT 200,100;

and so on.
Another option, you can wait for MDEV-31766 to be fixed and then try the fixed release on your data.

Comment by Jukka Santala [ 2023-10-28 ]

Bad news regarding that. The other MDEV looks superficially similar, but under ASAN turns out to be heap-buffer-overflow in /mariadb-10.4.31/storage/maria/ma_rt_index.c:859 root= &share->state.key_root[key->keyinfo->key_nr]; accessing key->keyinfo->key_nr. This is almost certainly unrelated to our case.

The crash happens on other instances of the same cluster with same table writes (not binary copy) as well, around 250k rows. However, it doesn't happen if the rows are inserted in different order OR from a known good copy of the table. The tables still compare as identical, only spatial index is broken. So figuring out the error in the logic or instrumenting it to that effect would seem like the best bet here.

Additionally, pick_seeds() could check if it fails to find a pair, and return merge failed instead of crashing. Though that would negate any possible performance improvement in not assigning them (doubt it's for performance at all in that case). Looks like EXTENDED check-table finds a clue to the problem though: "Record at: 368:23 Can't find key for index: 2" Is there easy way to peek at what we have?

Comment by Jukka Santala [ 2023-10-28 ]

Got a reproducer now, not that mysterious after:

CREATE TABLE `g_empty` (
 
  `g` geometry NOT NULL,
 
  SPATIAL KEY `g` (`g`)
 
) ENGINE=Aria;
 
DELIMITER //
 
CREATE PROCEDURE fill_with_empty_geometrycollection() BEGIN label: LOOP INSERT INTO g_empty VALUES (GeomFromText('GeometryCollection EMPTY')); END LOOP; END
 
    -> //
 
DELIMITER ;
 
call fill_with_empty_geometrycollection();

Generated at Thu Feb 08 10:32:34 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.