Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32598

crash in maria/ma_rt_split.c:210(_pcre_xclass) with Aria geospatial index page split

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.4.31
    • 10.4(EOL)

    Description

      First attempted to do CREATE LOCATIONS LIKE source.locations; INSERT INTO locations ( SELECT * FROM source.locations ); and it crashed. After that I tried REPAIR TABLE source.locations and it crashed again, then in recovery, and finally hung indefinitely. Moving source.locations out and killing process got it to start again.

      (gdb) where
      #0  0x0000000001a16afd in split_maria_rtree_node (node=0x7fff7a398320, n_entries=221, all_size=8196, key_size=37, min_size=2729, size1=2, size2=2, d_buffer=0x7fff7a39a050, n_dim=2) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_split.c:210
      #1  0x0000000001a182e9 in maria_rtree_split_page (key=0x7fff7a3a1c40, page=0x7fff7a39c910, new_page_offs=0x7fff7a3a1870) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_split.c:425
      #2  0x00000000019e7aa8 in maria_rtree_add_key (key=0x7fff7a3a1c40, page=0x7fff7a39c910, new_page=0x7fff7a3a1870) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_key.c:67
      #3  0x00000000019e1b67 in maria_rtree_insert_req (info=0x62a00003c270, key=0x7fff7a3a1c40, page_pos=43089920, new_page=0x7fff7a3a1870, ins_level=-1, level=2) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_index.c:690
      #4  0x00000000019e12fd in maria_rtree_insert_req (info=0x62a00003c270, key=0x7fff7a3a1c40, page_pos=27238400, new_page=0x7fff7a3a1870, ins_level=-1, level=1) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_index.c:631
      #5  0x00000000019e12fd in maria_rtree_insert_req (info=0x62a00003c270, key=0x7fff7a3a1c40, page_pos=5521408, new_page=0x7fff7a3a1870, ins_level=-1, level=0) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_index.c:631
      #6  0x00000000019e2694 in maria_rtree_insert_level (info=0x62a00003c270, key=0x7fff7a3a1c40, ins_level=-1, root=0x7fff7a3a1b20) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_index.c:758
      #7  0x00000000019e34ab in maria_rtree_insert (info=0x62a00003c270, key=0x7fff7a3a1c40) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_rt_index.c:862
      #8  0x00000000019ada61 in writekeys (sort_param=0x7fff7a3a2230) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_check.c:2960
      #9  0x00000000019ab660 in maria_repair (param=0x7fff79cef890, info=0x7fff7a398280, name=0x7fff7a3a3a50 "./avl/clientLocations", rep_quick=0 '\000') at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ma_check.c:2727
      #10 0x00000000018069f3 in ha_maria::repair (this=0x61d0001a2310, thd=0x62b00009a270, param=0x7fff79cef890, do_optimize=false) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ha_maria.cc:1673
      #11 0x000000000180490e in ha_maria::repair (this=0x61d0001a2310, thd=0x62b00009a270, check_opt=0x7fff7a3a3e30) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ha_maria.cc:1453
      #12 0x000000000180b704 in ha_maria::check_and_repair (this=0x61d0001a2310, thd=0x62b00009a270) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/maria/ha_maria.cc:2336
      #13 0x00000000011fbfac in handler::ha_check_and_repair (this=0x61d0001a2310, thd=0x62b00009a270) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/handler.cc:4610
      #14 0x00000000008ad8a3 in auto_repair_table (thd=0x62b00009a270, table_list=0x62b0000a12e8) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_base.cc:3121
      #15 0x00000000008ae81b in Open_table_context::recover_from_failed_open (this=0x7fff7a3a4170) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_base.cc:3352
      #16 0x00000000008b3ce4 in open_tables (thd=0x62b00009a270, options=..., start=0x7fff7a3a42d0, counter=0x7fff7a3a42a0, flags=1024, prelocking_strategy=0x7fff7a3a42b0) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_base.cc:4425
      #17 0x000000000089c6f2 in open_tables (thd=0x62b00009a270, tables=0x7fff7a3a42d0, counter=0x7fff7a3a42a0, flags=1024, prelocking_strategy=0x7fff7a3a42b0) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_base.h:257
      #18 0x00000000008b931b in open_normal_and_derived_tables (thd=0x62b00009a270, tables=0x7fff7a3a4fe0, flags=1024, dt_phases=3) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_base.cc:5404
      #19 0x0000000000bee094 in mysqld_list_fields (thd=0x62b00009a270, table_list=0x7fff7a3a4fe0, wild=0x62b0000a12d8 "") at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_show.cc:1585
      #20 0x0000000000a2dd52 in dispatch_command (command=COM_FIELD_LIST, thd=0x62b00009a270, packet=0x62500005f181 "", packet_length=16, is_com_multi=false, is_next_command=false) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_parse.cc:2062
      #21 0x0000000000a29201 in do_command (thd=0x62b00009a270) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_parse.cc:1378
      #22 0x0000000000e2953b in do_handle_one_connection (connect=0x611000041830) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_connect.cc:1420
      #23 0x0000000000e28da9 in handle_one_connection (arg=0x611000041830) at /var/lib/innodb/mariadb/mariadb-10.4.31/sql/sql_connect.cc:1324
      #24 0x0000000001b14640 in pfs_spawn_thread (arg=0x61600000fcf0) at /var/lib/innodb/mariadb/mariadb-10.4.31/storage/perfschema/pfs.cc:1869
      #25 0x00007ffff5eed1da in start_thread (arg=<optimized out>) at pthread_create.c:479
      #26 0x00007ffff5025e73 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
      (gdb) print a
      $37 = (SplitStruct *) 0x7fefffffffffffff
      (gdb) print b
      $38 = (SplitStruct *) 0x19f0ea5 <maria_rtree_d_mbr>
      

      This table is read & written extensively, with about 300k rows of about 160 bytes each, but otherwise no idea how to reproduce the issue. I'm seeing MDEV-31766 is similar, though another function in the code.

      It looks like in our case "square" of all the SplitStruct it's handling is INFINITY (I don't know why though), then the d in pick_seeds() becomes negative infinity so no combination passes, and pick_seeds() passes a & b (which are defined UNINIT_VAR) back unassigned.

      Attachments

        Activity

          we need a way to reproduce the issue to be able to fix it.
          one idea that might help — instead of inserting all rows at once, you could try to insert them in batches, like, for example

          INSERT INTO locations SELECT * FROM source.locations LIMIT 100;
          INSERT INTO locations SELECT * FROM source.locations LIMIT 100,100;
          INSERT INTO locations SELECT * FROM source.locations LIMIT 200,100;
          

          and so on.
          Another option, you can wait for MDEV-31766 to be fixed and then try the fixed release on your data.

          serg Sergei Golubchik added a comment - we need a way to reproduce the issue to be able to fix it. one idea that might help — instead of inserting all rows at once, you could try to insert them in batches, like, for example INSERT INTO locations SELECT * FROM source.locations LIMIT 100; INSERT INTO locations SELECT * FROM source.locations LIMIT 100,100; INSERT INTO locations SELECT * FROM source.locations LIMIT 200,100; and so on. Another option, you can wait for MDEV-31766 to be fixed and then try the fixed release on your data.
          jsantala Jukka Santala added a comment -

          Bad news regarding that. The other MDEV looks superficially similar, but under ASAN turns out to be heap-buffer-overflow in /mariadb-10.4.31/storage/maria/ma_rt_index.c:859 root= &share->state.key_root[key->keyinfo->key_nr]; accessing key->keyinfo->key_nr. This is almost certainly unrelated to our case.

          The crash happens on other instances of the same cluster with same table writes (not binary copy) as well, around 250k rows. However, it doesn't happen if the rows are inserted in different order OR from a known good copy of the table. The tables still compare as identical, only spatial index is broken. So figuring out the error in the logic or instrumenting it to that effect would seem like the best bet here.

          Additionally, pick_seeds() could check if it fails to find a pair, and return merge failed instead of crashing. Though that would negate any possible performance improvement in not assigning them (doubt it's for performance at all in that case). Looks like EXTENDED check-table finds a clue to the problem though: "Record at: 368:23 Can't find key for index: 2" Is there easy way to peek at what we have?

          jsantala Jukka Santala added a comment - Bad news regarding that. The other MDEV looks superficially similar, but under ASAN turns out to be heap-buffer-overflow in /mariadb-10.4.31/storage/maria/ma_rt_index.c:859 root= &share->state.key_root [key->keyinfo->key_nr] ; accessing key->keyinfo->key_nr. This is almost certainly unrelated to our case. The crash happens on other instances of the same cluster with same table writes (not binary copy) as well, around 250k rows. However, it doesn't happen if the rows are inserted in different order OR from a known good copy of the table. The tables still compare as identical, only spatial index is broken. So figuring out the error in the logic or instrumenting it to that effect would seem like the best bet here. Additionally, pick_seeds() could check if it fails to find a pair, and return merge failed instead of crashing. Though that would negate any possible performance improvement in not assigning them (doubt it's for performance at all in that case). Looks like EXTENDED check-table finds a clue to the problem though: "Record at: 368:23 Can't find key for index: 2" Is there easy way to peek at what we have?
          jsantala Jukka Santala added a comment -

          Got a reproducer now, not that mysterious after:

          CREATE TABLE `g_empty` (
            `g` geometry NOT NULL,
            SPATIAL KEY `g` (`g`)
          ) ENGINE=Aria;
          DELIMITER //
          CREATE PROCEDURE fill_with_empty_geometrycollection() BEGIN label: LOOP INSERT INTO g_empty VALUES (GeomFromText('GeometryCollection EMPTY')); END LOOP; END
              -> //
          DELIMITER ;
          call fill_with_empty_geometrycollection();
          

          jsantala Jukka Santala added a comment - Got a reproducer now, not that mysterious after: CREATE TABLE `g_empty` ( `g` geometry NOT NULL , SPATIAL KEY `g` (`g`) ) ENGINE=Aria; DELIMITER // CREATE PROCEDURE fill_with_empty_geometrycollection() BEGIN label: LOOP INSERT INTO g_empty VALUES (GeomFromText( 'GeometryCollection EMPTY' )); END LOOP; END -> // DELIMITER ; call fill_with_empty_geometrycollection();

          People

            holyfoot Alexey Botchkov
            jsantala Jukka Santala
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.