Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32591

InnoDB indexes are inconsistent after dropping partition

Details

    Description

      Tables are used to store time series data. Older data is removed by dropping partitions from the table. This has been in place for a number of years.

      Recently upgraded from 10.3->10.5. Since then, we've dropped a partition in two separate MariaDB instances. In both cases, after the partition was dropped, attempts to insert or delete rows from the table fails with errors like

      2023-10-26 16:32:48 467 [ERROR] Found index <index> whose column info does not match that of MariaDB.
      		 
      2023-10-26 16:32:48 467 [ERROR] InnoDB indexes are inconsistent with what defined in .frm for table ./<db>/<table>#P#P20230901
      		 
       
      		 
      2023-10-26 15:38:46 64307678 [Warning] InnoDB: Using a partial-field key prefix in search, index `<index>` of table `<db>`.`<table>` /* Partition `P20231101` */. Last data field length 8 bytes, key ptr now exceeds key end by 3 bytes. Key value in the MySQL format:
      		 
       len 7; hex b50399b1740000; asc     t  ;
      		 
       
      		 
      2023-10-26 15:38:46 64307678 [ERROR] InnoDB: Clustered record for sec rec not found index `<index>` of table `<db>`.`<table>` /* Partition `P20231101` */
      		 
      InnoDB: sec index record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
      		 
       0: len 2; hex 03b5; asc   ;;
      		 
       1: len 8; hex 87c174b199007005; asc   t   p ;;
      		 
       2: len 6; hex af05f8af05e2; asc       ;;
      		 
       
      		 
      InnoDB: clust index record PHYSICAL RECORD: n_fields 10; compact format; info bits 0
      		 
       0: len 6; hex 007005af066c; asc  p   l;;
      		 
       1: len 6; hex 000000000000; asc       ;;
      		 
       2: len 7; hex 80000000000000; asc        ;;
      		 
       3: len 2; hex 03b5; asc   ;;
      		 
       4: len 4; hex 000146ef; asc   F ;;
      		 
       5: len 8; hex 87e074b199000000; asc   t     ;;
      		 
       6: len 4; hex 00000000; asc     ;;
      		 
       7: len 4; hex 00000000; asc     ;;
      		 
       8: len 4; hex 15000000; asc     ;;
      		 
       9: len 2; hex 0000; asc   ;;
      		 
       
      		 
      TRANSACTION 421589657076240, ACTIVE 0 sec starting index read
      		 
      mysql tables in use 8, locked 0
      		 
      0 lock struct(s), heap size 1128, 0 row lock(s)
      		 
      MySQL thread id 64307678, OS thread handle 140095744042752, query id 791359259 <hostname> 192.168.255.80 statsadm Sending data
      		 
      SELECT
      		 
       <tableb>.hostname  AS hostname,
      		 
       MAX( <table>.numCurrentSessions ) AS max
      		 
      FROM
      		 
       <table>,
      		 
       <tableb>,
      		 
       <tablec>
      		 
      WHERE
      		 
       <tablec>.name = @site AND
      		 
       <table>.time BETWEEN @date AND @date + INTERVAL 1 DAY AND
      		 
       <table>.siteid = <tablec>.id AND
      		 
       <tableb>.id= <table>.serverid
      		 
      GROUP BY hostname
      		 
       
      		 
      InnoDB: Submit a detailed bug report to https://jira.mariadb.org/


      2023-10-26 15:40:30 64307291 [ERROR] [FATAL] InnoDB: Cannot insert tuple TUPLE (info_bits=0, 3 fields): {[2]  (0x02E2),[5]  t  (0x1DDE74B199),[6] p    (0x007005BDDCCA)}into index `<index>` of table `<db>`.`<table>` /* Partition `P20231101` */. Max size: 19
      		 
      231026 15:40:30 [ERROR] mysqld got signal 6 ;
      		 
      This could be because you hit a bug. It is also possible that this binary
      		 
      or one of the libraries it was linked against is corrupt, improperly built,
      		 
      or misconfigured. This error can also be caused by malfunctioning hardware.
      		 
       
      		 
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      		 
       
      		 
      We will try our best to scrape up some info that will hopefully help
      		 
      diagnose the problem, but since we have already crashed,
      		 
      something is definitely wrong and this may fail.
      		 
       
      		 
      Server version: 10.5.16-MariaDB-log
      		 
      key_buffer_size=134217728
      		 
      read_buffer_size=131072
      		 
      max_used_connections=56
      		 
      max_threads=153
      		 
      thread_count=30
      		 
      It is possible that mysqld could use up to
      		 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467873 K  bytes of memory
      		 
      Hope that's ok; if not, decrease some variables in the equation.
      		 
       
      		 
      Thread pointer: 0x7f6a9c39c5a8
      		 
      Attempting backtrace. You can use the following information to find out
      		 
      where mysqld died. If you see no messages after this, something went
      		 
      terribly wrong...
      		 
      stack_bottom = 0x7f6a94b44bd8 thread_stack 0x49000
      		 
      Can't start addr2line
      		 
      /usr/libexec/mysqld(my_print_stacktrace+0x41)[0x55a8ee32add1]
      		 
      /usr/libexec/mysqld(handle_fatal_signal+0x4a5)[0x55a8eddc0355]
      		 
      /lib64/libpthread.so.0(+0x12cf0)[0x7f6f0959fcf0]
      		 
      /lib64/libc.so.6(gsignal+0x10f)[0x7f6f088fbacf]
      		 
      /lib64/libc.so.6(abort+0x127)[0x7f6f088ceea5]
      		 
      /usr/libexec/mysqld(+0x644be7)[0x55a8edaa8be7]
      		 
      /usr/libexec/mysqld(+0x6485f6)[0x55a8edaac5f6]
      		 
      /usr/libexec/mysqld(+0xce28fd)[0x55a8ee1468fd]
      		 
      /usr/libexec/mysqld(+0xce78de)[0x55a8ee14b8de]
      		 
      /usr/libexec/mysqld(+0xce7fe4)[0x55a8ee14bfe4]
      		 
      /usr/libexec/mysqld(+0xcf8bd0)[0x55a8ee15cbd0]
      		 
      /usr/libexec/mysqld(+0xc411ed)[0x55a8ee0a51ed]
      		 
      /usr/libexec/mysqld(_ZN7handler12ha_write_rowEPKh+0x177)[0x55a8eddcdee7]
      		 
      /usr/libexec/mysqld(+0xb812f0)[0x55a8edfe52f0]
      		 
      /usr/libexec/mysqld(_ZN7handler12ha_write_rowEPKh+0x177)[0x55a8eddcdee7]
      		 
      /usr/libexec/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_infoP13select_result+0x1dd)[0x55a8edb8d34d]
      		 
      /usr/libexec/mysqld(_Z10mysql_loadP3THDPK12sql_exchangeP10TABLE_LISTR4ListI4ItemES9_S9_15enum_duplicatesbb+0x17de)[0x55a8edbba71e]
      		 
      /usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x185f)[0x55a8edbcba0e]
      		 
      /usr/libexec/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x238)[0x55a8edbbc956]
      		 
      /usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x177d)[0x55a8edbc842c]
      		 
      /usr/libexec/mysqld(_Z10do_commandP3THD+0x122)[0x55a8edbc9702]
      		 
      /usr/libexec/mysqld(_Z24do_handle_one_connectionP7CONNECTb+0x4e0)[0x55a8edcbaa00]
      		 
      /usr/libexec/mysqld(handle_one_connection+0x5d)[0x55a8edcbadad]
      		 
      /usr/libexec/mysqld(+0xb882d8)[0x55a8edfec2d8]
      		 
      /lib64/libpthread.so.0(+0x81ca)[0x7f6f095951ca]
      		 
      /lib64/libc.so.6(clone+0x43)[0x7f6f088e6e73]
      		 
       
      		 
      Trying to get some variables.
      		 
      Some pointers may be invalid and cause the dump to abort.
      		 
      Query (0x7f6a9c04a370): LOAD DATA LOCAL INFILE '/data/tmp/makeStats_12/<table>.bcp' INTO TABLE <table> (time,serverid,siteid,cpuUsed,memoryUsed,processes,numCurrentSessions)
      		 
       
      		 
      Connection ID (thread ID): 64307291
      		 
      Status: NOT_KILLED
      		 
       
      		 
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
      		 
       
      		 
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      		 
      information that should help you find out what is causing the crash.
      		 
      Writing a core file...
      		 
      Working directory at /data/db/data
      		 
      Resource Limits:
      		 
      Limit                     Soft Limit           Hard Limit           Units
      		 
      Max cpu time              unlimited            unlimited            seconds
      		 
      Max file size             unlimited            unlimited            bytes
      		 
      Max data size             unlimited            unlimited            bytes
      		 
      Max stack size            8388608              unlimited            bytes
      		 
      Max core file size        0                    unlimited            bytes
      		 
      Max resident set          unlimited            unlimited            bytes
      		 
      Max processes             94970                94970                processes
      		 
      Max open files            160000               160000               files
      		 
      Max locked memory         65536                65536                bytes
      		 
      Max address space         unlimited            unlimited            bytes
      		 
      Max file locks            unlimited            unlimited            locks
      		 
      Max pending signals       94970                94970                signals
      		 
      Max msgqueue size         819200               819200               bytes
      		 
      Max nice priority         0                    0
      		 
      Max realtime priority     0                    0
      		 
      Max realtime timeout      unlimited            unlimited            us
      		 
      Core pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e

      Attachments

        Activity

          Ascending order - Click to sort in descending order
          conormarkmurphy Conor Murphy created issue -
          conormarkmurphy Conor Murphy made changes -
          Field Original Value New Value
          Description Tables are used to store time series data. Older data is removed by dropping partitions from the table. This has been in place for a number of years.

          Recently upgraded from 10.3->10.5. Since then, we've dropped a partition in two separate MariaDB instances. In both cases, after the partition was dropped, attempts to insert or delete rows from the table fails with errors like

          {code}
          2023-10-26 16:32:48 467 [ERROR] Found index <index> whose column info does not match that of MariaDB.
          2023-10-26 16:32:48 467 [ERROR] InnoDB indexes are inconsistent with what defined in .frm for table ./<db>/<table>#P#P20230901

          2023-10-26 15:38:46 64307678 [Warning] InnoDB: Using a partial-field key prefix in search, index `<index>` of table `<db>`.`<table>` /* Partition `P20231101` */. Last data field length 8 bytes, key ptr now exceeds key end by 3 bytes. Key value in the MySQL format:
           len 7; hex b50399b1740000; asc t ;

          2023-10-26 15:38:46 64307678 [ERROR] InnoDB: Clustered record for sec rec not found index `<index>` of table `<db>`.`<table>` /* Partition `P20231101` */
          InnoDB: sec index record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
           0: len 2; hex 03b5; asc ;;
           1: len 8; hex 87c174b199007005; asc t p ;;
           2: len 6; hex af05f8af05e2; asc ;;

          InnoDB: clust index record PHYSICAL RECORD: n_fields 10; compact format; info bits 0
           0: len 6; hex 007005af066c; asc p l;;
           1: len 6; hex 000000000000; asc ;;
           2: len 7; hex 80000000000000; asc ;;
           3: len 2; hex 03b5; asc ;;
           4: len 4; hex 000146ef; asc F ;;
           5: len 8; hex 87e074b199000000; asc t ;;
           6: len 4; hex 00000000; asc ;;
           7: len 4; hex 00000000; asc ;;
           8: len 4; hex 15000000; asc ;;
           9: len 2; hex 0000; asc ;;

          TRANSACTION 421589657076240, ACTIVE 0 sec starting index read
          mysql tables in use 8, locked 0
          0 lock struct(s), heap size 1128, 0 row lock(s)
          MySQL thread id 64307678, OS thread handle 140095744042752, query id 791359259 <hostname> 192.168.255.80 statsadm Sending data
          SELECT
           <tableb>.hostname AS hostname,
           MAX( <table>.numCurrentSessions ) AS max
          FROM
           <table>,
           <tableb>,
           <tablec>
          WHERE
           <tablec>.name = @site AND
           <table>.time BETWEEN @date AND @date + INTERVAL 1 DAY AND
           <table>.siteid = <tablec>.id AND
           <tableb>.id= <table>.serverid
          GROUP BY hostname

          InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
          {code}


          {code}
          2023-10-26 15:40:30 64307291 [ERROR] [FATAL] InnoDB: Cannot insert tuple TUPLE (info_bits=0, 3 fields): {[2] (0x02E2),[5] t (0x1DDE74B199),[6] p (0x007005BDDCCA)}into index `amosSessionsInstridx` of table `statsdb`.`enm_amos_generalscripting_sessionsinstr` /* Partition `P20231101` */. Max size: 19
          231026 15:40:30 [ERROR] mysqld got signal 6 ;
          This could be because you hit a bug. It is also possible that this binary
          or one of the libraries it was linked against is corrupt, improperly built,
          or misconfigured. This error can also be caused by malfunctioning hardware.

          To report this bug, see https://mariadb.com/kb/en/reporting-bugs

          We will try our best to scrape up some info that will hopefully help
          diagnose the problem, but since we have already crashed,
          something is definitely wrong and this may fail.

          Server version: 10.5.16-MariaDB-log
          key_buffer_size=134217728
          read_buffer_size=131072
          max_used_connections=56
          max_threads=153
          thread_count=30
          It is possible that mysqld could use up to
          key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467873 K bytes of memory
          Hope that's ok; if not, decrease some variables in the equation.

          Thread pointer: 0x7f6a9c39c5a8
          Attempting backtrace. You can use the following information to find out
          where mysqld died. If you see no messages after this, something went
          terribly wrong...
          stack_bottom = 0x7f6a94b44bd8 thread_stack 0x49000
          Can't start addr2line
          /usr/libexec/mysqld(my_print_stacktrace+0x41)[0x55a8ee32add1]
          /usr/libexec/mysqld(handle_fatal_signal+0x4a5)[0x55a8eddc0355]
          /lib64/libpthread.so.0(+0x12cf0)[0x7f6f0959fcf0]
          /lib64/libc.so.6(gsignal+0x10f)[0x7f6f088fbacf]
          /lib64/libc.so.6(abort+0x127)[0x7f6f088ceea5]
          /usr/libexec/mysqld(+0x644be7)[0x55a8edaa8be7]
          /usr/libexec/mysqld(+0x6485f6)[0x55a8edaac5f6]
          /usr/libexec/mysqld(+0xce28fd)[0x55a8ee1468fd]
          /usr/libexec/mysqld(+0xce78de)[0x55a8ee14b8de]
          /usr/libexec/mysqld(+0xce7fe4)[0x55a8ee14bfe4]
          /usr/libexec/mysqld(+0xcf8bd0)[0x55a8ee15cbd0]
          /usr/libexec/mysqld(+0xc411ed)[0x55a8ee0a51ed]
          /usr/libexec/mysqld(_ZN7handler12ha_write_rowEPKh+0x177)[0x55a8eddcdee7]
          /usr/libexec/mysqld(+0xb812f0)[0x55a8edfe52f0]
          /usr/libexec/mysqld(_ZN7handler12ha_write_rowEPKh+0x177)[0x55a8eddcdee7]
          /usr/libexec/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_infoP13select_result+0x1dd)[0x55a8edb8d34d]
          /usr/libexec/mysqld(_Z10mysql_loadP3THDPK12sql_exchangeP10TABLE_LISTR4ListI4ItemES9_S9_15enum_duplicatesbb+0x17de)[0x55a8edbba71e]
          /usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x185f)[0x55a8edbcba0e]
          /usr/libexec/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x238)[0x55a8edbbc956]
          /usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x177d)[0x55a8edbc842c]
          /usr/libexec/mysqld(_Z10do_commandP3THD+0x122)[0x55a8edbc9702]
          /usr/libexec/mysqld(_Z24do_handle_one_connectionP7CONNECTb+0x4e0)[0x55a8edcbaa00]
          /usr/libexec/mysqld(handle_one_connection+0x5d)[0x55a8edcbadad]
          /usr/libexec/mysqld(+0xb882d8)[0x55a8edfec2d8]
          /lib64/libpthread.so.0(+0x81ca)[0x7f6f095951ca]
          /lib64/libc.so.6(clone+0x43)[0x7f6f088e6e73]

          Trying to get some variables.
          Some pointers may be invalid and cause the dump to abort.
          Query (0x7f6a9c04a370): LOAD DATA LOCAL INFILE '/data/tmp/makeStats_12/<table>.bcp' INTO TABLE <table> (time,serverid,siteid,cpuUsed,memoryUsed,processes,numCurrentSessions)

          Connection ID (thread ID): 64307291
          Status: NOT_KILLED

          Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

          The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
          information that should help you find out what is causing the crash.
          Writing a core file...
          Working directory at /data/db/data
          Resource Limits:
          Limit Soft Limit Hard Limit Units
          Max cpu time unlimited unlimited seconds
          Max file size unlimited unlimited bytes
          Max data size unlimited unlimited bytes
          Max stack size 8388608 unlimited bytes
          Max core file size 0 unlimited bytes
          Max resident set unlimited unlimited bytes
          Max processes 94970 94970 processes
          Max open files 160000 160000 files
          Max locked memory 65536 65536 bytes
          Max address space unlimited unlimited bytes
          Max file locks unlimited unlimited locks
          Max pending signals 94970 94970 signals
          Max msgqueue size 819200 819200 bytes
          Max nice priority 0 0
          Max realtime priority 0 0
          Max realtime timeout unlimited unlimited us
          Core pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
          {code}           		Tables are used to store time series data. Older data is removed by dropping partitions from the table. This has been in place for a number of years.

          Recently upgraded from 10.3->10.5. Since then, we've dropped a partition in two separate MariaDB instances. In both cases, after the partition was dropped, attempts to insert or delete rows from the table fails with errors like

          {code}
          2023-10-26 16:32:48 467 [ERROR] Found index <index> whose column info does not match that of MariaDB.
          2023-10-26 16:32:48 467 [ERROR] InnoDB indexes are inconsistent with what defined in .frm for table ./<db>/<table>#P#P20230901

          2023-10-26 15:38:46 64307678 [Warning] InnoDB: Using a partial-field key prefix in search, index `<index>` of table `<db>`.`<table>` /* Partition `P20231101` */. Last data field length 8 bytes, key ptr now exceeds key end by 3 bytes. Key value in the MySQL format:
           len 7; hex b50399b1740000; asc t ;

          2023-10-26 15:38:46 64307678 [ERROR] InnoDB: Clustered record for sec rec not found index `<index>` of table `<db>`.`<table>` /* Partition `P20231101` */
          InnoDB: sec index record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
           0: len 2; hex 03b5; asc ;;
           1: len 8; hex 87c174b199007005; asc t p ;;
           2: len 6; hex af05f8af05e2; asc ;;

          InnoDB: clust index record PHYSICAL RECORD: n_fields 10; compact format; info bits 0
           0: len 6; hex 007005af066c; asc p l;;
           1: len 6; hex 000000000000; asc ;;
           2: len 7; hex 80000000000000; asc ;;
           3: len 2; hex 03b5; asc ;;
           4: len 4; hex 000146ef; asc F ;;
           5: len 8; hex 87e074b199000000; asc t ;;
           6: len 4; hex 00000000; asc ;;
           7: len 4; hex 00000000; asc ;;
           8: len 4; hex 15000000; asc ;;
           9: len 2; hex 0000; asc ;;

          TRANSACTION 421589657076240, ACTIVE 0 sec starting index read
          mysql tables in use 8, locked 0
          0 lock struct(s), heap size 1128, 0 row lock(s)
          MySQL thread id 64307678, OS thread handle 140095744042752, query id 791359259 <hostname> 192.168.255.80 statsadm Sending data
          SELECT
           <tableb>.hostname AS hostname,
           MAX( <table>.numCurrentSessions ) AS max
          FROM
           <table>,
           <tableb>,
           <tablec>
          WHERE
           <tablec>.name = @site AND
           <table>.time BETWEEN @date AND @date + INTERVAL 1 DAY AND
           <table>.siteid = <tablec>.id AND
           <tableb>.id= <table>.serverid
          GROUP BY hostname

          InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
          {code}


          {code}
          2023-10-26 15:40:30 64307291 [ERROR] [FATAL] InnoDB: Cannot insert tuple TUPLE (info_bits=0, 3 fields): {[2] (0x02E2),[5] t (0x1DDE74B199),[6] p (0x007005BDDCCA)}into index `<index>` of table `<db>`.`<table>` /* Partition `P20231101` */. Max size: 19
          231026 15:40:30 [ERROR] mysqld got signal 6 ;
          This could be because you hit a bug. It is also possible that this binary
          or one of the libraries it was linked against is corrupt, improperly built,
          or misconfigured. This error can also be caused by malfunctioning hardware.

          To report this bug, see https://mariadb.com/kb/en/reporting-bugs

          We will try our best to scrape up some info that will hopefully help
          diagnose the problem, but since we have already crashed,
          something is definitely wrong and this may fail.

          Server version: 10.5.16-MariaDB-log
          key_buffer_size=134217728
          read_buffer_size=131072
          max_used_connections=56
          max_threads=153
          thread_count=30
          It is possible that mysqld could use up to
          key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467873 K bytes of memory
          Hope that's ok; if not, decrease some variables in the equation.

          Thread pointer: 0x7f6a9c39c5a8
          Attempting backtrace. You can use the following information to find out
          where mysqld died. If you see no messages after this, something went
          terribly wrong...
          stack_bottom = 0x7f6a94b44bd8 thread_stack 0x49000
          Can't start addr2line
          /usr/libexec/mysqld(my_print_stacktrace+0x41)[0x55a8ee32add1]
          /usr/libexec/mysqld(handle_fatal_signal+0x4a5)[0x55a8eddc0355]
          /lib64/libpthread.so.0(+0x12cf0)[0x7f6f0959fcf0]
          /lib64/libc.so.6(gsignal+0x10f)[0x7f6f088fbacf]
          /lib64/libc.so.6(abort+0x127)[0x7f6f088ceea5]
          /usr/libexec/mysqld(+0x644be7)[0x55a8edaa8be7]
          /usr/libexec/mysqld(+0x6485f6)[0x55a8edaac5f6]
          /usr/libexec/mysqld(+0xce28fd)[0x55a8ee1468fd]
          /usr/libexec/mysqld(+0xce78de)[0x55a8ee14b8de]
          /usr/libexec/mysqld(+0xce7fe4)[0x55a8ee14bfe4]
          /usr/libexec/mysqld(+0xcf8bd0)[0x55a8ee15cbd0]
          /usr/libexec/mysqld(+0xc411ed)[0x55a8ee0a51ed]
          /usr/libexec/mysqld(_ZN7handler12ha_write_rowEPKh+0x177)[0x55a8eddcdee7]
          /usr/libexec/mysqld(+0xb812f0)[0x55a8edfe52f0]
          /usr/libexec/mysqld(_ZN7handler12ha_write_rowEPKh+0x177)[0x55a8eddcdee7]
          /usr/libexec/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_infoP13select_result+0x1dd)[0x55a8edb8d34d]
          /usr/libexec/mysqld(_Z10mysql_loadP3THDPK12sql_exchangeP10TABLE_LISTR4ListI4ItemES9_S9_15enum_duplicatesbb+0x17de)[0x55a8edbba71e]
          /usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x185f)[0x55a8edbcba0e]
          /usr/libexec/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x238)[0x55a8edbbc956]
          /usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x177d)[0x55a8edbc842c]
          /usr/libexec/mysqld(_Z10do_commandP3THD+0x122)[0x55a8edbc9702]
          /usr/libexec/mysqld(_Z24do_handle_one_connectionP7CONNECTb+0x4e0)[0x55a8edcbaa00]
          /usr/libexec/mysqld(handle_one_connection+0x5d)[0x55a8edcbadad]
          /usr/libexec/mysqld(+0xb882d8)[0x55a8edfec2d8]
          /lib64/libpthread.so.0(+0x81ca)[0x7f6f095951ca]
          /lib64/libc.so.6(clone+0x43)[0x7f6f088e6e73]

          Trying to get some variables.
          Some pointers may be invalid and cause the dump to abort.
          Query (0x7f6a9c04a370): LOAD DATA LOCAL INFILE '/data/tmp/makeStats_12/<table>.bcp' INTO TABLE <table> (time,serverid,siteid,cpuUsed,memoryUsed,processes,numCurrentSessions)

          Connection ID (thread ID): 64307291
          Status: NOT_KILLED

          Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

          The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
          information that should help you find out what is causing the crash.
          Writing a core file...
          Working directory at /data/db/data
          Resource Limits:
          Limit Soft Limit Hard Limit Units
          Max cpu time unlimited unlimited seconds
          Max file size unlimited unlimited bytes
          Max data size unlimited unlimited bytes
          Max stack size 8388608 unlimited bytes
          Max core file size 0 unlimited bytes
          Max resident set unlimited unlimited bytes
          Max processes 94970 94970 processes
          Max open files 160000 160000 files
          Max locked memory 65536 65536 bytes
          Max address space unlimited unlimited bytes
          Max file locks unlimited unlimited locks
          Max pending signals 94970 94970 signals
          Max msgqueue size 819200 819200 bytes
          Max nice priority 0 0
          Max realtime priority 0 0
          Max realtime timeout unlimited unlimited us
          Core pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
          {code}
          conormarkmurphy Conor Murphy added a comment -

          Dropping and re-creating the index seems to stop the errors/crashes.

          conormarkmurphy Conor Murphy added a comment - Dropping and re-creating the index seems to stop the errors/crashes.
          serg Sergei Golubchik made changes -
          Assignee Matthias Leich [ mleich ]
          mleich Matthias Leich added a comment -

          Hi Conor,
          would you please send the insert statement and the SHOW CREATE TABLE .... output for all involved base tables?
          Many thanks in advance

          mleich Matthias Leich added a comment - Hi Conor, would you please send the insert statement and the SHOW CREATE TABLE .... output for all involved base tables? Many thanks in advance
          conormarkmurphy Conor Murphy added a comment -

          Issue was resolved by updating to 10.5.22

          conormarkmurphy Conor Murphy added a comment - Issue was resolved by updating to 10.5.22
          serg Sergei Golubchik made changes -
          Fix Version/s N/A [ 14700 ]
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Closed [ 6 ]
          serg Sergei Golubchik made changes -
          Fix Version/s 10.5.22 [ 29011 ]
          Fix Version/s N/A [ 14700 ]

          People

            mleich Matthias Leich
            conormarkmurphy Conor Murphy
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.