[MCOL-4364] LOAD DATA crashes mariadb process Created: 2020-10-16  Updated: 2021-04-19  Resolved: 2020-11-02

Status: Closed
Project: MariaDB ColumnStore
Component/s: MariaDB Server
Affects Version/s: 1.4.4
Fix Version/s: 5.4.3

Type: Bug Priority: Blocker
Reporter: Rick Pizzi Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 1
Labels: None

Issue Links:
Problem/Incident
is caused by MCOL-4005 mishandling multi-byte chars in DML e... Closed
Relates
relates to MCOL-4122 mariadb server crash on INSERT Closed
relates to MCOL-4320 INSERT..SELECT doesn't work from Inno... Closed

 Description   

Inserting one line of data with load data causes mariadb server to crash with the below stacktrace.

stack_bottom = 0x7ff7840a3c38 thread_stack 0x49000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c4492bd3ee]
 
/usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c448d3daef]
 
sigaction.c:0(__restore_rt)[0x7ff7c4b99630]
 
:0(__memcpy_ssse3_back)[0x7ff7c2d1ea85]
 
/lib64/libstdc++.so.6(_ZNSs15_M_replace_safeEmmPKcm+0x3b)[0x7ff7c33581bb]
 
/usr/lib64/mysql/plugin/ha_columnstore.so(_Z28ha_mcs_impl_write_batch_row_PKhP5TABLERN11cal_impl_if19cal_connection_infoE+0x1125)[0x7ff7842683d5]
 
/usr/lib64/mysql/plugin/ha_columnstore.so(_Z21ha_mcs_impl_write_rowPKhP5TABLEm+0x2cc)[0x7ff78423f8cc]
 
/usr/lib64/mysql/plugin/ha_columnstore.so(_ZN6ha_mcs9write_rowEPKh+0x1f)[0x7ff78423a7cf]
 
/usr/sbin/mysqld(_ZN7handler12ha_write_rowEPKh+0x16d)[0x55c448d4903d]
 
/usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x69)[0x55c448afa949]
 
/usr/sbin/mysqld(_Z10mysql_loadP3THDPK12sql_exchangeP10TABLE_LISTR4ListI4ItemES9_S9_15enum_duplicatesbb+0x1ba0)[0x55c448b23b50]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x59a6)[0x55c448b324c6]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x36d)[0x55c448b3605d]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x2209)[0x55c448b38a39]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x109)[0x55c448b39e99]
 
/usr/sbin/mysqld(_Z11tp_callbackP13TP_connection+0xd2)[0x55c448d0f902]
 
/usr/sbin/mysqld(+0xa5b3d0)[0x55c448ecc3d0]
 
/usr/sbin/mysqld(+0xdfd8ed)[0x55c44926e8ed]
 
pthread_create.c:0(start_thread)[0x7ff7c4b91ea5]
 
/lib64/libc.so.6(clone+0x6d)[0x7ff7c2cc88dd]

Command used:

LOAD DATA LOCAL INFILE 'Rick2.txt'  INTO TABLE mytable  FIELDS OPTIONALLY ENCLOSED BY '"' TERMINATED BY '#' LINES TERMINATED BY ''

Table structure:

CREATE TABLE `mytable` (
 
  `Id` bigint(20) NOT NULL,
 
  `redacted` bigint(20) unsigned DEFAULT NULL,
 
  `redacted` int(11) NOT NULL,
 
  `redacted` int(11) DEFAULT NULL,
 
  `redacted` int(11) DEFAULT NULL,
 
  `redacted` bigint(20) DEFAULT NULL,
 
  `redacted` int(11) NOT NULL,
 
  `redacted` bigint(20) NOT NULL,
 
  `redacted` bigint(20) NOT NULL,
 
  `redacted` int(4) unsigned DEFAULT NULL,
 
  `redacted` int(4) unsigned DEFAULT NULL,
 
  `redacted` int(4) unsigned DEFAULT NULL,
 
  `redacted` int(4) unsigned DEFAULT NULL,
 
  `redacted` int(4) unsigned DEFAULT NULL,
 
  `redacted` varchar(63) DEFAULT NULL,
 
  `redacted` decimal(18,8) NOT NULL,
 
  `redacted` decimal(18,8) NOT NULL,
 
  `redacted` tinyint(1) NOT NULL,
 
  `redacted` int(11) NOT NULL DEFAULT 0,
 
  `redacted` decimal(18,8) NOT NULL DEFAULT 0.00000000,
 
  `redacted` decimal(18,8) NOT NULL,
 
  `redacted` int(11) DEFAULT NULL,
 
  `redacted` int(11) DEFAULT NULL,
 
  `redacted` varchar(255) DEFAULT NULL,
 
  `redacted` decimal(18,8) DEFAULT 0.00000000,
 
  `redacted` decimal(18,8) DEFAULT NULL,
 
  `redacted` decimal(18,8) DEFAULT NULL,
 
  `redacted` decimal(5,2) DEFAULT NULL,
 
  `redacted` decimal(5,2) DEFAULT NULL,
 
  `redacted` decimal(8,4) DEFAULT NULL,
 
  `redacted` decimal(5,2) DEFAULT NULL,
 
  `redacted` decimal(8,2) DEFAULT NULL,
 
  `redacted` decimal(6,4) DEFAULT NULL,
 
  `redacted` int(3) DEFAULT NULL,
 
  `redacted` int(11) NOT NULL,
 
  `redacted` text DEFAULT NULL,
 
  `redacted` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
 
  `redacted` text DEFAULT NULL,
 
  `redacted` text DEFAULT NULL
 
) ENGINE=Columnstore DEFAULT CHARSET=utf8

Data row that makes it crash:

3871250004#955385077888386499#131083#249881#171605#3826160523#5003#1577836919000#1577836919000#1#1#2020#0#3#"5141360283FC07BF"#40.54220581#-3.45858359#1#0#0.00001526#1154.44339348#NULL#NULL#""#0.00000000#NULL#NULL#NULL#NULL#NULL#NULL#NULL#NULL#NULL#37#"[]"#"2020-01-01 00:01:59"#NULL#01000000E066454440000000C02DAB0BC0D83B685E6F0100000000000000000000000000E066454440000000C02DAB0BC0604F685E6F0100000000000000000000000000E066454440000000C02DAB0BC0E862685E6F0100000000000000000000000000E066454440000000E02DAB0BC07076685E6F01000000000000000000000000000067454440000000E02DAB0BC0F889685E6F01000000000000000000000000000067454440000000E02DAB0BC0809D685E6F01000000000000000000000000000067454440000000E02DAB0BC008B1685E6F0100000000000000000000

The customer is unable to load data from application because of this bug, hence a blocker. cpimport can't be used because requires data to be on CS server.

 Comments   
Comment by Rick Pizzi [ 2020-10-16 ]

Additional information:

The customer says that wrapping the LOAD DATA inside a transaction avoids the crash (although this doesn't make sense)

Comment by David Hall (Inactive) [ 2020-10-16 ]

What is LINES TERMINATED BY ''? How can you terminate with nothing? Might have confused it. Not saying this is the problem, it's just something to investigate.

Comment by David Hall (Inactive) [ 2020-10-16 ]

Transactional LDI treats the data as a series of transactional inserts, while non-transactional LDI uses bulk insert. The two are different code paths, so that explains the discrepancy.

Comment by Rick Pizzi [ 2020-10-19 ]

Actual load data statement is

LOAD DATA LOCAL INFILE 'Rick2.txt' INTO TABLE mytable FIELDS OPTIONALLY ENCLOSED BY '"' TERMINATED BY '#' LINES TERMINATED BY '\n';

Comment by David Hall (Inactive) [ 2020-10-20 ]

As a workaround, assign an explicit length to text fields (something less than 21845).
Ex.
`redacted34` text(2000) DEFAULT NULL,
`redacted35` timestamp NOT NULL,
`redacted36` text(2000) DEFAULT NULL,
`redacted37` text(2000) DEFAULT NULL

Comment by Rick Pizzi [ 2020-10-20 ]

Don't think that is a proper workaround. A TEXT can be much longer than 21845 or whatever value we assign, so this workaround kinda defeats the purpose of a TEXT column IMHO.

Comment by David Hall (Inactive) [ 2020-10-21 ]

Well, another option is (size > 65536 AND < 5592405) OR size > 16777216. Or, don't use UTF8, if possible. Then the bug goes away.
We are working on a fix, this is just some ideas to get by.

Comment by Rick Pizzi [ 2020-10-22 ]

All these workarounds are unapplicable. Customer can't control content of source data. And cannot avoid UTF8, which is today's de facto standard. It has been suggested by PS to go to Columnstore 5, is this fixed in that release?

Comment by Gagan Goel (Inactive) [ 2020-10-26 ]

For QA: Steps to reproduce and ensure the crash is fixed:

MariaDB [test]> drop table if exists mcol4364_cs;
 
Query OK, 0 rows affected (0.795 sec)
 
 
 
MariaDB [test]> CREATE TABLE `mcol4364_cs` (
 
    ->   `Id` bigint(20) NOT NULL,
 
    ->   `redacted00` bigint(20) unsigned DEFAULT NULL,
 
    ->   `redacted01` int(11) NOT NULL,
 
    ->   `redacted02` int(11) DEFAULT NULL,
 
    ->   `redacted03` int(11) DEFAULT NULL,
 
    ->   `redacted04` bigint(20) DEFAULT NULL,
 
    ->   `redacted05` int(11) NOT NULL,
 
    ->   `redacted06` bigint(20) NOT NULL,
 
    ->   `redacted07` bigint(20) NOT NULL,
 
    ->   `redacted08` int(4) unsigned DEFAULT NULL,
 
    ->   `redacted09` int(4) unsigned DEFAULT NULL,
 
    ->   `redacted10` int(4) unsigned DEFAULT NULL,
 
    ->   `redacted11` int(4) unsigned DEFAULT NULL,
 
    ->   `redacted12` int(4) unsigned DEFAULT NULL,
 
    ->   `redacted13` varchar(63) DEFAULT NULL,
 
    ->   `redacted14` decimal(18,8) NOT NULL,
 
    ->   `redacted15` decimal(18,8) NOT NULL,
 
    ->   `redacted16` tinyint(1) NOT NULL,
 
    ->   `redacted17` int(11) NOT NULL DEFAULT 0,
 
    ->   `redacted18` decimal(18,8) NOT NULL DEFAULT 0.00000000,
 
    ->   `redacted19` decimal(18,8) NOT NULL,
 
    ->   `redacted20` int(11) DEFAULT NULL,
 
    ->   `redacted21` int(11) DEFAULT NULL,
 
    ->   `redacted22` varchar(255) DEFAULT NULL,
 
    ->   `redacted23` decimal(18,8) DEFAULT 0.00000000,
 
    ->   `redacted24` decimal(18,8) DEFAULT NULL,
 
    ->   `redacted25` decimal(18,8) DEFAULT NULL,
 
    ->   `redacted26` decimal(5,2) DEFAULT NULL,
 
    ->   `redacted27` decimal(5,2) DEFAULT NULL,
 
    ->   `redacted28` decimal(8,4) DEFAULT NULL,
 
    ->   `redacted29` decimal(5,2) DEFAULT NULL,
 
    ->   `redacted30` decimal(8,2) DEFAULT NULL,
 
    ->   `redacted31` decimal(6,4) DEFAULT NULL,
 
    ->   `redacted32` int(3) DEFAULT NULL,
 
    ->   `redacted33` int(11) NOT NULL,
 
    ->   `redacted34` text DEFAULT NULL,
 
    ->   `redacted35` timestamp NOT NULL,
 
    ->   `redacted36` text DEFAULT NULL,
 
    ->   `redacted37` text DEFAULT NULL
 
    -> ) engine=columnstore DEFAULT CHARSET=utf8;
 
Query OK, 0 rows affected (0.947 sec)
 
 
 
MariaDB [test]> LOAD DATA LOCAL INFILE '/tmp/mcol4364.txt' INTO TABLE mcol4364_cs FIELDS TERMINATED BY ',' ENCLOSED BY '"';
 
ERROR 2013 (HY000): Lost connection to MySQL server during query


root@tntnatbry-3:~# cat /tmp/mcol4364.txt 
"3871250004","955385077888386499","131083","249881","171605","3826160523","5003","1577836919000","1577836919000","1","1","2020","0","3","5141360283FC07BF","40.54220581","-3.45858359","1","0","0.00001526","1154.44339348",\N,\N,"value","0.00000000",\N,\N,\N,\N,\N,\N,\N,\N,\N,"37","[]","2020-01-01 00:01:59",\N,"01000000E066454440000000C02DAB0BC0D83B685E6F0100000000000000000000000000E066454440000000C02DAB0BC0604F685E6F0100000000000000000000000000E066454440000000C02DAB0BC0E862685E6F0100000000000000000000000000E066454440000000E02DAB0BC07076685E6F01000000000000000000000000000067454440000000E02DAB0BC0F889685E6F01000000000000000000000000000067454440000000E02DAB0BC0809D685E6F01000000000000000000000000000067454440000000E02DAB0BC008B1685E6F0100000000000000000000"

Comment by David Hall (Inactive) [ 2020-11-02 ]

While not identical, the cause is similar enough to MCOL-4320 such that the fixes for this are included in the Pull Requests of MCOL-4320.

Comment by Daniel Lee (Inactive) [ 2020-11-02 ]

Build verified: 5.5.1-1 (Drone #1036)

Reproduced the issue in release 5.4.1-1

MariaDB [mytest]> LOAD DATA LOCAL INFILE '/tmp/t.txt' INTO TABLE mcol4364_cs FIELDS TERMINATED BY ',' ENCLOSED BY '"';
ERROR 2013 (HY000): Lost connection to MySQL server during query

Verified the issue in 5.5.1-1

MariaDB [mytest]> LOAD DATA LOCAL INFILE '/tmp/t.txt' INTO TABLE mcol4364_cs FIELDS TERMINATED BY ',' ENCLOSED BY '"';
Query OK, 1 row affected (1.273 sec)
Records: 1 Deleted: 0 Skipped: 0 Warnings: 0

Comment by Daniel Lee (Inactive) [ 2020-11-05 ]

Build verified: 5.4.2-1 hot fix (Enterprise Jenkins build #523)

Server version: 10.5.6-4.5.4.2-MariaDB-enterprise MariaDB Enterprise Server

Comment by Daniel Lee (Inactive) [ 2020-11-09 ]

Build verified: 5.4.3-1 hot fix (Enterprise Jenkins build #854)
Server version: 10.5.6-4.5.4.3-MariaDB-enterprise MariaDB Enterprise Server

Generated at Thu Feb 08 02:49:47 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.