Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-41425

DELETE and UPDATE on a CONNECT CSV table of 4300000100 bytes give an ftell error and cut the file to 5032704 bytes

    XMLWordPrintable

Details

    Description

      --let $c= query_get_value(SELECT COUNT(*) AS c FROM information_schema.ENGINES WHERE ENGINE='CONNECT' AND SUPPORT='YES', c, 1)
      if (!$c)
      {
        --skip Needs the CONNECT engine
      }
      # t1.csv holds one line with a=2, then 43000000 lines with a=1, 100 bytes each
      --perl
      open(my $f, ">", "$ENV{MYSQLTEST_VARDIR}/tmp/t1.csv") or die;
      print $f "2," . ("b" x 97) . "\n";
      my $s= ("1," . ("a" x 97) . "\n") x 10000;
      print $f $s for 1..4300;
      close($f);
      EOF
      --replace_result $MYSQLTEST_VARDIR MYSQLTEST_VARDIR
      --eval CREATE TABLE t1 (a INT NOT NULL, b CHAR(97) NOT NULL) ENGINE=CONNECT TABLE_TYPE=CSV FILE_NAME='$MYSQLTEST_VARDIR/tmp/t1.csv' SEP_CHAR=',' HUGE=1
      --error 0,ER_GET_ERRMSG
      DELETE FROM t1 WHERE a = 2;
      DROP TABLE t1;
      # The size of t1.csv after the DELETE
      --perl
      my $n= -s "$ENV{MYSQLTEST_VARDIR}/tmp/t1.csv";
      open(my $f, ">", "$ENV{MYSQLTEST_VARDIR}/tmp/t1.inc") or die;
      print $f "--let \$n= $n\n";
      close($f);
      EOF
      --source $MYSQLTEST_VARDIR/tmp/t1.inc
      --remove_file $MYSQLTEST_VARDIR/tmp/t1.inc
      --remove_file $MYSQLTEST_VARDIR/tmp/t1.csv
      --let $cut= query_get_value(SELECT $n <> 4300000000 AS cut, cut, 1)
      if ($cut)
      {
        --die The DELETE of one 100 byte row from a CONNECT CSV table over a 4300000100 byte file left a file of $n bytes, not 4300000000
      }

      With this .cnf file beside it:

      !include include/default_my.cnf
       
      [mysqld.1]
      plugin-load-add=ha_connect

      Leads to:

      CS 13.1.2 87e62ca17270e163a6d41b3fc8d6f4d440f1bb88 (Optimized, Clang 22.1.8-20260622) Build 05/10/2026

      mysqltest: At line 32: The DELETE of one 100 byte row from a CONNECT CSV table over a 4300000100 byte file left a file of 5032704 bytes, not 4300000000

      CS 13.1.0 da18481158c81ca94689702073c3e04aad85a6a3 (Debug, Clang 22.1.8-20260622) Build 18/08/2026

      mysqltest: At line 32: The DELETE of one 100 byte row from a CONNECT CSV table over a 4300000100 byte file left a file of 5032704 bytes, not 4300000000

      CS 10.11.19 8f00e6caca633c783140db86d3a48a96de67cf38 (Optimized, Clang 22.1.8-20260622) Build 18/08/2026

      mysqltest: At line 32: The DELETE of one 100 byte row from a CONNECT CSV table over a 4300000100 byte file left a file of 5032704 bytes, not 4300000000

      The same test with 21470000 lines (a file of 2147000100 bytes) passes. In the client, the DELETE gives an error, but the file is already cut. 5032704 bytes is 4300000100 minus 4294967296, less the deleted row. An UPDATE of the same row leaves 5032708 bytes.

      CS 13.1.2 87e62ca17270e163a6d41b3fc8d6f4d440f1bb88 (Optimized, Clang 22.1.8-20260622) Build 05/10/2026

      CREATE TABLE t1 (a INT NOT NULL, b CHAR(97) NOT NULL) ENGINE=CONNECT TABLE_TYPE=CSV FILE_NAME='.../c99.csv' SEP_CHAR=',' HUGE=1
      Query OK, 0 rows affected (0.013 sec)
       
      DELETE FROM t1 WHERE a = 2
      ERROR 1296 (HY000) at line 4: Got error 122 'ftell error for recd=0: Invalid argument' from CONNECT

      With a file of 2148000100 bytes, so between 2 GB and 4 GB, SELECT fails with "Error in _filelength" on 13.1 and on 10.11. DELETE and UPDATE differ by version. On 11.4 and later, they affect 0 rows with no error, and the file is not changed:

      CS 13.1.2 87e62ca17270e163a6d41b3fc8d6f4d440f1bb88 (Optimized, Clang 22.1.8-20260622) Build 05/10/2026

      CREATE TABLE t1 (a INT NOT NULL, b CHAR(97) NOT NULL) ENGINE=CONNECT TABLE_TYPE=CSV FILE_NAME='.../c37.csv' SEP_CHAR=',' HUGE=1
      Query OK, 0 rows affected (0.010 sec)
       
      SELECT COUNT(*) FROM t1
      ERROR 1296 (HY000) at line 4: Got error 174 'Error in _filelength for .../c37.csv' from CONNECT
       
      SELECT a, LEFT(b, 3) FROM t1 WHERE a = 2
      ERROR 1296 (HY000) at line 6: Got error 174 'Error in _filelength for .../c37.csv' from CONNECT
       
      DELETE FROM t1 WHERE a = 2
      Query OK, 0 rows affected (0.000 sec)
       
      SHOW WARNINGS
      Empty set (0.000 sec)

      Without HUGE=1, the result is the same.

      On 10.11, DELETE and UPDATE report 1 row affected with no error, and the file loses the other rows. After the DELETE of the first row the file is 0 bytes. After an UPDATE of that row it is 4 bytes, "2,x" and the line end:

      CS 10.11.19 8f00e6caca633c783140db86d3a48a96de67cf38 (Optimized, Clang 22.1.8-20260622) Build 18/08/2026

      CREATE TABLE t1 (a INT NOT NULL, b CHAR(97) NOT NULL) ENGINE=CONNECT TABLE_TYPE=CSV FILE_NAME='.../c37.csv' SEP_CHAR=','
      Query OK, 0 rows affected (0.009 sec)
       
      UPDATE t1 SET b='x' WHERE a = 2
      Query OK, 1 row affected (15.154 sec)
      Rows matched: 1  Changed: 1  Warnings: 0
       
      SHOW WARNINGS
      Empty set (0.000 sec)

      The error log at the same time:

      CS 10.11.19 8f00e6caca633c783140db86d3a48a96de67cf38 (Optimized, Clang 22.1.8-20260622) Build 18/08/2026

      rnd_next CONNECT: ftell error for recd=0: No such file or directory

      From the code it seems that TXTFAM::GetFileLength() stores the unsigned long result of _filelength() (storage/connect/osutil.c:125) in an int (storage/connect/filamtxt.cpp:195). For a file of 4300000100 bytes this is 5032804. For a file of 2148000100 bytes it is negative, which gives the "Error in _filelength" error (storage/connect/filamtxt.cpp:215). DOSFAM::RecordPos() stores the ftell() result in the int Fpos (storage/connect/filamtxt.cpp:700), which gives the "ftell error" past 2147483647 bytes. The other positions that DELETE and UPDATE use to move the rows of the file are int too (storage/connect/filamtxt.h:116 to storage/connect/filamtxt.h:118). HUGE=1 is used for fixed format files only (storage/connect/tabdos.cpp:414), and a CSV file uses DOSFAM (storage/connect/tabdos.cpp:444).

      For the 2148000100 byte file on 13.1: ha_connect::info() returns HA_ERR_INITIALIZATION after the _filelength error (storage/connect/ha_connect.cc:4383) and does not set stats.records. mysql_delete() does not check the return value of info() (sql/sql_delete.cc:613). With stat_records() at 0, it handles the WHERE as impossible (sql/sql_delete.cc:624), and UPDATE does the same (sql/sql_update.cc:523). These checks came with MDEV-30256 in 11.0, so 10.11 reads the file.

      Attachments

        Issue Links

          Activity

            People

              danblack Daniel Black
              Roel Roel Van de Paar
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 1h
                  1h

                  Git Integration

                    Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.