[MCOL-3392] select * from table : Internal error: IDB-2039: Data file does not exist Created: 2019-06-24  Updated: 2023-03-06  Resolved: 2023-03-06

Status: Closed
Project: MariaDB ColumnStore
Component/s: N/A
Affects Version/s: 1.2.4
Fix Version/s: Icebox

Type: Bug Priority: Major
Reporter: Nicola Assignee: Unassigned
Resolution: Won't Do Votes: 1
Labels: None
Environment:

3PM and 2UM with glusterfs

Attachments: File columnstoreSupportReport.COLUMNSTORE-DWH.tar.gz     PNG File screenshot-1.png     Zip Archive strace_logs_pm_01_02_03.zip    
Issue Links:
PartOf
includes MCOL-1853 ColumnStore breaks on S3 NFS mounts Closed

 Description   

Hi,
I make the simple query
select * from audit;

after about 1,3M of the records processed i received this error :

Jun 24 12:01:08 cstore-pm01 PrimProc[2394]: 08.237188 |0|0|0| C 28 CAL0053: PrimProc could not open file for OID 3904; /000.dir/000.dir/015.dir/064.dir/008.dir/FILE002.cdf:No such file or directory
Jun 24 12:03:40 cstore-pm01 PrimProc[2394]: 40.766674 |0|0|0| C 28 CAL0053: PrimProc could not open file for OID 3904; /000.dir/000.dir/015.dir/064.dir/008.dir/FILE002.cdf:No such file or directory
Jun 24 12:05:15 cstore-pm01 PrimProc[2394]: 15.721891 |0|0|0| C 28 CAL0053: PrimProc could not open file for OID 3904; /000.dir/000.dir/015.dir/064.dir/008.dir/FILE002.cdf:No such file or directory

But this file exist :
/usr/local/mariadb/columnstore/gluster/brick2/000.dir/000.dir/015.dir/064.dir/008.dir/FILE002.cdf

Why not find this file?
Thanks,
Regards.
Nicola Battista

 Comments   
Comment by Nicola [ 2019-06-28 ]

Hi,
Any news?

Thanks,
Regards
Nicola Battista.

Comment by Nicola [ 2019-07-04 ]

Hi,
Any news?

Thanks,
Regards
Nicola Battista.

Comment by Nicola [ 2019-07-19 ]

MariaDB [(none)]> select * from information_schema.columnstore_files where object_id=3904
-> ;
-------------------------------------------------------------------------------------------------------------------------------------------------------+

OBJECT_ID SEGMENT_ID PARTITION_ID FILENAME FILE_SIZE COMPRESSED_DATA_SIZE

-------------------------------------------------------------------------------------------------------------------------------------------------------+

3904 2 0 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/000.dir/FILE002.cdf 1946427392 1479442432
3904 0 1 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/001.dir/FILE000.cdf 1677991936 1227800576
3904 0 2 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/002.dir/FILE000.cdf 537141248 405602304
3904 0 0 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/000.dir/FILE000.cdf 1745100800 1314177024
3904 1 0 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/000.dir/FILE001.cdf 1342447616 923369472
3904 0 3 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/003.dir/FILE000.cdf 805576704 606117888
3904 0 4 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/004.dir/FILE000.cdf 1946427392 1488052224
3904 1 1 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/001.dir/FILE001.cdf 2013536256 1519714304
3904 0 5 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/005.dir/FILE000.cdf 2080645120 1555505152
3904 2 1 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/001.dir/FILE002.cdf 2080645120 1544241152
3904 1 2 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/002.dir/FILE001.cdf 2080645120 1568161792
3904 0 6 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/006.dir/FILE000.cdf 2080645120 1555390464
3904 1 3 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/003.dir/FILE001.cdf 2080645120 1542496256
3904 2 2 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/002.dir/FILE002.cdf 2080645120 1565884416
3904 0 7 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/007.dir/FILE000.cdf 2080645120 1548328960
3904 1 4 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/004.dir/FILE001.cdf 2147753984 1651367936
3904 2 3 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/003.dir/FILE002.cdf 2147753984 1641857024
3904 0 8 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/008.dir/FILE000.cdf 2214862848 1671102464
3904 1 5 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/005.dir/FILE001.cdf 2080645120 1560608768
3904 0 9 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/009.dir/FILE000.cdf 2080645120 1578786816
3904 2 4 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/004.dir/FILE002.cdf 1141121024 861323264
3904 1 7 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/007.dir/FILE001.cdf 2147753984 1607196672
3904 1 8 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/008.dir/FILE001.cdf 2147753984 1605926912
3904 0 10 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/010.dir/FILE000.cdf 2147753984 1623785472
3904 0 11 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/011.dir/FILE000.cdf 2147753984 1653661696

| 3904 | 2 | 8 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/008.dir/FILE002.cdf | 2147753984 | 1646280704 |

3904 1 9 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/009.dir/FILE001.cdf 2013536256 1522106368
3904 1 10 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/010.dir/FILE001.cdf 2013536256 1505804288
3904 0 13 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/013.dir/FILE000.cdf 2080645120 1565417472
3904 2 10 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/010.dir/FILE002.cdf 2080645120 1576329216
3904 1 11 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/011.dir/FILE001.cdf 2013536256 1495556096
3904 0 14 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/014.dir/FILE000.cdf 2013536256 1504559104
3904 2 11 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/011.dir/FILE002.cdf 1946427392 1488633856
3904 0 15 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/015.dir/FILE000.cdf 1812209664 1369063424
3904 1 12 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/012.dir/FILE001.cdf 1946427392 1468047360
3904 0 12 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/012.dir/FILE000.cdf 1946427392 1483374592
3904 2 9 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/009.dir/FILE002.cdf 2013536256 1511571456
3904 2 12 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/012.dir/FILE002.cdf 2013536256 1541857280
3904 0 16 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/016.dir/FILE000.cdf 2013536256 1527414784
3904 1 13 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/013.dir/FILE001.cdf 1946427392 1470980096
3904 2 13 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/013.dir/FILE002.cdf 2080645120 1571250176
3904 0 17 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/017.dir/FILE000.cdf 2080645120 1595531264
3904 1 14 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/014.dir/FILE001.cdf 1879318528 1389658112
3904 2 14 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/014.dir/FILE002.cdf 2013536256 1539735552
3904 0 18 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/018.dir/FILE000.cdf 2013536256 1533771776
3904 1 16 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/016.dir/FILE001.cdf 2013536256 1527177216
3904 1 15 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/015.dir/FILE001.cdf 2013536256 1514790912
3904 0 19 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/019.dir/FILE000.cdf 2080645120 1602781184
3904 0 20 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/020.dir/FILE000.cdf 2080645120 1571987456
3904 2 16 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/016.dir/FILE002.cdf 1946427392 1481867264
3904 1 17 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/017.dir/FILE001.cdf 2013536256 1518739456
3904 0 21 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/021.dir/FILE000.cdf 2080645120 1559232512
3904 2 17 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/017.dir/FILE002.cdf 2013536256 1507934208
3904 1 18 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/018.dir/FILE001.cdf 1946427392 1465122816
3904 0 22 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/022.dir/FILE000.cdf 2013536256 1526349824
3904 2 18 /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/018.dir/FILE002.cdf 1141121024 866205696
3904 1 19 /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/019.dir/FILE001.cdf 738467840 527532032
3904 0 23 /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/023.dir/FILE000.cdf 537141248 385040384

-------------------------------------------------------------------------------------------------------------------------------------------------------+
58 rows in set (0.983 sec)

Comment by Nicola [ 2019-08-29 ]

Hi all,
together Mariadb ColumnStore development team, i'have opened the ticket at glusterfs, because we thought it was a glusterfs problem.
They examined all the trace logs and came to the conclusion that this is not a glusterfs problem.

This is the link of the ticket :
https://bugzilla.redhat.com/show_bug.cgi?id=1744883

Finally they asked me:
Is there a chance of races in the application where a thread tries to access the file before a creat() from another thread?

Can you help me?
Thanks
Regards
Nicola Battista

Comment by Nicola [ 2019-12-16 ]

Hi all,
we have the same problem with another table :

Log PM columnstore :
Dec 16 12:43:03 cstore-pm01 PrimProc[76079]: 03.775764 |0|0|0| C 28 CAL0000: 0 0 66805
Dec 16 12:43:03 cstore-pm01 PrimProc[76079]: 03.775973 |0|0|0| C 28 CAL0000: 0 0 66806
Dec 16 12:43:03 cstore-pm01 PrimProc[76079]: 03.776198 |0|0|0| C 28 CAL0000: 0 0 66807
Dec 16 12:43:03 cstore-pm01 IDBFile[76079]: 03.934215 |0|0|0| D 35 CAL0002: Failed to open file: /000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf, exception: unable to open Unbuffered file
Dec 16 12:43:04 cstore-pm01 IDBFile[76079]: 04.934447 |0|0|0| D 35 CAL0002: Failed to open file: /000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf, exception: unable to open Unbuffered file
Dec 16 12:43:05 cstore-pm01 IDBFile[76079]: 05.934703 |0|0|0| D 35 CAL0002: Failed to open file: /000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf, exception: unable to open Unbuffered file
Dec 16 12:43:06 cstore-pm01 IDBFile[76079]: 06.935802 |0|0|0| D 35 CAL0002: Failed to open file: /000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf, exception: unable to open Unbuffered file
Dec 16 12:43:07 cstore-pm01 IDBFile[76079]: 07.936101 |0|0|0| D 35 CAL0002: Failed to open file: /000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf, exception: unable to open Unbuffered file
Dec 16 12:43:08 cstore-pm01 PrimProc[76079]: 08.938419 |0|0|0| W 28 CAL0000: IDB-2039: Data file does not exist, please contact your system administrator for more information.

But in the table columnstore_file not find this file /000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf.

SELECT * FROM COLUMNSTORE_FILES WHERE filename LIKE '%233.dir/000.dir/FILE000.cdf%'

The path /000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf there isn't in the Columnstore_files, i've found only /usr/local/mariadb/columnstore/data3/000.dir/000.dir/012.dir/233.dir/000.dir/FILE000.cdf.

How does it find the file if it is not mapped in the database?
Thanks,
Regards
Nicola Battista

Generated at Thu Feb 08 02:42:24 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.