Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Won't Do
-
1.2.4
-
None
-
3PM and 2UM with glusterfs
Description
Hi,
I make the simple query
select * from audit;
after about 1,3M of the records processed i received this error :
Jun 24 12:01:08 cstore-pm01 PrimProc[2394]: 08.237188 |0|0|0| C 28 CAL0053: PrimProc could not open file for OID 3904; /000.dir/000.dir/015.dir/064.dir/008.dir/FILE002.cdf:No such file or directory
Jun 24 12:03:40 cstore-pm01 PrimProc[2394]: 40.766674 |0|0|0| C 28 CAL0053: PrimProc could not open file for OID 3904; /000.dir/000.dir/015.dir/064.dir/008.dir/FILE002.cdf:No such file or directory
Jun 24 12:05:15 cstore-pm01 PrimProc[2394]: 15.721891 |0|0|0| C 28 CAL0053: PrimProc could not open file for OID 3904; /000.dir/000.dir/015.dir/064.dir/008.dir/FILE002.cdf:No such file or directory
But this file exist :
/usr/local/mariadb/columnstore/gluster/brick2/000.dir/000.dir/015.dir/064.dir/008.dir/FILE002.cdf
Why not find this file?
Thanks,
Regards.
Nicola Battista
Attachments
Issue Links
- includes
-
MCOL-1853 ColumnStore breaks on S3 NFS mounts
- Closed
Activity
MariaDB [(none)]> select * from information_schema.columnstore_files where object_id=3904
-> ;
-------------------------------------------------------------------------------------------------------------------------------------------------------+
OBJECT_ID | SEGMENT_ID | PARTITION_ID | FILENAME | FILE_SIZE | COMPRESSED_DATA_SIZE |
-------------------------------------------------------------------------------------------------------------------------------------------------------+
3904 | 2 | 0 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/000.dir/FILE002.cdf | 1946427392 | 1479442432 |
3904 | 0 | 1 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/001.dir/FILE000.cdf | 1677991936 | 1227800576 |
3904 | 0 | 2 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/002.dir/FILE000.cdf | 537141248 | 405602304 |
3904 | 0 | 0 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/000.dir/FILE000.cdf | 1745100800 | 1314177024 |
3904 | 1 | 0 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/000.dir/FILE001.cdf | 1342447616 | 923369472 |
3904 | 0 | 3 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/003.dir/FILE000.cdf | 805576704 | 606117888 |
3904 | 0 | 4 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/004.dir/FILE000.cdf | 1946427392 | 1488052224 |
3904 | 1 | 1 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/001.dir/FILE001.cdf | 2013536256 | 1519714304 |
3904 | 0 | 5 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/005.dir/FILE000.cdf | 2080645120 | 1555505152 |
3904 | 2 | 1 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/001.dir/FILE002.cdf | 2080645120 | 1544241152 |
3904 | 1 | 2 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/002.dir/FILE001.cdf | 2080645120 | 1568161792 |
3904 | 0 | 6 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/006.dir/FILE000.cdf | 2080645120 | 1555390464 |
3904 | 1 | 3 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/003.dir/FILE001.cdf | 2080645120 | 1542496256 |
3904 | 2 | 2 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/002.dir/FILE002.cdf | 2080645120 | 1565884416 |
3904 | 0 | 7 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/007.dir/FILE000.cdf | 2080645120 | 1548328960 |
3904 | 1 | 4 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/004.dir/FILE001.cdf | 2147753984 | 1651367936 |
3904 | 2 | 3 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/003.dir/FILE002.cdf | 2147753984 | 1641857024 |
3904 | 0 | 8 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/008.dir/FILE000.cdf | 2214862848 | 1671102464 |
3904 | 1 | 5 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/005.dir/FILE001.cdf | 2080645120 | 1560608768 |
3904 | 0 | 9 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/009.dir/FILE000.cdf | 2080645120 | 1578786816 |
3904 | 2 | 4 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/004.dir/FILE002.cdf | 1141121024 | 861323264 |
3904 | 1 | 7 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/007.dir/FILE001.cdf | 2147753984 | 1607196672 |
3904 | 1 | 8 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/008.dir/FILE001.cdf | 2147753984 | 1605926912 |
3904 | 0 | 10 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/010.dir/FILE000.cdf | 2147753984 | 1623785472 |
3904 | 0 | 11 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/011.dir/FILE000.cdf | 2147753984 | 1653661696 |
| 3904 | 2 | 8 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/008.dir/FILE002.cdf | 2147753984 | 1646280704 |
3904 | 1 | 9 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/009.dir/FILE001.cdf | 2013536256 | 1522106368 |
3904 | 1 | 10 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/010.dir/FILE001.cdf | 2013536256 | 1505804288 |
3904 | 0 | 13 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/013.dir/FILE000.cdf | 2080645120 | 1565417472 |
3904 | 2 | 10 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/010.dir/FILE002.cdf | 2080645120 | 1576329216 |
3904 | 1 | 11 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/011.dir/FILE001.cdf | 2013536256 | 1495556096 |
3904 | 0 | 14 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/014.dir/FILE000.cdf | 2013536256 | 1504559104 |
3904 | 2 | 11 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/011.dir/FILE002.cdf | 1946427392 | 1488633856 |
3904 | 0 | 15 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/015.dir/FILE000.cdf | 1812209664 | 1369063424 |
3904 | 1 | 12 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/012.dir/FILE001.cdf | 1946427392 | 1468047360 |
3904 | 0 | 12 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/012.dir/FILE000.cdf | 1946427392 | 1483374592 |
3904 | 2 | 9 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/009.dir/FILE002.cdf | 2013536256 | 1511571456 |
3904 | 2 | 12 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/012.dir/FILE002.cdf | 2013536256 | 1541857280 |
3904 | 0 | 16 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/016.dir/FILE000.cdf | 2013536256 | 1527414784 |
3904 | 1 | 13 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/013.dir/FILE001.cdf | 1946427392 | 1470980096 |
3904 | 2 | 13 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/013.dir/FILE002.cdf | 2080645120 | 1571250176 |
3904 | 0 | 17 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/017.dir/FILE000.cdf | 2080645120 | 1595531264 |
3904 | 1 | 14 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/014.dir/FILE001.cdf | 1879318528 | 1389658112 |
3904 | 2 | 14 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/014.dir/FILE002.cdf | 2013536256 | 1539735552 |
3904 | 0 | 18 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/018.dir/FILE000.cdf | 2013536256 | 1533771776 |
3904 | 1 | 16 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/016.dir/FILE001.cdf | 2013536256 | 1527177216 |
3904 | 1 | 15 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/015.dir/FILE001.cdf | 2013536256 | 1514790912 |
3904 | 0 | 19 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/019.dir/FILE000.cdf | 2080645120 | 1602781184 |
3904 | 0 | 20 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/020.dir/FILE000.cdf | 2080645120 | 1571987456 |
3904 | 2 | 16 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/016.dir/FILE002.cdf | 1946427392 | 1481867264 |
3904 | 1 | 17 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/017.dir/FILE001.cdf | 2013536256 | 1518739456 |
3904 | 0 | 21 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/021.dir/FILE000.cdf | 2080645120 | 1559232512 |
3904 | 2 | 17 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/017.dir/FILE002.cdf | 2013536256 | 1507934208 |
3904 | 1 | 18 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/018.dir/FILE001.cdf | 1946427392 | 1465122816 |
3904 | 0 | 22 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/022.dir/FILE000.cdf | 2013536256 | 1526349824 |
3904 | 2 | 18 | /usr/local/mariadb/columnstore/data2/000.dir/000.dir/015.dir/064.dir/018.dir/FILE002.cdf | 1141121024 | 866205696 |
3904 | 1 | 19 | /usr/local/mariadb/columnstore/data3/000.dir/000.dir/015.dir/064.dir/019.dir/FILE001.cdf | 738467840 | 527532032 |
3904 | 0 | 23 | /usr/local/mariadb/columnstore/data1/000.dir/000.dir/015.dir/064.dir/023.dir/FILE000.cdf | 537141248 | 385040384 |
-------------------------------------------------------------------------------------------------------------------------------------------------------+
58 rows in set (0.983 sec)
Hi all,
together Mariadb ColumnStore development team, i'have opened the ticket at glusterfs, because we thought it was a glusterfs problem.
They examined all the trace logs and came to the conclusion that this is not a glusterfs problem.
This is the link of the ticket :
https://bugzilla.redhat.com/show_bug.cgi?id=1744883
Finally they asked me:
Is there a chance of races in the application where a thread tries to access the file before a creat() from another thread?
Can you help me?
Thanks
Regards
Nicola Battista
Hi all,
we have the same problem with another table :
Log PM columnstore :
Dec 16 12:43:03 cstore-pm01 PrimProc[76079]: 03.775764 |0|0|0| C 28 CAL0000: 0 0 66805
Dec 16 12:43:03 cstore-pm01 PrimProc[76079]: 03.775973 |0|0|0| C 28 CAL0000: 0 0 66806
Dec 16 12:43:03 cstore-pm01 PrimProc[76079]: 03.776198 |0|0|0| C 28 CAL0000: 0 0 66807
Dec 16 12:43:03 cstore-pm01 IDBFile[76079]: 03.934215 |0|0|0| D 35 CAL0002: Failed to open file: /000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf, exception: unable to open Unbuffered file
Dec 16 12:43:04 cstore-pm01 IDBFile[76079]: 04.934447 |0|0|0| D 35 CAL0002: Failed to open file: /000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf, exception: unable to open Unbuffered file
Dec 16 12:43:05 cstore-pm01 IDBFile[76079]: 05.934703 |0|0|0| D 35 CAL0002: Failed to open file: /000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf, exception: unable to open Unbuffered file
Dec 16 12:43:06 cstore-pm01 IDBFile[76079]: 06.935802 |0|0|0| D 35 CAL0002: Failed to open file: /000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf, exception: unable to open Unbuffered file
Dec 16 12:43:07 cstore-pm01 IDBFile[76079]: 07.936101 |0|0|0| D 35 CAL0002: Failed to open file: /000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf, exception: unable to open Unbuffered file
Dec 16 12:43:08 cstore-pm01 PrimProc[76079]: 08.938419 |0|0|0| W 28 CAL0000: IDB-2039: Data file does not exist, please contact your system administrator for more information.
But in the table columnstore_file not find this file /000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf.
SELECT * FROM COLUMNSTORE_FILES WHERE filename LIKE '%233.dir/000.dir/FILE000.cdf%'
The path /000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf there isn't in the Columnstore_files, i've found only /usr/local/mariadb/columnstore/data3/000.dir/000.dir/012.dir/233.dir/000.dir/FILE000.cdf.
How does it find the file if it is not mapped in the database?
Thanks,
Regards
Nicola Battista
Hi,
Any news?
Thanks,
Regards
Nicola Battista.