[MCOL-1173] insert failed Created: 2018-01-20  Updated: 2021-01-15

Status: Closed
Project: MariaDB ColumnStore
Component/s: None
Affects Version/s: 1.0.11
Fix Version/s: None

Type: Bug Priority: Major
Reporter: JuneHu Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

CentOS release 6.9 x86_64


Attachments: Text File pm1_configReport.txt     Text File pm1_dbmsReport.txt     Text File pm1_hardwareReport.txt     Text File pm1_logReport.txt     Text File pm1_resourceReport.txt     Text File pm1_softwareReport.txt     PNG File upload-jira-error.png    

 Description   

General error: 1815 Internal error: CAL0001: Insert Failed: IDB-2009: Unable to perform the batch insert operation because DMLProc batchinsert with PID 1333049 is currently holding the table lock for session 84816.



 Comments   
Comment by Andrew Hutchings (Inactive) [ 2018-01-20 ]

Hi,

Your DMLProc currently has three locks on tables at the moment. There is not enough information here to explain why but DMLProc believes they are all mid-transaction, we would need to see a full support report output for that. But it looks like your mysqld may have been restarted at some point and this could have caused these transactions to hang.

You should be able to roll them back with:

/usr/local/mariadb/columnstore/bin/cleartablelock <lockID>

The lockIDs you want to clear are 719605, 720407 and 721181.

You can also view them using:

/usr/local/mariadb/columnstore/bin/viewtablelock

Comment by JuneHu [ 2018-01-22 ]

First thank you for your reply.

I export a log file with the command:

/data/mariadb/columnstore/bin/columnstoreSupport -a

But it's too big, so I upload some little file.

When I try to stop the MCS system, I see the process id which lock the table, so I kill the process with the process id, and then it works.

Here is stop command's output:

[root@bak ~]# mcsadmin stopSystem
stopsystem   Sat Jan 20 17:58:49 2018
 
This command stops the processing of applications on all Modules within the MariaDB ColumnStore System
 
   Checking for active transactions
The following tables are locked:
LockID Name                         Process             PID     Session CreationTime           State    DBRoots
719605 vc.vstat_code          DMLProc batchinsert 1333049 84816   2018-01-20 03:00:04 PM LOADING  1
720407 vc.vstat_servers       DMLProc batchinsert 1333049 721     2018-01-20 03:47:09 PM LOADING  1
721181 vc.channel_data DMLProc batchinsert 1333049 1568    2018-01-20 04:40:28 PM LOADING  1
 
There are active transactions being processed

Comment by Andrew Hutchings (Inactive) [ 2018-01-22 ]

Hi,

Large files can be uploaded to ftp://ftp.mariadb.com/uploads if needed.

Killing DMLProc (the process ID) would also work because ColumnStore's internal monitoring system would restart DMLProc and during the start of the processes it will rollback previous transactions.

Comment by JuneHu [ 2018-01-24 ]

Hi,

I upload the file columnstoreSupportReport.columnstore-1.tar.gz to ftp://ftp.mariadb.com/uploads. But when it finished, the file does not exist under the directory /uploads.

Here is the screenshot:

When I wget it, it failed with the error message:

550 Failed to open file.
 
No such file ‘columnstoreSupportReport.columnstore-1.tar.gz’.

So, I upload the gz file to a site, here is url: columnstoreSupportReport.columnstore-1.tar.gz

Comment by Andrew Hutchings (Inactive) [ 2018-01-27 ]

Hi,

The uploads directory is write-only for security reasons. Many thanks for the upload. We will look into it ASAP.

Comment by JuneHu [ 2018-03-17 ]

Hi, how's the progress coming along?

Comment by Andrew Hutchings (Inactive) [ 2018-03-19 ]

So, the logs appear to show WriteEngineServer restarting during DML queries causing the locks to be held open as the DML never finishes. This is likely to be due to a crash. There are two options to help us find out why:

1. Upgrade to 1.0.13 or 1.1.3 which has a crash handler built in that will log crashes as can be seen here and send us the crash traces:

https://mariadb.com/kb/en/library/system-troubleshooting-mariadb-columnstore/#crash-trace-files

2. Enable core files and send us one of these when it happens:

https://mariadb.com/kb/en/library/system-troubleshooting-mariadb-columnstore/#enabledisable-core-files

Generated at Thu Feb 08 02:26:45 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.