danblackour use case is indeed quite similar as described by terryburton's description. With the exception that we don't directly require an result from the database. Our setup is a follows; We've a set of MariaDB instances where we store the tasks/data in a raw format and simple format. The data is then processed by a set of threads/workers and then inserted into another (master)MariaDB instance using XA transactions. Each thread/worker is general and can process any kind of raw data. The worker process consists of 2 phases, so that locks on the table are minimized and in case the (master)MariaDB isn't available or slow, insertion of new data isn't affected by this.
In the first phase each worker/thread assigns a id to the raw record, thus essentially assigning it self to do the work in the future. This process is our critical phase, and is protected by
GET_LOCK('shared-id-between-workers')
|
. So that only 1 worker at the time can assign raw data to itself. This is where i would like to use the SKIP LOCKED feature like in an UPDATE. thus instead of:
SELECT GET_LOCK('shared-id-between-workers');
|
UPDATE rawdata SET workerid=1 LIMIT 1000;
|
DO RELEASE_LOCK('shared-id-between-workers');
|
have:
UPDATE rawdata SET workerid=1 LIMIT 1000 SKIP LOCKED;
|
The second phase simply does a
SELECT * FROM rawdata where workerid=1
|
, which theoretically just cause a read lock on only it's owning record (sporadically i think i suffer from a GAP lock), and not be in the way of any other workers or the ongoing insertion of new data.
As requested i've created a new issue for this; MDEV-25338
I've checked the code files, but i don't think i'll be able to create a PR for this. But if i can contribute in any other way i'm happy todo so.
danblackour use case is indeed quite similar as described by terryburton's description. With the exception that we don't directly require an result from the database. Our setup is a follows; We've a set of MariaDB instances where we store the tasks/data in a raw format and simple format. The data is then processed by a set of threads/workers and then inserted into another (master)MariaDB instance using XA transactions. Each thread/worker is general and can process any kind of raw data. The worker process consists of 2 phases, so that locks on the table are minimized and in case the (master)MariaDB isn't available or slow, insertion of new data isn't affected by this.
In the first phase each worker/thread assigns a id to the raw record, thus essentially assigning it self to do the work in the future. This process is our critical phase, and is protected by
GET_LOCK('shared-id-between-workers')
. So that only 1 worker at the time can assign raw data to itself. This is where i would like to use the SKIP LOCKED feature like in an UPDATE. thus instead of:
SELECT GET_LOCK('shared-id-between-workers');
UPDATE rawdata SET workerid=1 LIMIT 1000;
DO RELEASE_LOCK('shared-id-between-workers');
have:
UPDATE rawdata SET workerid=1 LIMIT 1000 SKIP LOCKED;
The second phase simply does a
SELECT * FROM rawdata where workerid=1
, which theoretically just cause a read lock on only it's owning record (sporadically i think i suffer from a GAP lock), and not be in the way of any other workers or the ongoing insertion of new data.
As requested i've created a new issue for this; MDEV-25338
I've checked the code files, but i don't think i'll be able to create a PR for this. But if i can contribute in any other way i'm happy todo so.