Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13115

Implement SELECT [FOR UPDATE|LOCK IN SHARED MODE] SKIP LOCKED

Details

    Attachments

      Issue Links

        Activity

          olafbuitelaar Olaf Buitelaar added a comment - - edited

          danblackour use case is indeed quite similar as described by terryburton's description. With the exception that we don't directly require an result from the database. Our setup is a follows; We've a set of MariaDB instances where we store the tasks/data in a raw format and simple format. The data is then processed by a set of threads/workers and then inserted into another (master)MariaDB instance using XA transactions. Each thread/worker is general and can process any kind of raw data. The worker process consists of 2 phases, so that locks on the table are minimized and in case the (master)MariaDB isn't available or slow, insertion of new data isn't affected by this.
          In the first phase each worker/thread assigns a id to the raw record, thus essentially assigning it self to do the work in the future. This process is our critical phase, and is protected by

          GET_LOCK('shared-id-between-workers')

          . So that only 1 worker at the time can assign raw data to itself. This is where i would like to use the SKIP LOCKED feature like in an UPDATE. thus instead of:

          SELECT GET_LOCK('shared-id-between-workers');
          UPDATE rawdata SET workerid=1 LIMIT 1000;
          DO RELEASE_LOCK('shared-id-between-workers');
          

          have:

          UPDATE rawdata SET workerid=1 LIMIT 1000 SKIP LOCKED;
          

          The second phase simply does a

          SELECT * FROM rawdata where workerid=1

          , which theoretically just cause a read lock on only it's owning record (sporadically i think i suffer from a GAP lock), and not be in the way of any other workers or the ongoing insertion of new data.

          As requested i've created a new issue for this; MDEV-25338
          I've checked the code files, but i don't think i'll be able to create a PR for this. But if i can contribute in any other way i'm happy todo so.

          olafbuitelaar Olaf Buitelaar added a comment - - edited danblack our use case is indeed quite similar as described by terryburton 's description. With the exception that we don't directly require an result from the database. Our setup is a follows; We've a set of MariaDB instances where we store the tasks/data in a raw format and simple format. The data is then processed by a set of threads/workers and then inserted into another (master)MariaDB instance using XA transactions. Each thread/worker is general and can process any kind of raw data. The worker process consists of 2 phases, so that locks on the table are minimized and in case the (master)MariaDB isn't available or slow, insertion of new data isn't affected by this. In the first phase each worker/thread assigns a id to the raw record, thus essentially assigning it self to do the work in the future. This process is our critical phase, and is protected by GET_LOCK('shared-id-between-workers') . So that only 1 worker at the time can assign raw data to itself. This is where i would like to use the SKIP LOCKED feature like in an UPDATE. thus instead of: SELECT GET_LOCK('shared-id-between-workers'); UPDATE rawdata SET workerid=1 LIMIT 1000; DO RELEASE_LOCK('shared-id-between-workers'); have: UPDATE rawdata SET workerid=1 LIMIT 1000 SKIP LOCKED; The second phase simply does a SELECT * FROM rawdata where workerid=1 , which theoretically just cause a read lock on only it's owning record (sporadically i think i suffer from a GAP lock), and not be in the way of any other workers or the ongoing insertion of new data. As requested i've created a new issue for this; MDEV-25338 I've checked the code files, but i don't think i'll be able to create a PR for this. But if i can contribute in any other way i'm happy todo so.
          danblack Daniel Black added a comment -

          Will be out in 10.6.0. Thanks for your patience all.

          danblack Daniel Black added a comment - Will be out in 10.6.0. Thanks for your patience all.
          danblack Daniel Black added a comment -

          Folks, can you comment on MDEV-25388 if have strong opinions if SKIP LOCKED should be blocked, or not, by table level locks (like 'LOCK TABLES' or a concurrent 'ALTER TABLE'). What should the behavior be?

          danblack Daniel Black added a comment - Folks, can you comment on MDEV-25388 if have strong opinions if SKIP LOCKED should be blocked, or not, by table level locks (like 'LOCK TABLES' or a concurrent 'ALTER TABLE'). What should the behavior be?
          danblack Daniel Black added a comment -

          Also I'm removing the interoperability of WAIT/NOWAIT with SKIP LOCKED reverting to MySQl-8.0 behaviour of an implicit no WAIT. If you have a desire for `WAIT` and `SKIP LOCKED` please so say so on MDEV-25433

          danblack Daniel Black added a comment - Also I'm removing the interoperability of WAIT/NOWAIT with SKIP LOCKED reverting to MySQl-8.0 behaviour of an implicit no WAIT. If you have a desire for `WAIT` and `SKIP LOCKED` please so say so on MDEV-25433
          olafbuitelaar Olaf Buitelaar added a comment - - edited

          Personally i don't have a strong opinion about this. I would say if you choose for SKIP LOCKED, a whole table lock would include this, and thus no records would return. But i can imagine a whole table lock are for special cases like backups or alter's, in which case you do want to wait till it complets. I would leave it to the application to check if a full table lock is present, and decides what to do if these cases are important. But generally i would say, if you opt for lock skipping, that is what you would get, regardless the kind of lock.
          I also see the point of not having the WAIT being interoperable with the SKIP LOCKED, as they are more or less are mutually exclusive.

          olafbuitelaar Olaf Buitelaar added a comment - - edited Personally i don't have a strong opinion about this. I would say if you choose for SKIP LOCKED, a whole table lock would include this, and thus no records would return. But i can imagine a whole table lock are for special cases like backups or alter's, in which case you do want to wait till it complets. I would leave it to the application to check if a full table lock is present, and decides what to do if these cases are important. But generally i would say, if you opt for lock skipping, that is what you would get, regardless the kind of lock. I also see the point of not having the WAIT being interoperable with the SKIP LOCKED, as they are more or less are mutually exclusive.

          People

            danblack Daniel Black
            Nothing4You Richard Schwab
            Votes:
            26 Vote for this issue
            Watchers:
            34 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.