Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-41385

wsrep_sync_wait silently ignored after COM_CHANGE_USER (causal reads skipped, stale reads)

    XMLWordPrintable

Details

    • Can result in unexpected behaviour
    • The client connection's execution state in cluster was lost after the session has called mysql_change_user command. This has been fixed in current version.
    • Q4/2026 Replic. Maintenance

    Description

      Upstream references:

      Summary

      After a client connection executes COM_CHANGE_USER, wsrep_sync_wait no longer has any effect on that connection.
      The session still reports @@session.wsrep_sync_wait=1, but reads do not perform a causal wait: wsrep_causal_reads does not increase, and reads can return data older than writes already committed on another node.
      Running SET SESSION wsrep_sync_wait=1 again after COM_CHANGE_USER does not restore the behaviour.
      A new connection works correctly.
      Reproduced on 10.6.17-13 and on 10.6.28-24, the latest 10.6 Enterprise release, so it is not fixed in current 10.6.

      Impact

      Connection poolers that reset backend connections with COM_CHANGE_USER hit this on almost every request.
      ProxySQL does this by default (mysql-reset_connection_algorithm=2, mysql-connpoll_reset_queue_length=50).
      The same applies to other client libraries or pools that use change_user for resets.
      Any application that relies on wsrep_sync_wait for read-after-write consistency across nodes silently gets stale reads.
      In a real-world setup, this shows up as intermittent application errors: a row updated on one node is not found when it is read on another node through ProxySQL.
      Direct connections to MariaDB, which never issue COM_CHANGE_USER, do not reproduce the problem.

      How to repeat

      Live 2-node cluster (attached: galera613_test.py)

      Setup: 2 nodes, run on 10.6.17-13 (Galera 26.4.17) and on 10.6.28-24 (Galera 26.4.27), SET GLOBAL wsrep_sync_wait=1 on both nodes, no other traffic, no ProxySQL.

      For each scenario, a writer connection on node 2 inserts a row with autocommit, and a reader connection on node 1, opened fresh for each scenario, immediately runs SELECT COUNT for that id.
      This is repeated 200 times.
      The script records the increase of wsrep_causal_reads on node 1, the number of reads that did not find the row (stale), and the average read latency.

      python3 galera613_test.py --node1 <node1> --node2 <node2> --user <user> [--password <pw>]
      

      Requirements: pip install mysql-connector-python (or mariadb). The user needs CREATE, INSERT, SELECT and DROP on the test database (default: test) and must be able to connect over TCP to both nodes.

      Results on 10.6.17-13-MariaDB-enterprise, Galera 26.4.17:

      Case Scenario session sync_wait causal reads stale avg read ms
      0 fresh connection, SET SESSION wsrep_sync_wait=1 1 200/200 0 0.888
      1 fresh connection, global value 1 200/200 0 0.832
      2 fresh connection, SET SESSION wsrep_sync_wait=0 (control) 0 0/200 38 0.405
      3 after one COM_CHANGE_USER 1 0/200 48 0.409
      4 COM_CHANGE_USER + SET NAMES (ProxySQL handout) 1 0/200 31 0.401
      5 COM_CHANGE_USER before every read (ProxySQL per request) 1 0/200 27 0.429
      6 COM_CHANGE_USER + SET SESSION wsrep_sync_wait=1 1 0/200 36 0.402

      Results on 10.6.28-24-MariaDB-enterprise, Galera 26.4.27:

      Case Scenario session sync_wait causal reads stale avg read ms
      0 fresh connection, SET SESSION wsrep_sync_wait=1 1 200/200 0 0.984
      1 fresh connection, global value 1 200/200 0 1.030
      2 fresh connection, SET SESSION wsrep_sync_wait=0 (control) 0 0/200 50 0.486
      3 after one COM_CHANGE_USER 1 0/200 46 0.521
      4 COM_CHANGE_USER + SET NAMES (ProxySQL handout) 1 0/200 39 0.487
      5 COM_CHANGE_USER before every read (ProxySQL per request) 1 0/200 29 0.505
      6 COM_CHANGE_USER + SET SESSION wsrep_sync_wait=1 1 0/200 41 0.534

      On both versions, in every scenario after COM_CHANGE_USER, the connection behaves exactly like wsrep_sync_wait=0, in causal read count, stale reads and latency, although the variable reads 1.
      Scenario 6 shows that re-setting the variable explicitly does not help.
      This suggests the causal-wait path depends on some per-connection wsrep state that is not re-initialised by COM_CHANGE_USER, rather than on the variable value itself (not analysed in code).

      Workarounds

      • ProxySQL: mysql-reset_connection_algorithm=1 and mysql-connpoll_reset_queue_length=0, so connections that would need COM_CHANGE_USER are not reused (PR #3929).
      • Session affinity at the application layer, so that reads go to the node that performed the write.
      • Re-setting wsrep_sync_wait after COM_CHANGE_USER does not work (scenario 6).
      • A draft for the galera suite is attached (galera_sync_wait_change_user.test). It has not been run.

      Attachments

        1. galera_sync_wait_change_user.test
          3 kB
          Claudio Nanni
        2. galera613_test.py
          6 kB
          Claudio Nanni

        Activity

          People

            seppo Seppo Jaakola
            claudio.nanni Claudio Nanni
            Seppo Jaakola Seppo Jaakola
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 2d
                2d
                Remaining:
                Remaining Estimate - 0d
                0d
                Logged:
                Time Spent - 5h Time Not Required
                5h

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.