Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-31517

Wrong variable name in the configuration leads Galera to think SST/IST failed, at next restart will request a full SST

    XMLWordPrintable

Details

    • Bug
    • Status: In Progress (View Workflow)
    • Critical
    • Resolution: Unresolved
    • 10.6.14
    • 10.6
    • Galera, Galera SST
    • None

    Description

      When there's a wrong variable name MariaDB Server stops and exits.

      There is thou a misleading error message logged by Galera which can lead to think that SST failed because of that (even when no SST actually happened.

       
      2023-06-21 15:04:07 3 [Note] WSREP: Server status change joiner -> initializing
      2023-06-21 15:04:07 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-06-21 15:04:07 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
      2023-06-21 15:04:07 0 [Note] InnoDB: Number of pools: 1
      2023-06-21 15:04:07 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
      2023-06-21 15:04:07 0 [Note] InnoDB: Using Linux native AIO
      2023-06-21 15:04:07 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
      2023-06-21 15:04:07 0 [Note] InnoDB: Completed initialization of buffer pool
      2023-06-21 15:04:07 0 [Note] InnoDB: 128 rollback segments are active.
      2023-06-21 15:04:07 0 [Note] InnoDB: Creating shared tablespace for temporary tables
      2023-06-21 15:04:07 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
      2023-06-21 15:04:07 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
      2023-06-21 15:04:07 0 [Note] InnoDB: 10.6.14 started; log sequence number 3807484; transaction id 185
      2023-06-21 15:04:07 0 [Note] Plugin 'FEEDBACK' is disabled.
      2023-06-21 15:04:07 0 [Note] InnoDB: Loading buffer pool(s) from /run/media/claudio/FedoraData/myharem/instances/mariadb-enterprise-10.6.14-9-centos-9-x86_64.30614/data/ib_buffer_pool
      2023-06-21 15:04:07 0 [ERROR] /run/media/claudio/FedoraData/myharem/instances/mariadb-enterprise-10.6.14-9-centos-9-x86_64.30614/bin/mariadbd: unknown variable 'table_definitio_cache=4K'
      2023-06-21 15:04:07 0 [ERROR] Aborting
      2023-06-21 15:04:07 3 [ERROR] WSREP: sst_received failed: State wait was interrupted
      2023-06-21 15:04:07 2 [ERROR] WSREP: Application received wrong state:
      
      

      And this is without a wrong variable name:

      2023-06-21 15:01:13 3 [Note] WSREP: Server status change joiner -> initializing
      2023-06-21 15:01:13 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-06-21 15:01:13 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
      2023-06-21 15:01:13 0 [Note] InnoDB: Number of pools: 1
      2023-06-21 15:01:13 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
      2023-06-21 15:01:13 0 [Note] InnoDB: Using Linux native AIO
      2023-06-21 15:01:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
      2023-06-21 15:01:13 0 [Note] InnoDB: Completed initialization of buffer pool
      2023-06-21 15:01:13 0 [Note] InnoDB: 128 rollback segments are active.
      2023-06-21 15:01:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables
      2023-06-21 15:01:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
      2023-06-21 15:01:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
      2023-06-21 15:01:13 0 [Note] InnoDB: 10.6.14 started; log sequence number 3803750; transaction id 172
      2023-06-21 15:01:13 0 [Note] Plugin 'FEEDBACK' is disabled.
      2023-06-21 15:01:13 0 [Note] InnoDB: Loading buffer pool(s) from /run/media/claudio/FedoraData/myharem/instances/mariadb-enterprise-10.6.14-9-centos-9-x86_64.30614/data/ib_buffer_pool
      2023-06-21 15:01:13 0 [Note] InnoDB: Buffer pool(s) load completed at 230621 15:01:13
      2023-06-21 15:01:13 0 [Note] Server socket created on IP: '0.0.0.0'.
      2023-06-21 15:01:13 0 [Note] Server socket created on IP: '::'.
      2023-06-21 15:01:13 0 [Warning] WSREP: Ignoring server id for non bootstrap node.
      2023-06-21 15:01:13 0 [Note] WSREP: wsrep_init_schema_and_SR 0x0
      2023-06-21 15:01:13 0 [Note] WSREP: Server initialized
      2023-06-21 15:01:13 0 [Note] WSREP: Server status change initializing -> initialized
      2023-06-21 15:01:13 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-06-21 15:01:13 3 [Note] WSREP: Recovered position from storage: fc65b47a-0ebd-11ee-a691-eb1618faaa0e:72
      2023-06-21 15:01:13 3 [Note] WSREP: Recovered view from SST:
      
      

      You can see the difference after: [Note] InnoDB: Loading buffer pool(s)

      My impression is that Galera inherits somehow an error happening in that phase wrongly attributing it to SST phase which is formally still open.

      The real problem is that at next restart the node will request a full SST even if that's not really required.
      This happens also with no activity whatsovere in the cluster.

      To reproduce:

      • Start Galera cluster
      • Stop a Node
      • Add non_existing_variable=ON in the config
      • Start the node
      • See how it fails and notice the SST failed messages
      • Remove the wrong variable
      • Start the node
      • Observe a full SST happening

      Attachments

        Activity

          People

            sysprg Julius Goryavsky
            claudio.nanni Claudio Nanni
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.