Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-39277

Performance regression in 12.3.0 when using legacy binlog and innodb_flush_log_at_trx_commit=1

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Blocker
    • Resolution: Fixed
    • 13.0, 12.3.1
    • 12.3.2
    • Related to performance
    • Q2/2026 Server Maintenance

    Description

      Using a very limited cache configuration comparing MariaDB version 11.8 to 13 showed regression.

      Configuration used:

      [mysqld]
      performance_schema=OFF
      log_bin=OFF
      sync_binlog=0
      general_log=OFF
      slow_query_log=OFF
      default_storage_engine=InnoDB
      default_authentication_plugin=mysql_native_password
      tls_version = TLSv1.2,TLSv1.3
      character_set_server=utf8mb4
      collation_server=utf8mb4_general_ci
      max_connections=200
      open_files_limit=65535
      table_open_cache=4000
      thread_cache_size=128
       
      # Innodb
      innodb_buffer_pool_size=30G
      innodb_flush_log_at_trx_commit=1
      innodb_doublewrite=1
      innodb_flush_method=O_DIRECT
      innodb_flush_neighbors=0
      innodb_log_file_size=8G
      innodb_undo_tablespaces=2
      innodb_undo_log_truncate=ON
      innodb_io_capacity=2000
      innodb_io_capacity_max=4000
      innodb_read_io_threads=8
      innodb_write_io_threads=8
      innodb_file_per_table=1
      innodb_default_row_format=dynamic
      innodb_compression_algorithm=none
      innodb_compression_level=0
       
      # sync
      sync_master_info=0
      sync_relay_log=0
      sync_relay_log_info=0
       
      aria_pagecache_buffer_size = 128M
      aria_sort_buffer_size = 128M
      

      Note: publicly downloaded versions test which show 12.3.0 is the start.

      Not being expert of MariaDB git usage, I created a list I thought was correct, produced 339 commits, and 22 builds later I end up as test case being regression, was able to find small 12.3 tag start and regenrate list of 124 commits, and with in 9 build had issolated to 7081f2a58ec459c

      Note: Each commit was built by a script I created which I will attach. The CMAKE used is as follows

      cmake "$SRC_DIR" \
        -DCMAKE_BUILD_TYPE=RelWithDebInfo \
        -DCMAKE_INSTALL_PREFIX="$INSTALL_DIR" \
        -DWITH_SSL=system \
        -DWITH_ZLIB=system \
        -DWITH_PCRE=system \
        -DWITH_UNIT_TESTS=OFF
      

      [server]$ git checkout  7081f2a58e
      [server]$ git log -2
      commit 7081f2a58ec459c55296dcf700e2f4cc1c85b4e7 (HEAD)
      Author: Kristian Nielsen <knielsen@knielsen-hq.org>
      Date:   Sun Feb 25 17:41:50 2024 +0100
       
          Binlog-in-engine: New binlog implementation integrated in InnoDB
       
          Implement an improved binlog implementation that is integrated into
          the storage engine. The new implementation is enabled with the
          --binlog-storage-engine option. Initially the InnoDB storage engine
      ...
      ..
      commit d228f237f276f1dc7583e918e3039e6f74ceb2db
      Author: Kristian Nielsen <knielsen@knielsen-hq.org>
      Date:   Sun Nov 9 12:22:31 2025 +0100
       
          Fix that InnoDB rolls back entire trx in failed CREATE TEMPORARY
       
          For CREATE TEMPORARY TABLE ... SELECT, InnoDB had code to not start a new
          transaction for the CREATE TEMPORARY (correct). But the code that handled
       
      d228f237f276f1dc758  vs  7081f2a58ec459c (Matching original regression found)
       !image-2026-04-07-09-46-20-359.png|thumbnail! 
      

      Bin Log On: Classic

      Even bin log on in classic shows a mild regression.

      eg. Only config diff "log_bin"

      HammerDB complaining when bin log on, innodb, etc....

      HammerDB logs are showing:

      Vuser 29:mariaexec/db server: Record has changed since last read in table 'district'
      Vuser 21:mariaexec/db server: Record has changed since last read in table 'district'
      Vuser 24:mariaexec/db server: Record has changed since last read in table 'new_order'
      Vuser 44:mariaexec/db server: Record has changed since last read in table 'district'
      Vuser 12:mariaexec/db server: Record has changed since last read in table 'new_order'
      Vuser 4:mariaexec/db server: Record has changed since last read in table 'district'
      Vuser 35:mariaexec/db server: Record has changed since last read in table 'district'
      Vuser 22:mariaexec/db server: Record has changed since last read in table 'district'
      Vuser 21:mariaexec/db server: Record has changed since last read in table 'district'
      Vuser 24:mariaexec/db server: Record has changed since last read in table 'new_order'
      Vuser 4:mariaexec/db server: Record has changed since last read in table 'new_order'
      ...
      

      Have not yet verified exactly when these started.

      The CPU usage:

      Watching MariaDBD during non regressed, and regressed showed that the regressed actually uses less CPU, not more...

      Non regressed:

      Regressed:

      Being that these are with debug info, I profiled both commits.

      Commit before:

      Target:

      Made a diff of them:

      Profiling data uploaded.

      Please let me know if additional testing is needed, or if patch testing is needed.

      Please let me know if you have questions, concerns, or if something is not clear.

      PS: Could not include 12.3.0

      Attachments

        1. 11.8.6_cpu_56vu_latency.png
          11.8.6_cpu_56vu_latency.png
          24 kB
        2. 11.8.6_cpu_56vu_met.png
          11.8.6_cpu_56vu_met.png
          39 kB
        3. 11.8.6_cpu_56vu_result.png
          11.8.6_cpu_56vu_result.png
          53 kB
        4. 11.8.6_cpu_56vu_tpm.png
          11.8.6_cpu_56vu_tpm.png
          33 kB
        5. 12.3.0_commits_binary_search_list.txt
          9 kB
        6. 12.3.1_cpu_56vu_latency.png
          12.3.1_cpu_56vu_latency.png
          24 kB
        7. 12.3.1_cpu_56vu_met.png
          12.3.1_cpu_56vu_met.png
          40 kB
        8. 12.3.1_cpu_56vu_result.png
          12.3.1_cpu_56vu_result.png
          52 kB
        9. 12.3.1_cpu_56vu_tpm.png
          12.3.1_cpu_56vu_tpm.png
          35 kB
        10. 12.3.1vs11.8.6_prof_cpu.png
          12.3.1vs11.8.6_prof_cpu.png
          51 kB
        11. 12.3.1vs11.8.6_prof_io.png
          12.3.1vs11.8.6_prof_io.png
          53 kB
        12. 7081f2a58ec.svg
          1.22 MB
        13. 7081f2a58ec459c-perf-data.tar.gz
          6.13 MB
        14. CommitBuilder.sh
          0.9 kB
        15. d228f237f27.svg
          897 kB
        16. d228f237f276f1dc7583-perf-data.tar.gz
          7.97 MB
        17. diff.svg
          1.28 MB
        18. image-2026-04-07-09-32-17-910.png
          image-2026-04-07-09-32-17-910.png
          63 kB
        19. image-2026-04-07-09-38-17-523.png
          image-2026-04-07-09-38-17-523.png
          52 kB
        20. image-2026-04-07-09-39-48-655.png
          image-2026-04-07-09-39-48-655.png
          36 kB
        21. image-2026-04-07-09-40-07-551.png
          image-2026-04-07-09-40-07-551.png
          15 kB
        22. image-2026-04-07-09-46-20-359.png
          image-2026-04-07-09-46-20-359.png
          39 kB
        23. image-2026-04-07-09-49-08-472.png
          image-2026-04-07-09-49-08-472.png
          38 kB
        24. image-2026-04-07-09-54-57-248.png
          image-2026-04-07-09-54-57-248.png
          13 kB
        25. image-2026-04-07-09-55-17-840.png
          image-2026-04-07-09-55-17-840.png
          15 kB
        26. image-2026-04-07-09-56-20-157.png
          image-2026-04-07-09-56-20-157.png
          216 kB
        27. image-2026-04-07-09-57-01-698.png
          image-2026-04-07-09-57-01-698.png
          215 kB
        28. image-2026-04-07-09-57-34-008.png
          image-2026-04-07-09-57-34-008.png
          188 kB
        29. image-2026-04-07-10-08-48-321.png
          image-2026-04-07-10-08-48-321.png
          8 kB
        30. mariadb_cache_bin_log.cnf
          0.9 kB
        31. mariadb_cache.cnf
          0.9 kB
        32. mariadb_tprocc.properties
          5 kB
        33. mariadb_tprocc.sh
          2 kB
        34. screenshot-1.png
          screenshot-1.png
          48 kB
        35. TPROCC_11.8_vs_13.html
          13 kB
        36. TPROCC_20260405_225240.chartplus_testinfo.html
          19 kB
        37. TPROCC_d228f237f27_vs_7081f2a58ec_binlog_on.html
          8 kB
        38. TPROCC_TPROCC_d228f237f27_vs_7081f2a58ec.html
          8 kB
        39. TPROCC_versions_compared.html
          15 kB
        40. WALsAtWork.png
          WALsAtWork.png
          67 kB

        Activity

          People

            knielsen Kristian Nielsen
            jeb Jonathan Jeb Miller
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.