Details
-
Bug
-
Status: Closed (View Workflow)
-
Blocker
-
Resolution: Fixed
-
10.6
Description
Multi-master load hangs in seconds possible causes:
Attachments
Issue Links
- is blocked by
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
-
- Closed
-
Activity
| Field | Original Value | New Value |
|---|---|---|
| Link |
This issue relates to |
| Link |
This issue relates to |
| Priority | Major [ 3 ] | Blocker [ 1 ] |
| Fix Version/s | 10.6.0 [ 24431 ] | |
| Fix Version/s | 10.6 [ 24028 ] |
| Labels | regression |
| Labels | regression | hang regression |
Update: using innodb_lock_wait_timeout = 1 tps: 0.0 disappear but number are still low
- build1
jan@galapq:~$ sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/10.6-bug/mysql-test/var/tmp/mysqld.2.sock --table-size=1000000 --mysql-user=root runsysbench 1.0.18 (using system LuaJIT 2.1.0-beta3)Running the test with following options:Number of threads: 16Report intermediate results every 10 second(s)Initializing random number generator from current timeInitializing worker threads...Threads started![ 10s ] thds: 16 tps: 75.98 qps: 1553.72 (r/w/o: 1089.33/168.16/296.23) lat (ms,95%): 383.33 err/s: 0.30 reconn/s: 0.00[ 20s ] thds: 16 tps: 154.91 qps: 3124.85 (r/w/o: 2187.90/383.52/553.43) lat (ms,95%): 153.02 err/s: 1.30 reconn/s: 0.00[ 30s ] thds: 16 tps: 124.97 qps: 2509.21 (r/w/o: 1757.19/341.32/410.70) lat (ms,95%): 158.63 err/s: 0.60 reconn/s: 0.00[ 40s ] thds: 16 tps: 153.14 qps: 3090.62 (r/w/o: 2164.30/448.00/478.31) lat (ms,95%): 158.63 err/s: 1.40 reconn/s: 0.00[ 50s ] thds: 16 tps: 110.20 qps: 2224.93 (r/w/o: 1558.22/341.90/324.80) lat (ms,95%): 158.63 err/s: 1.10 reconn/s: 0.00[ 60s ] thds: 16 tps: 154.40 qps: 3113.26 (r/w/o: 2180.67/492.29/440.29) lat (ms,95%): 150.29 err/s: 1.40 reconn/s: 0.00[ 70s ] thds: 16 tps: 155.30 qps: 3141.60 (r/w/o: 2200.20/511.10/430.30) lat (ms,95%): 150.29 err/s: 2.00 reconn/s: 0.00[ 80s ] thds: 16 tps: 151.90 qps: 3063.00 (r/w/o: 2145.20/508.50/409.30) lat (ms,95%): 155.80 err/s: 1.20 reconn/s: 0.00[ 90s ] thds: 16 tps: 153.50 qps: 3096.00 (r/w/o: 2167.90/521.40/406.70) lat (ms,95%): 153.02 err/s: 1.30 reconn/s: 0.00[ 100s ] thds: 16 tps: 129.10 qps: 2601.09 (r/w/o: 1821.40/445.00/334.70) lat (ms,95%): 167.44 err/s: 1.00 reconn/s: 0.00[ 110s ] thds: 16 tps: 70.60 qps: 1418.60 (r/w/o: 993.90/241.10/183.60) lat (ms,95%): 1069.86 err/s: 0.40 reconn/s: 0.00[ 120s ] thds: 16 tps: 32.30 qps: 652.80 (r/w/o: 456.50/112.90/83.40) lat (ms,95%): 1479.41 err/s: 0.30 reconn/s: 0.00[ 130s ] thds: 16 tps: 3.40 qps: 65.00 (r/w/o: 45.40/11.50/8.10) lat (ms,95%): 7215.39 err/s: 0.00 reconn/s: 0.00[ 140s ] thds: 16 tps: 40.70 qps: 826.59 (r/w/o: 579.40/144.10/103.10) lat (ms,95%): 2045.74 err/s: 0.60 reconn/s: 0.00[ 150s ] thds: 16 tps: 51.50 qps: 1028.40 (r/w/o: 720.30/176.80/131.30) lat (ms,95%): 419.45 err/s: 0.60 reconn/s: 0.00[ 160s ] thds: 16 tps: 122.30 qps: 2486.39 (r/w/o: 1742.99/431.70/311.70) lat (ms,95%): 186.54 err/s: 1.50 reconn/s: 0.00[ 170s ] thds: 16 tps: 127.60 qps: 2573.20 (r/w/o: 1800.70/449.20/323.30) lat (ms,95%): 183.21 err/s: 1.10 reconn/s: 0.00[ 180s ] thds: 16 tps: 122.80 qps: 2464.19 (r/w/o: 1725.69/427.80/310.70) lat (ms,95%): 196.89 err/s: 0.50 reconn/s: 0.00[ 190s ] thds: 16 tps: 124.10 qps: 2515.06 (r/w/o: 1761.67/441.39/311.99) lat (ms,95%): 196.89 err/s: 1.60 reconn/s: 0.00[ 200s ] thds: 16 tps: 127.10 qps: 2561.15 (r/w/o: 1793.34/447.71/320.11) lat (ms,95%): 189.93 err/s: 1.20 reconn/s: 0.00[ 210s ] thds: 16 tps: 133.80 qps: 2706.80 (r/w/o: 1896.10/475.90/334.80) lat (ms,95%): 173.58 err/s: 1.60 reconn/s: 0.00[ 220s ] thds: 16 tps: 133.30 qps: 2688.52 (r/w/o: 1882.72/476.70/329.10) lat (ms,95%): 173.58 err/s: 1.10 reconn/s: 0.00[ 230s ] thds: 16 tps: 135.10 qps: 2718.89 (r/w/o: 1904.19/478.10/336.60) lat (ms,95%): 170.48 err/s: 1.10 reconn/s: 0.00[ 240s ] thds: 16 tps: 137.20 qps: 2760.87 (r/w/o: 1933.68/486.00/341.20) lat (ms,95%): 164.45 err/s: 0.70 reconn/s: 0.00[ 250s ] thds: 16 tps: 130.20 qps: 2633.83 (r/w/o: 1844.02/466.81/323.00) lat (ms,95%): 183.21 err/s: 1.50 reconn/s: 0.00[ 260s ] thds: 16 tps: 99.90 qps: 2013.80 (r/w/o: 1410.40/350.10/253.30) lat (ms,95%): 227.40 err/s: 0.80 reconn/s: 0.00[ 270s ] thds: 16 tps: 25.50 qps: 510.39 (r/w/o: 358.09/89.70/62.60) lat (ms,95%): 4203.93 err/s: 0.10 reconn/s: 0.00[ 280s ] thds: 16 tps: 136.80 qps: 2767.71 (r/w/o: 1937.90/490.70/339.10) lat (ms,95%): 176.73 err/s: 1.60 reconn/s: 0.00[ 290s ] thds: 16 tps: 133.40 qps: 2695.15 (r/w/o: 1888.03/475.91/331.21) lat (ms,95%): 183.21 err/s: 1.50 reconn/s: 0.00[ 300s ] thds: 16 tps: 132.00 qps: 2660.31 (r/w/o: 1862.81/471.30/326.20) lat (ms,95%): 173.58 err/s: 1.10 reconn/s: 0.00[ 310s ] thds: 16 tps: 113.50 qps: 2292.85 (r/w/o: 1605.57/404.99/282.29) lat (ms,95%): 325.98 err/s: 1.10 reconn/s: 0.00[ 320s ] thds: 16 tps: 123.00 qps: 2482.85 (r/w/o: 1738.83/439.21/304.81) lat (ms,95%): 189.93 err/s: 1.20 reconn/s: 0.00[ 330s ] thds: 16 tps: 135.40 qps: 2720.85 (r/w/o: 1905.96/484.19/330.69) lat (ms,95%): 170.48 err/s: 0.90 reconn/s: 0.00[ 340s ] thds: 16 tps: 127.80 qps: 2575.01 (r/w/o: 1802.61/456.60/315.80) lat (ms,95%): 170.48 err/s: 0.80 reconn/s: 0.00[ 350s ] thds: 16 tps: 133.80 qps: 2696.33 (r/w/o: 1888.62/479.01/328.70) lat (ms,95%): 176.73 err/s: 1.10 reconn/s: 0.00[ 360s ] thds: 16 tps: 135.00 qps: 2723.61 (r/w/o: 1908.10/485.40/330.10) lat (ms,95%): 176.73 err/s: 1.30 reconn/s: 0.00[ 370s ] thds: 16 tps: 123.70 qps: 2501.67 (r/w/o: 1751.48/442.79/307.40) lat (ms,95%): 189.93 err/s: 1.40 reconn/s: 0.00[ 380s ] thds: 16 tps: 135.10 qps: 2732.81 (r/w/o: 1913.81/481.10/337.90) lat (ms,95%): 167.44 err/s: 1.60 reconn/s: 0.00[ 390s ] thds: 16 tps: 133.10 qps: 2678.30 (r/w/o: 1875.60/478.80/323.90) lat (ms,95%): 179.94 err/s: 1.00 reconn/s: 0.00[ 400s ] thds: 16 tps: 130.40 qps: 2627.81 (r/w/o: 1840.00/468.50/319.30) lat (ms,95%): 170.48 err/s: 0.90 reconn/s: 0.00[ 410s ] thds: 16 tps: 135.00 qps: 2712.80 (r/w/o: 1899.70/481.80/331.30) lat (ms,95%): 170.48 err/s: 0.70 reconn/s: 0.00[ 420s ] thds: 16 tps: 122.20 qps: 2476.75 (r/w/o: 1734.66/438.49/303.59) lat (ms,95%): 200.47 err/s: 1.70 reconn/s: 0.00[ 430s ] thds: 16 tps: 132.40 qps: 2660.92 (r/w/o: 1864.31/472.50/324.10) lat (ms,95%): 173.58 err/s: 0.90 reconn/s: 0.00[ 440s ] thds: 16 tps: 127.40 qps: 2582.61 (r/w/o: 1807.91/457.60/317.10) lat (ms,95%): 170.48 err/s: 1.60 reconn/s: 0.00[ 450s ] thds: 16 tps: 134.60 qps: 2718.59 (r/w/o: 1904.00/481.30/333.30) lat (ms,95%): 179.94 err/s: 1.40 reconn/s: 0.00[ 460s ] thds: 16 tps: 124.20 qps: 2507.42 (r/w/o: 1756.31/448.30/302.80) lat (ms,95%): 186.54 err/s: 1.30 reconn/s: 0.00[ 470s ] thds: 16 tps: 110.70 qps: 2235.95 (r/w/o: 1565.87/399.49/270.59) lat (ms,95%): 204.11 err/s: 1.10 reconn/s: 0.00[ 480s ] thds: 16 tps: 56.00 qps: 1126.93 (r/w/o: 789.32/201.51/136.10) lat (ms,95%): 1376.60 err/s: 0.40 reconn/s: 0.00[ 490s ] thds: 16 tps: 132.50 qps: 2676.50 (r/w/o: 1875.50/477.10/323.90) lat (ms,95%): 179.94 err/s: 1.60 reconn/s: 0.00[ 500s ] thds: 16 tps: 133.90 qps: 2706.86 (r/w/o: 1894.97/480.79/331.09) lat (ms,95%): 173.58 err/s: 1.30 reconn/s: 0.00[ 510s ] thds: 16 tps: 132.90 qps: 2683.65 (r/w/o: 1879.83/479.21/324.61) lat (ms,95%): 167.44 err/s: 1.40 reconn/s: 0.00[ 520s ] thds: 16 tps: 124.99 qps: 2521.21 (r/w/o: 1765.87/448.57/306.78) lat (ms,95%): 179.94 err/s: 1.20 reconn/s: 0.00[ 530s ] thds: 16 tps: 113.81 qps: 2298.27 (r/w/o: 1609.52/410.33/278.42) lat (ms,95%): 189.93 err/s: 1.10 reconn/s: 0.00[ 540s ] thds: 16 tps: 130.00 qps: 2621.88 (r/w/o: 1835.89/467.20/318.80) lat (ms,95%): 183.21 err/s: 1.20 reconn/s: 0.00[ 550s ] thds: 16 tps: 111.00 qps: 2249.82 (r/w/o: 1576.32/401.30/272.20) lat (ms,95%): 231.53 err/s: 1.50 reconn/s: 0.00[ 560s ] thds: 16 tps: 105.40 qps: 2117.86 (r/w/o: 1482.57/378.59/256.70) lat (ms,95%): 240.02 err/s: 0.50 reconn/s: 0.00[ 570s ] thds: 16 tps: 104.50 qps: 2099.61 (r/w/o: 1471.50/373.10/255.00) lat (ms,95%): 227.40 err/s: 0.70 reconn/s: 0.00[ 580s ] thds: 16 tps: 128.50 qps: 2589.61 (r/w/o: 1812.51/461.40/315.70) lat (ms,95%): 189.93 err/s: 0.90 reconn/s: 0.00[ 590s ] thds: 16 tps: 139.00 qps: 2807.40 (r/w/o: 1966.00/503.30/338.10) lat (ms,95%): 167.44 err/s: 1.40 reconn/s: 0.00[ 600s ] thds: 16 tps: 161.60 qps: 3243.49 (r/w/o: 2271.69/582.10/389.70) lat (ms,95%): 161.51 err/s: 1.00 reconn/s: 0.00SQL statistics:queries performed:read: 1008098write: 247740other: 183620total: 1439458transactions: 71360 (118.92 per sec.)queries: 1439458 (2398.77 per sec.)ignored errors: 647 (1.08 per sec.)reconnects: 0 (0.00 per sec.)General statistics:total time: 600.0801stotal number of events: 71360Latency (ms):min: 19.00avg: 134.54max: 8283.6995th percentile: 189.93sum: 9600542.94Threads fairness:events (avg/stddev): 4460.0000/8.59execution time (avg/stddev): 600.0339/0.02 - build2 :
jan@galapq:~$ sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/10.6-bug/mysql-test/var/tmp/mysqld.2.sock --table-size=1000000 --mysql-user=root runsysbench 1.0.18 (using system LuaJIT 2.1.0-beta3)Running the test with following options:Number of threads: 16Report intermediate results every 10 second(s)Initializing random number generator from current timeInitializing worker threads...Threads started![ 10s ] thds: 16 tps: 199.65 qps: 4044.13 (r/w/o: 2832.92/445.79/765.42) lat (ms,95%): 142.39 err/s: 1.10 reconn/s: 0.00[ 20s ] thds: 16 tps: 235.91 qps: 4754.25 (r/w/o: 3329.30/636.12/788.82) lat (ms,95%): 104.84 err/s: 1.90 reconn/s: 0.00[ 30s ] thds: 16 tps: 248.61 qps: 5026.51 (r/w/o: 3520.77/753.82/751.92) lat (ms,95%): 95.81 err/s: 2.90 reconn/s: 0.00[ 40s ] thds: 16 tps: 236.50 qps: 4750.05 (r/w/o: 3326.27/763.09/660.69) lat (ms,95%): 102.97 err/s: 1.30 reconn/s: 0.00[ 50s ] thds: 16 tps: 236.20 qps: 4758.33 (r/w/o: 3332.52/793.91/631.90) lat (ms,95%): 104.84 err/s: 1.80 reconn/s: 0.00[ 60s ] thds: 16 tps: 229.70 qps: 4640.34 (r/w/o: 3249.06/787.39/603.89) lat (ms,95%): 102.97 err/s: 2.30 reconn/s: 0.00[ 70s ] thds: 16 tps: 218.00 qps: 4392.84 (r/w/o: 3076.03/759.71/557.10) lat (ms,95%): 110.66 err/s: 1.60 reconn/s: 0.00[ 80s ] thds: 16 tps: 225.10 qps: 4559.69 (r/w/o: 3194.19/796.60/568.90) lat (ms,95%): 106.75 err/s: 3.10 reconn/s: 0.00[ 90s ] thds: 16 tps: 223.50 qps: 4508.95 (r/w/o: 3157.56/786.99/564.39) lat (ms,95%): 110.66 err/s: 2.10 reconn/s: 0.00[ 100s ] thds: 16 tps: 193.10 qps: 3894.24 (r/w/o: 2727.23/679.81/487.20) lat (ms,95%): 132.49 err/s: 1.60 reconn/s: 0.00[ 110s ] thds: 16 tps: 161.80 qps: 3256.30 (r/w/o: 2280.80/576.70/398.80) lat (ms,95%): 153.02 err/s: 1.30 reconn/s: 0.00[ 120s ] thds: 16 tps: 221.80 qps: 4468.90 (r/w/o: 3130.70/784.60/553.60) lat (ms,95%): 110.66 err/s: 2.20 reconn/s: 0.00[ 130s ] thds: 16 tps: 195.30 qps: 3958.61 (r/w/o: 2771.51/699.70/487.40) lat (ms,95%): 104.84 err/s: 2.10 reconn/s: 0.00[ 140s ] thds: 16 tps: 156.70 qps: 3151.04 (r/w/o: 2206.36/554.79/389.89) lat (ms,95%): 150.29 err/s: 0.90 reconn/s: 0.00[ 150s ] thds: 16 tps: 232.20 qps: 4684.56 (r/w/o: 3281.24/828.21/575.11) lat (ms,95%): 102.97 err/s: 2.20 reconn/s: 0.00[ 160s ] thds: 16 tps: 222.50 qps: 4481.55 (r/w/o: 3137.76/796.79/546.99) lat (ms,95%): 108.68 err/s: 1.60 reconn/s: 0.00[ 170s ] thds: 16 tps: 209.40 qps: 4207.58 (r/w/o: 2946.36/745.61/515.61) lat (ms,95%): 112.67 err/s: 1.20 reconn/s: 0.00[ 180s ] thds: 16 tps: 181.10 qps: 3662.85 (r/w/o: 2565.47/652.89/444.49) lat (ms,95%): 137.35 err/s: 2.00 reconn/s: 0.00[ 190s ] thds: 16 tps: 218.70 qps: 4409.49 (r/w/o: 3088.39/786.30/534.80) lat (ms,95%): 114.72 err/s: 1.90 reconn/s: 0.00[ 200s ] thds: 16 tps: 222.69 qps: 4496.31 (r/w/o: 3148.47/797.87/549.98) lat (ms,95%): 112.67 err/s: 2.20 reconn/s: 0.00[ 210s ] thds: 16 tps: 230.01 qps: 4633.02 (r/w/o: 3244.36/823.94/564.73) lat (ms,95%): 106.75 err/s: 1.80 reconn/s: 0.00[ 220s ] thds: 16 tps: 227.70 qps: 4597.53 (r/w/o: 3220.22/820.21/557.10) lat (ms,95%): 104.84 err/s: 2.30 reconn/s: 0.00[ 230s ] thds: 16 tps: 225.10 qps: 4549.93 (r/w/o: 3186.72/808.01/555.20) lat (ms,95%): 106.75 err/s: 2.60 reconn/s: 0.00[ 240s ] thds: 16 tps: 230.00 qps: 4640.74 (r/w/o: 3250.56/823.39/566.79) lat (ms,95%): 102.97 err/s: 2.20 reconn/s: 0.00[ 250s ] thds: 16 tps: 226.50 qps: 4581.66 (r/w/o: 3208.14/819.91/553.61) lat (ms,95%): 106.75 err/s: 2.50 reconn/s: 0.00[ 260s ] thds: 16 tps: 209.10 qps: 4212.40 (r/w/o: 2949.80/748.80/513.80) lat (ms,95%): 116.80 err/s: 1.60 reconn/s: 0.00[ 270s ] thds: 16 tps: 210.90 qps: 4259.75 (r/w/o: 2983.37/759.59/516.79) lat (ms,95%): 121.08 err/s: 2.20 reconn/s: 0.00[ 280s ] thds: 16 tps: 212.50 qps: 4297.92 (r/w/o: 3010.32/764.20/523.40) lat (ms,95%): 112.67 err/s: 2.60 reconn/s: 0.00[ 290s ] thds: 16 tps: 224.90 qps: 4534.23 (r/w/o: 3176.62/805.01/552.60) lat (ms,95%): 104.84 err/s: 2.30 reconn/s: 0.00[ 300s ] thds: 16 tps: 215.80 qps: 4364.98 (r/w/o: 3056.09/777.80/531.10) lat (ms,95%): 112.67 err/s: 2.30 reconn/s: 0.00[ 310s ] thds: 16 tps: 226.50 qps: 4564.40 (r/w/o: 3196.60/814.50/553.30) lat (ms,95%): 108.68 err/s: 1.70 reconn/s: 0.00[ 320s ] thds: 16 tps: 252.30 qps: 5082.41 (r/w/o: 3559.54/904.68/618.19) lat (ms,95%): 101.13 err/s: 1.90 reconn/s: 0.00[ 330s ] thds: 16 tps: 223.90 qps: 4513.06 (r/w/o: 3160.24/801.91/550.91) lat (ms,95%): 112.67 err/s: 2.00 reconn/s: 0.00[ 340s ] thds: 16 tps: 234.60 qps: 4737.37 (r/w/o: 3317.58/847.99/571.80) lat (ms,95%): 101.13 err/s: 2.20 reconn/s: 0.00[ 350s ] thds: 16 tps: 221.40 qps: 4475.55 (r/w/o: 3134.64/798.51/542.41) lat (ms,95%): 114.72 err/s: 2.50 reconn/s: 0.00[ 360s ] thds: 16 tps: 214.00 qps: 4329.71 (r/w/o: 3033.40/772.90/523.40) lat (ms,95%): 116.80 err/s: 2.70 reconn/s: 0.00[ 370s ] thds: 16 tps: 231.00 qps: 4674.92 (r/w/o: 3275.31/835.20/564.40) lat (ms,95%): 102.97 err/s: 3.00 reconn/s: 0.00[ 380s ] thds: 16 tps: 224.00 qps: 4526.86 (r/w/o: 3169.17/816.79/540.90) lat (ms,95%): 106.75 err/s: 2.30 reconn/s: 0.00[ 390s ] thds: 16 tps: 222.50 qps: 4486.60 (r/w/o: 3141.70/804.20/540.70) lat (ms,95%): 108.68 err/s: 1.90 reconn/s: 0.00[ 400s ] thds: 16 tps: 206.10 qps: 4154.43 (r/w/o: 2909.22/744.41/500.80) lat (ms,95%): 121.08 err/s: 1.80 reconn/s: 0.00[ 410s ] thds: 16 tps: 163.50 qps: 3299.97 (r/w/o: 2311.48/591.69/396.80) lat (ms,95%): 150.29 err/s: 1.60 reconn/s: 0.00[ 420s ] thds: 16 tps: 211.70 qps: 4262.09 (r/w/o: 2985.29/763.10/513.70) lat (ms,95%): 121.08 err/s: 1.60 reconn/s: 0.00[ 430s ] thds: 16 tps: 216.40 qps: 4367.63 (r/w/o: 3059.32/788.41/519.90) lat (ms,95%): 108.68 err/s: 2.10 reconn/s: 0.00[ 440s ] thds: 16 tps: 226.90 qps: 4579.55 (r/w/o: 3206.27/823.29/549.99) lat (ms,95%): 104.84 err/s: 2.00 reconn/s: 0.00[ 450s ] thds: 16 tps: 215.60 qps: 4333.02 (r/w/o: 3034.11/779.10/519.80) lat (ms,95%): 112.67 err/s: 1.20 reconn/s: 0.00[ 460s ] thds: 16 tps: 212.10 qps: 4277.09 (r/w/o: 2994.49/766.20/516.40) lat (ms,95%): 114.72 err/s: 1.70 reconn/s: 0.00[ 470s ] thds: 16 tps: 223.40 qps: 4503.76 (r/w/o: 3154.14/809.11/540.51) lat (ms,95%): 108.68 err/s: 2.00 reconn/s: 0.00[ 480s ] thds: 16 tps: 225.90 qps: 4560.18 (r/w/o: 3193.48/822.10/544.60) lat (ms,95%): 104.84 err/s: 2.10 reconn/s: 0.00[ 490s ] thds: 16 tps: 211.60 qps: 4271.09 (r/w/o: 2991.59/762.20/517.30) lat (ms,95%): 118.92 err/s: 2.10 reconn/s: 0.00[ 500s ] thds: 16 tps: 227.90 qps: 4582.86 (r/w/o: 3209.07/819.99/553.79) lat (ms,95%): 102.97 err/s: 1.40 reconn/s: 0.00[ 510s ] thds: 16 tps: 229.60 qps: 4630.35 (r/w/o: 3242.33/837.41/550.61) lat (ms,95%): 102.97 err/s: 1.90 reconn/s: 0.00[ 520s ] thds: 16 tps: 221.80 qps: 4475.12 (r/w/o: 3134.62/803.20/537.30) lat (ms,95%): 108.68 err/s: 2.10 reconn/s: 0.00[ 530s ] thds: 16 tps: 233.50 qps: 4714.11 (r/w/o: 3301.74/844.48/567.89) lat (ms,95%): 102.97 err/s: 2.40 reconn/s: 0.00[ 540s ] thds: 16 tps: 212.40 qps: 4286.63 (r/w/o: 3002.12/768.11/516.40) lat (ms,95%): 114.72 err/s: 2.00 reconn/s: 0.00[ 550s ] thds: 16 tps: 217.90 qps: 4396.43 (r/w/o: 3078.72/790.01/527.70) lat (ms,95%): 110.66 err/s: 2.00 reconn/s: 0.00[ 560s ] thds: 16 tps: 224.50 qps: 4541.21 (r/w/o: 3180.63/817.38/543.19) lat (ms,95%): 110.66 err/s: 2.80 reconn/s: 0.00[ 570s ] thds: 16 tps: 229.00 qps: 4624.08 (r/w/o: 3238.56/833.02/552.51) lat (ms,95%): 102.97 err/s: 2.20 reconn/s: 0.00[ 580s ] thds: 16 tps: 212.10 qps: 4268.30 (r/w/o: 2989.80/765.80/512.70) lat (ms,95%): 127.81 err/s: 1.50 reconn/s: 0.00[ 590s ] thds: 16 tps: 229.10 qps: 4617.68 (r/w/o: 3232.89/830.70/554.10) lat (ms,95%): 104.84 err/s: 1.80 reconn/s: 0.00[ 600s ] thds: 16 tps: 231.80 qps: 4680.37 (r/w/o: 3277.68/840.30/562.40) lat (ms,95%): 104.84 err/s: 2.30 reconn/s: 0.00SQL statistics:queries performed:read: 1853054write: 463743other: 329182total: 2645979transactions: 131156 (218.57 per sec.)queries: 2645979 (4409.46 per sec.)ignored errors: 1205 (2.01 per sec.)reconnects: 0 (0.00 per sec.)General statistics:total time: 600.0670stotal number of events: 131156Latency (ms):min: 10.01avg: 73.20max: 1224.6495th percentile: 112.67sum: 9599990.55Threads fairness:events (avg/stddev): 8197.2500/15.40execution time (avg/stddev): 599.9994/0.02 - Analysis: build2 is still ~2x better.
Update:
- 10.5.8 commit 7da6353b1558adce73320c803f0413c9bbd81185 single master
[ 10s ] thds: 16 tps: 293.63 qps: 5898.13 (r/w/o: 4129.97/649.54/1118.62) lat (ms,95%): 78.60 err/s: 0.10 reconn/s: 0.00[ 20s ] thds: 16 tps: 497.43 qps: 9952.31 (r/w/o: 6967.05/1310.17/1675.09) lat (ms,95%): 52.89 err/s: 0.00 reconn/s: 0.00[ 30s ] thds: 16 tps: 491.59 qps: 9831.65 (r/w/o: 6881.70/1468.78/1481.18) lat (ms,95%): 52.89 err/s: 0.00 reconn/s: 0.00[ 40s ] thds: 16 tps: 472.11 qps: 9436.63 (r/w/o: 6607.56/1511.14/1317.93) lat (ms,95%): 56.84 err/s: 0.00 reconn/s: 0.00[ 50s ] thds: 16 tps: 488.00 qps: 9764.70 (r/w/o: 6833.60/1620.70/1310.40) lat (ms,95%): 52.89 err/s: 0.00 reconn/s: 0.00[ 60s ] thds: 16 tps: 460.79 qps: 9215.24 (r/w/o: 6452.19/1566.67/1196.38) lat (ms,95%): 55.82 err/s: 0.00 reconn/s: 0.00[ 70s ] thds: 16 tps: 446.20 qps: 8908.87 (r/w/o: 6233.75/1540.31/1134.81) lat (ms,95%): 57.87 err/s: 0.00 reconn/s: 0.00[ 80s ] thds: 16 tps: 456.90 qps: 9157.65 (r/w/o: 6411.14/1594.91/1151.61) lat (ms,95%): 58.92 err/s: 0.00 reconn/s: 0.00[ 90s ] thds: 16 tps: 452.59 qps: 9050.45 (r/w/o: 6336.30/1578.67/1135.48) lat (ms,95%): 57.87 err/s: 0.00 reconn/s: 0.00[ 100s ] thds: 16 tps: 443.80 qps: 8875.96 (r/w/o: 6212.04/1555.91/1108.01) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00[ 110s ] thds: 16 tps: 456.30 qps: 9119.52 (r/w/o: 6383.75/1603.99/1131.79) lat (ms,95%): 58.92 err/s: 0.00 reconn/s: 0.00[ 120s ] thds: 16 tps: 421.41 qps: 8432.22 (r/w/o: 5902.98/1490.22/1039.01) lat (ms,95%): 63.32 err/s: 0.00 reconn/s: 0.00[ 130s ] thds: 16 tps: 358.50 qps: 7160.37 (r/w/o: 5012.38/1257.49/890.50) lat (ms,95%): 78.60 err/s: 0.00 reconn/s: 0.00[ 140s ] thds: 16 tps: 378.50 qps: 7583.39 (r/w/o: 5307.89/1346.50/929.00) lat (ms,95%): 71.83 err/s: 0.00 reconn/s: 0.00[ 150s ] thds: 16 tps: 458.00 qps: 9146.66 (r/w/o: 6404.07/1615.09/1127.50) lat (ms,95%): 62.19 err/s: 0.00 reconn/s: 0.00[ 160s ] thds: 16 tps: 459.31 qps: 9196.67 (r/w/o: 6437.42/1643.63/1115.62) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 170s ] thds: 16 tps: 453.69 qps: 9063.78 (r/w/o: 6342.72/1615.38/1105.69) lat (ms,95%): 63.32 err/s: 0.00 reconn/s: 0.00[ 180s ] thds: 16 tps: 455.21 qps: 9117.14 (r/w/o: 6382.70/1613.92/1120.52) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00[ 190s ] thds: 16 tps: 459.20 qps: 9183.93 (r/w/o: 6428.75/1631.79/1123.39) lat (ms,95%): 62.19 err/s: 0.00 reconn/s: 0.00[ 200s ] thds: 16 tps: 445.60 qps: 8910.35 (r/w/o: 6238.44/1584.31/1087.61) lat (ms,95%): 63.32 err/s: 0.00 reconn/s: 0.00[ 210s ] thds: 16 tps: 462.39 qps: 9236.15 (r/w/o: 6464.49/1643.27/1128.38) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 220s ] thds: 16 tps: 465.40 qps: 9310.52 (r/w/o: 6516.51/1656.70/1137.30) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 230s ] thds: 16 tps: 423.30 qps: 8466.52 (r/w/o: 5926.75/1509.49/1030.29) lat (ms,95%): 62.19 err/s: 0.00 reconn/s: 0.00[ 240s ] thds: 16 tps: 463.61 qps: 9276.62 (r/w/o: 6495.75/1650.54/1130.33) lat (ms,95%): 63.32 err/s: 0.00 reconn/s: 0.00[ 250s ] thds: 16 tps: 469.60 qps: 9394.21 (r/w/o: 6575.33/1668.98/1149.89) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 260s ] thds: 16 tps: 458.40 qps: 9170.08 (r/w/o: 6417.79/1631.60/1120.70) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 270s ] thds: 16 tps: 464.40 qps: 9281.50 (r/w/o: 6496.67/1652.12/1132.71) lat (ms,95%): 58.92 err/s: 0.00 reconn/s: 0.00[ 280s ] thds: 16 tps: 463.70 qps: 9278.08 (r/w/o: 6496.69/1652.30/1129.10) lat (ms,95%): 58.92 err/s: 0.00 reconn/s: 0.00[ 290s ] thds: 16 tps: 456.60 qps: 9126.88 (r/w/o: 6387.09/1632.90/1106.90) lat (ms,95%): 62.19 err/s: 0.00 reconn/s: 0.00[ 300s ] thds: 16 tps: 462.90 qps: 9264.85 (r/w/o: 6486.83/1659.81/1118.21) lat (ms,95%): 64.47 err/s: 0.00 reconn/s: 0.00[ 310s ] thds: 16 tps: 462.90 qps: 9260.30 (r/w/o: 6480.90/1653.60/1125.80) lat (ms,95%): 62.19 err/s: 0.00 reconn/s: 0.00[ 320s ] thds: 16 tps: 450.50 qps: 9008.11 (r/w/o: 6306.54/1604.68/1096.89) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 330s ] thds: 16 tps: 462.00 qps: 9232.00 (r/w/o: 6460.97/1652.12/1118.91) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00[ 340s ] thds: 16 tps: 466.39 qps: 9337.74 (r/w/o: 6536.99/1673.57/1127.18) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00[ 350s ] thds: 16 tps: 457.71 qps: 9150.60 (r/w/o: 6406.87/1631.62/1112.11) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 360s ] thds: 16 tps: 466.70 qps: 9336.41 (r/w/o: 6534.71/1671.50/1130.20) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 370s ] thds: 16 tps: 457.00 qps: 9137.41 (r/w/o: 6397.31/1638.70/1101.40) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 380s ] thds: 16 tps: 459.20 qps: 9179.59 (r/w/o: 6425.09/1638.20/1116.30) lat (ms,95%): 63.32 err/s: 0.00 reconn/s: 0.00[ 390s ] thds: 16 tps: 465.89 qps: 9322.76 (r/w/o: 6525.40/1676.78/1120.58) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00[ 400s ] thds: 16 tps: 467.11 qps: 9337.97 (r/w/o: 6537.92/1669.13/1130.92) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00[ 410s ] thds: 16 tps: 448.00 qps: 8967.38 (r/w/o: 6275.19/1613.90/1078.30) lat (ms,95%): 64.47 err/s: 0.00 reconn/s: 0.00[ 420s ] thds: 16 tps: 430.10 qps: 8593.24 (r/w/o: 6015.86/1539.29/1038.09) lat (ms,95%): 68.05 err/s: 0.00 reconn/s: 0.00[ 430s ] thds: 16 tps: 364.10 qps: 7290.68 (r/w/o: 5102.89/1309.50/878.30) lat (ms,95%): 82.96 err/s: 0.00 reconn/s: 0.00[ 440s ] thds: 16 tps: 394.50 qps: 7885.98 (r/w/o: 5521.95/1413.11/950.91) lat (ms,95%): 74.46 err/s: 0.00 reconn/s: 0.00[ 450s ] thds: 16 tps: 460.37 qps: 9201.77 (r/w/o: 6439.53/1653.90/1108.34) lat (ms,95%): 64.47 err/s: 0.00 reconn/s: 0.00[ 460s ] thds: 16 tps: 458.93 qps: 9186.52 (r/w/o: 6430.36/1651.49/1104.66) lat (ms,95%): 62.19 err/s: 0.00 reconn/s: 0.00[ 470s ] thds: 16 tps: 463.40 qps: 9258.04 (r/w/o: 6483.03/1658.21/1116.81) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 480s ] thds: 16 tps: 457.50 qps: 9160.31 (r/w/o: 6409.94/1647.08/1103.29) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 490s ] thds: 16 tps: 465.30 qps: 9301.47 (r/w/o: 6513.38/1669.70/1118.40) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00[ 500s ] thds: 16 tps: 460.70 qps: 9214.52 (r/w/o: 6448.82/1664.60/1101.10) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 510s ] thds: 16 tps: 455.37 qps: 9105.18 (r/w/o: 6373.24/1639.81/1092.14) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 520s ] thds: 16 tps: 461.93 qps: 9242.66 (r/w/o: 6470.29/1662.20/1110.17) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 530s ] thds: 16 tps: 466.37 qps: 9325.99 (r/w/o: 6529.47/1683.59/1112.93) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00[ 540s ] thds: 16 tps: 455.13 qps: 9105.75 (r/w/o: 6371.65/1635.42/1098.68) lat (ms,95%): 62.19 err/s: 0.00 reconn/s: 0.00[ 550s ] thds: 16 tps: 465.10 qps: 9301.61 (r/w/o: 6511.41/1678.50/1111.70) lat (ms,95%): 62.19 err/s: 0.00 reconn/s: 0.00[ 560s ] thds: 16 tps: 458.30 qps: 9168.24 (r/w/o: 6417.56/1660.09/1090.59) lat (ms,95%): 58.92 err/s: 0.00 reconn/s: 0.00[ 570s ] thds: 16 tps: 451.60 qps: 9019.64 (r/w/o: 6313.26/1624.69/1081.69) lat (ms,95%): 62.19 err/s: 0.00 reconn/s: 0.00[ 580s ] thds: 16 tps: 462.59 qps: 9264.17 (r/w/o: 6485.41/1665.28/1113.48) lat (ms,95%): 58.92 err/s: 0.00 reconn/s: 0.00[ 590s ] thds: 16 tps: 462.21 qps: 9244.18 (r/w/o: 6470.92/1673.83/1099.42) lat (ms,95%): 62.19 err/s: 0.00 reconn/s: 0.00[ 600s ] thds: 16 tps: 458.29 qps: 9165.87 (r/w/o: 6416.11/1653.88/1095.88) lat (ms,95%): 63.32 err/s: 0.00 reconn/s: 0.00SQL statistics:queries performed:read: 3787840write: 950649other: 672709total: 5411198transactions: 270559 (450.91 per sec.)queries: 5411198 (9018.21 per sec.)ignored errors: 1 (0.00 per sec.)reconnects: 0 (0.00 per sec.)General statistics:total time: 600.0282stotal number of events: 270559Latency (ms):min: 4.56avg: 35.48max: 958.0595th percentile: 62.19sum: 9599210.55Threads fairness:events (avg/stddev): 16909.9375/26.07execution time (avg/stddev): 599.9507/0.01 - 10.5.8 multi-master
[ 10s ] thds: 16 tps: 345.12 qps: 6954.67 (r/w/o: 4870.86/1258.51/825.31) lat (ms,95%): 94.10 err/s: 1.20 reconn/s: 0.00[ 20s ] thds: 16 tps: 225.61 qps: 4565.30 (r/w/o: 3197.84/822.94/544.52) lat (ms,95%): 106.75 err/s: 2.90 reconn/s: 0.00[ 30s ] thds: 16 tps: 220.90 qps: 4481.26 (r/w/o: 3138.67/812.19/530.40) lat (ms,95%): 108.68 err/s: 3.20 reconn/s: 0.00[ 40s ] thds: 16 tps: 215.80 qps: 4375.96 (r/w/o: 3065.98/787.99/522.00) lat (ms,95%): 112.67 err/s: 3.20 reconn/s: 0.00[ 50s ] thds: 16 tps: 219.90 qps: 4434.56 (r/w/o: 3105.24/804.21/525.11) lat (ms,95%): 110.66 err/s: 1.90 reconn/s: 0.00[ 60s ] thds: 16 tps: 214.30 qps: 4320.92 (r/w/o: 3026.41/784.30/510.20) lat (ms,95%): 112.67 err/s: 2.00 reconn/s: 0.00[ 70s ] thds: 16 tps: 222.90 qps: 4498.76 (r/w/o: 3150.37/817.69/530.69) lat (ms,95%): 110.66 err/s: 2.10 reconn/s: 0.00[ 80s ] thds: 16 tps: 210.50 qps: 4270.41 (r/w/o: 2991.81/770.20/508.40) lat (ms,95%): 112.67 err/s: 3.20 reconn/s: 0.00[ 90s ] thds: 16 tps: 134.90 qps: 2787.60 (r/w/o: 1954.40/504.90/328.30) lat (ms,95%): 158.63 err/s: 4.60 reconn/s: 0.00[ 100s ] thds: 16 tps: 126.70 qps: 2563.78 (r/w/o: 1796.19/463.80/303.80) lat (ms,95%): 170.48 err/s: 1.80 reconn/s: 0.00[ 110s ] thds: 16 tps: 131.70 qps: 2697.14 (r/w/o: 1890.33/488.91/317.91) lat (ms,95%): 158.63 err/s: 3.20 reconn/s: 0.00[ 120s ] thds: 16 tps: 132.30 qps: 2700.09 (r/w/o: 1892.30/495.90/311.90) lat (ms,95%): 161.51 err/s: 2.80 reconn/s: 0.00[ 130s ] thds: 16 tps: 136.10 qps: 2767.38 (r/w/o: 1938.08/503.20/326.10) lat (ms,95%): 161.51 err/s: 2.50 reconn/s: 0.00[ 140s ] thds: 16 tps: 127.60 qps: 2647.71 (r/w/o: 1857.40/479.80/310.50) lat (ms,95%): 167.44 err/s: 4.90 reconn/s: 0.00[ 150s ] thds: 16 tps: 127.10 qps: 2611.19 (r/w/o: 1830.09/473.30/307.80) lat (ms,95%): 164.45 err/s: 3.70 reconn/s: 0.00[ 160s ] thds: 16 tps: 132.30 qps: 2704.61 (r/w/o: 1895.41/487.70/321.50) lat (ms,95%): 161.51 err/s: 3.00 reconn/s: 0.00[ 170s ] thds: 16 tps: 182.00 qps: 3688.92 (r/w/o: 2584.41/672.00/432.50) lat (ms,95%): 142.39 err/s: 2.60 reconn/s: 0.00[ 180s ] thds: 16 tps: 277.69 qps: 5599.18 (r/w/o: 3921.02/1014.28/663.89) lat (ms,95%): 95.81 err/s: 2.40 reconn/s: 0.00[ 190s ] thds: 16 tps: 225.10 qps: 4547.92 (r/w/o: 3185.22/831.10/531.60) lat (ms,95%): 112.67 err/s: 2.50 reconn/s: 0.00[ 200s ] thds: 16 tps: 226.80 qps: 4583.80 (r/w/o: 3210.67/833.12/540.01) lat (ms,95%): 106.75 err/s: 2.50 reconn/s: 0.00[ 210s ] thds: 16 tps: 216.20 qps: 4361.80 (r/w/o: 3054.50/792.00/515.30) lat (ms,95%): 114.72 err/s: 1.90 reconn/s: 0.00[ 220s ] thds: 16 tps: 215.20 qps: 4356.78 (r/w/o: 3051.98/793.20/511.60) lat (ms,95%): 112.67 err/s: 2.80 reconn/s: 0.00[ 230s ] thds: 16 tps: 222.60 qps: 4516.11 (r/w/o: 3164.01/824.00/528.10) lat (ms,95%): 108.68 err/s: 3.40 reconn/s: 0.00[ 240s ] thds: 16 tps: 180.50 qps: 3669.57 (r/w/o: 2570.58/667.49/431.50) lat (ms,95%): 139.85 err/s: 3.20 reconn/s: 0.00[ 250s ] thds: 16 tps: 132.70 qps: 2718.09 (r/w/o: 1905.20/494.80/318.10) lat (ms,95%): 164.45 err/s: 3.30 reconn/s: 0.00[ 260s ] thds: 16 tps: 131.90 qps: 2700.59 (r/w/o: 1892.79/495.20/312.60) lat (ms,95%): 158.63 err/s: 3.30 reconn/s: 0.00[ 270s ] thds: 16 tps: 169.40 qps: 3429.83 (r/w/o: 2402.42/626.40/401.00) lat (ms,95%): 158.63 err/s: 2.20 reconn/s: 0.00[ 280s ] thds: 16 tps: 172.30 qps: 3482.81 (r/w/o: 2439.81/632.70/410.30) lat (ms,95%): 150.29 err/s: 2.00 reconn/s: 0.00[ 290s ] thds: 16 tps: 197.00 qps: 3993.49 (r/w/o: 2798.69/727.80/467.00) lat (ms,95%): 139.85 err/s: 3.10 reconn/s: 0.00[ 300s ] thds: 16 tps: 217.90 qps: 4410.91 (r/w/o: 3088.00/803.90/519.00) lat (ms,95%): 112.67 err/s: 2.50 reconn/s: 0.00[ 310s ] thds: 16 tps: 201.50 qps: 4065.29 (r/w/o: 2846.90/742.90/475.50) lat (ms,95%): 110.66 err/s: 1.80 reconn/s: 0.00[ 320s ] thds: 16 tps: 214.10 qps: 4319.79 (r/w/o: 3025.39/785.50/508.90) lat (ms,95%): 116.80 err/s: 2.00 reconn/s: 0.00[ 330s ] thds: 16 tps: 228.80 qps: 4603.62 (r/w/o: 3224.22/838.00/541.40) lat (ms,95%): 106.75 err/s: 1.60 reconn/s: 0.00[ 340s ] thds: 16 tps: 217.40 qps: 4384.38 (r/w/o: 3070.18/799.40/514.80) lat (ms,95%): 112.67 err/s: 1.80 reconn/s: 0.00[ 350s ] thds: 16 tps: 276.70 qps: 5568.65 (r/w/o: 3899.03/1019.21/650.41) lat (ms,95%): 94.10 err/s: 1.80 reconn/s: 0.00[ 360s ] thds: 16 tps: 236.60 qps: 4773.20 (r/w/o: 3342.90/873.40/556.90) lat (ms,95%): 106.75 err/s: 2.30 reconn/s: 0.00[ 370s ] thds: 16 tps: 223.80 qps: 4520.81 (r/w/o: 3167.11/827.30/526.40) lat (ms,95%): 108.68 err/s: 2.30 reconn/s: 0.00[ 380s ] thds: 16 tps: 219.30 qps: 4424.77 (r/w/o: 3098.18/811.89/514.70) lat (ms,95%): 112.67 err/s: 2.00 reconn/s: 0.00[ 390s ] thds: 16 tps: 208.30 qps: 4209.52 (r/w/o: 2948.41/770.60/490.50) lat (ms,95%): 112.67 err/s: 2.30 reconn/s: 0.00[ 400s ] thds: 16 tps: 211.20 qps: 4280.94 (r/w/o: 2999.86/782.89/498.19) lat (ms,95%): 112.67 err/s: 3.50 reconn/s: 0.00[ 410s ] thds: 16 tps: 213.90 qps: 4333.82 (r/w/o: 3034.91/793.30/505.60) lat (ms,95%): 112.67 err/s: 2.50 reconn/s: 0.00[ 420s ] thds: 16 tps: 220.37 qps: 4453.06 (r/w/o: 3119.15/821.18/512.73) lat (ms,95%): 108.68 err/s: 2.40 reconn/s: 0.00[ 430s ] thds: 16 tps: 221.33 qps: 4457.56 (r/w/o: 3121.26/818.82/517.48) lat (ms,95%): 110.66 err/s: 1.60 reconn/s: 0.00[ 440s ] thds: 16 tps: 210.60 qps: 4252.17 (r/w/o: 2978.08/777.69/496.40) lat (ms,95%): 116.80 err/s: 2.30 reconn/s: 0.00[ 450s ] thds: 16 tps: 217.20 qps: 4384.23 (r/w/o: 3071.42/805.81/507.00) lat (ms,95%): 114.72 err/s: 2.20 reconn/s: 0.00[ 460s ] thds: 16 tps: 216.40 qps: 4373.32 (r/w/o: 3061.72/803.80/507.80) lat (ms,95%): 110.66 err/s: 2.10 reconn/s: 0.00[ 470s ] thds: 16 tps: 215.20 qps: 4342.93 (r/w/o: 3041.62/796.31/505.00) lat (ms,95%): 114.72 err/s: 2.10 reconn/s: 0.00[ 480s ] thds: 16 tps: 250.60 qps: 5038.09 (r/w/o: 3527.69/923.90/586.50) lat (ms,95%): 102.97 err/s: 1.50 reconn/s: 0.00[ 490s ] thds: 16 tps: 157.30 qps: 3197.66 (r/w/o: 2240.87/586.39/370.39) lat (ms,95%): 147.61 err/s: 2.60 reconn/s: 0.00[ 500s ] thds: 16 tps: 132.10 qps: 2702.74 (r/w/o: 1894.13/496.01/312.60) lat (ms,95%): 170.48 err/s: 3.30 reconn/s: 0.00[ 510s ] thds: 16 tps: 133.10 qps: 2726.95 (r/w/o: 1911.07/502.79/313.09) lat (ms,95%): 158.63 err/s: 3.30 reconn/s: 0.00[ 520s ] thds: 16 tps: 187.90 qps: 3804.46 (r/w/o: 2665.34/699.91/439.21) lat (ms,95%): 137.35 err/s: 2.50 reconn/s: 0.00[ 530s ] thds: 16 tps: 216.70 qps: 4377.89 (r/w/o: 3065.69/804.00/508.20) lat (ms,95%): 108.68 err/s: 2.40 reconn/s: 0.00[ 540s ] thds: 16 tps: 224.80 qps: 4545.93 (r/w/o: 3184.22/834.71/527.00) lat (ms,95%): 108.68 err/s: 2.50 reconn/s: 0.00[ 550s ] thds: 16 tps: 135.00 qps: 2762.28 (r/w/o: 1936.49/503.70/322.10) lat (ms,95%): 155.80 err/s: 3.40 reconn/s: 0.00[ 560s ] thds: 16 tps: 123.10 qps: 2528.59 (r/w/o: 1772.09/464.90/291.60) lat (ms,95%): 161.51 err/s: 3.40 reconn/s: 0.00[ 570s ] thds: 16 tps: 125.40 qps: 2547.62 (r/w/o: 1784.81/468.40/294.40) lat (ms,95%): 200.47 err/s: 2.10 reconn/s: 0.00[ 580s ] thds: 16 tps: 174.10 qps: 3511.55 (r/w/o: 2459.77/643.49/408.29) lat (ms,95%): 142.39 err/s: 1.60 reconn/s: 0.00[ 590s ] thds: 16 tps: 209.10 qps: 4223.90 (r/w/o: 2958.00/775.50/490.40) lat (ms,95%): 121.08 err/s: 2.30 reconn/s: 0.00[ 600s ] thds: 16 tps: 207.80 qps: 4188.87 (r/w/o: 2933.45/772.71/482.71) lat (ms,95%): 121.08 err/s: 1.70 reconn/s: 0.00SQL statistics:queries performed:read: 1662570write: 433091other: 277849total: 2373510transactions: 117224 (195.35 per sec.)queries: 2373510 (3955.44 per sec.)ignored errors: 1531 (2.55 per sec.)reconnects: 0 (0.00 per sec.)General statistics:total time: 600.0602stotal number of events: 117224Latency (ms):min: 7.93avg: 81.90max: 774.6695th percentile: 137.35sum: 9600265.28Threads fairness:events (avg/stddev): 7326.5000/12.52execution time (avg/stddev): 600.0166/0.01 - Analysis: About the same as 10.6 with build2
| Fix Version/s | 10.6 [ 24028 ] | |
| Fix Version/s | 10.6.0 [ 24431 ] |
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Status | In Progress [ 3 ] | Stalled [ 10000 ] |
| Status | Stalled [ 10000 ] | In Review [ 10002 ] |
I used now again two builds and galapq:
- build1: 10.6 with Seppo's PR
- build2: 10.5 commit e7ddf46632db84f55af85cd4f401d59c6ab29242
- I used only multi-master i.e. sysbench to nodes: 1 and 2 (node 3 as slave). I will report results from one node only.
- I used default innodb-lock-wait-timeout
- build1:
jan@galapq:~$ sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/mariadb-server/mysql-test/var/tmp/mysqld.2.sock --table-size=1000000 --mysql-user=root runsysbench 1.0.18 (using system LuaJIT 2.1.0-beta3)Running the test with following options:Number of threads: 16Report intermediate results every 10 second(s)Initializing random number generator from current timeInitializing worker threads...Threads started![ 10s ] thds: 16 tps: 114.57 qps: 2329.86 (r/w/o: 1633.32/251.04/445.50) lat (ms,95%): 231.53 err/s: 0.60 reconn/s: 0.00[ 20s ] thds: 16 tps: 132.60 qps: 2665.99 (r/w/o: 1865.86/331.41/468.71) lat (ms,95%): 167.44 err/s: 0.70 reconn/s: 0.00[ 30s ] thds: 16 tps: 156.50 qps: 3140.33 (r/w/o: 2198.52/431.40/510.40) lat (ms,95%): 153.02 err/s: 0.40 reconn/s: 0.00[ 40s ] thds: 16 tps: 154.99 qps: 3133.00 (r/w/o: 2194.66/460.47/477.87) lat (ms,95%): 155.80 err/s: 2.00 reconn/s: 0.00[ 50s ] thds: 16 tps: 114.01 qps: 2313.74 (r/w/o: 1621.60/358.92/333.22) lat (ms,95%): 211.60 err/s: 1.70 reconn/s: 0.00[ 60s ] thds: 16 tps: 132.50 qps: 2666.29 (r/w/o: 1866.39/422.80/377.10) lat (ms,95%): 186.54 err/s: 0.80 reconn/s: 0.00[ 70s ] thds: 16 tps: 147.50 qps: 2968.23 (r/w/o: 2078.22/484.30/405.70) lat (ms,95%): 170.48 err/s: 1.20 reconn/s: 0.00[ 80s ] thds: 16 tps: 133.90 qps: 2702.08 (r/w/o: 1892.08/453.40/356.60) lat (ms,95%): 167.44 err/s: 0.90 reconn/s: 0.00[ 90s ] thds: 16 tps: 148.90 qps: 2995.05 (r/w/o: 2097.17/505.19/392.69) lat (ms,95%): 155.80 err/s: 0.90 reconn/s: 0.00[ 100s ] thds: 16 tps: 138.60 qps: 2798.66 (r/w/o: 1960.04/470.61/368.01) lat (ms,95%): 176.73 err/s: 1.40 reconn/s: 0.00[ 110s ] thds: 16 tps: 120.40 qps: 2428.77 (r/w/o: 1700.98/420.39/307.40) lat (ms,95%): 183.21 err/s: 1.10 reconn/s: 0.00[ 120s ] thds: 16 tps: 118.70 qps: 2403.69 (r/w/o: 1683.80/417.50/302.40) lat (ms,95%): 186.54 err/s: 1.60 reconn/s: 0.00[ 130s ] thds: 16 tps: 133.10 qps: 2685.62 (r/w/o: 1880.62/472.50/332.50) lat (ms,95%): 173.58 err/s: 1.20 reconn/s: 0.00[ 140s ] thds: 16 tps: 132.70 qps: 2675.99 (r/w/o: 1874.29/465.70/336.00) lat (ms,95%): 173.58 err/s: 1.20 reconn/s: 0.00[ 150s ] thds: 16 tps: 131.90 qps: 2661.98 (r/w/o: 1863.99/466.10/331.90) lat (ms,95%): 176.73 err/s: 1.30 reconn/s: 0.00[ 160s ] thds: 16 tps: 132.70 qps: 2673.55 (r/w/o: 1872.93/466.11/334.51) lat (ms,95%): 173.58 err/s: 1.00 reconn/s: 0.00[ 170s ] thds: 16 tps: 126.89 qps: 2559.31 (r/w/o: 1791.90/444.35/323.06) lat (ms,95%): 179.94 err/s: 1.30 reconn/s: 0.00[ 180s ] thds: 16 tps: 124.61 qps: 2516.37 (r/w/o: 1762.49/443.05/310.83) lat (ms,95%): 186.54 err/s: 1.20 reconn/s: 0.00[ 190s ] thds: 16 tps: 136.50 qps: 2752.70 (r/w/o: 1927.60/486.80/338.30) lat (ms,95%): 170.48 err/s: 1.10 reconn/s: 0.00[ 200s ] thds: 16 tps: 136.80 qps: 2742.79 (r/w/o: 1920.29/480.20/342.30) lat (ms,95%): 170.48 err/s: 0.50 reconn/s: 0.00[ 210s ] thds: 16 tps: 131.10 qps: 2646.12 (r/w/o: 1853.12/466.20/326.80) lat (ms,95%): 176.73 err/s: 1.10 reconn/s: 0.00[ 220s ] thds: 16 tps: 143.00 qps: 2885.61 (r/w/o: 2021.51/513.50/350.60) lat (ms,95%): 161.51 err/s: 1.60 reconn/s: 0.00[ 230s ] thds: 16 tps: 135.50 qps: 2755.06 (r/w/o: 1929.27/490.09/335.70) lat (ms,95%): 176.73 err/s: 2.20 reconn/s: 0.00[ 240s ] thds: 16 tps: 129.50 qps: 2622.23 (r/w/o: 1836.82/465.01/320.40) lat (ms,95%): 179.94 err/s: 1.60 reconn/s: 0.00[ 250s ] thds: 16 tps: 137.30 qps: 2766.60 (r/w/o: 1937.60/491.00/338.00) lat (ms,95%): 170.48 err/s: 1.10 reconn/s: 0.00[ 260s ] thds: 16 tps: 127.80 qps: 2565.11 (r/w/o: 1796.20/455.10/313.80) lat (ms,95%): 176.73 err/s: 0.50 reconn/s: 0.00[ 270s ] thds: 16 tps: 131.20 qps: 2666.19 (r/w/o: 1867.59/471.60/327.00) lat (ms,95%): 186.54 err/s: 2.20 reconn/s: 0.00[ 280s ] thds: 16 tps: 133.00 qps: 2688.49 (r/w/o: 1883.00/476.60/328.90) lat (ms,95%): 176.73 err/s: 1.50 reconn/s: 0.00[ 290s ] thds: 16 tps: 131.80 qps: 2655.50 (r/w/o: 1859.80/471.00/324.70) lat (ms,95%): 170.48 err/s: 1.10 reconn/s: 0.00[ 300s ] thds: 16 tps: 133.80 qps: 2702.69 (r/w/o: 1893.19/480.10/329.40) lat (ms,95%): 170.48 err/s: 1.40 reconn/s: 0.00[ 310s ] thds: 16 tps: 133.90 qps: 2707.42 (r/w/o: 1896.01/480.20/331.20) lat (ms,95%): 170.48 err/s: 1.50 reconn/s: 0.00[ 320s ] thds: 16 tps: 135.10 qps: 2722.69 (r/w/o: 1906.80/479.90/336.00) lat (ms,95%): 183.21 err/s: 1.10 reconn/s: 0.00[ 330s ] thds: 16 tps: 128.60 qps: 2584.46 (r/w/o: 1809.87/460.49/314.09) lat (ms,95%): 186.54 err/s: 0.70 reconn/s: 0.00[ 340s ] thds: 16 tps: 136.00 qps: 2741.55 (r/w/o: 1919.74/486.71/335.11) lat (ms,95%): 170.48 err/s: 1.10 reconn/s: 0.00[ 350s ] thds: 16 tps: 121.90 qps: 2453.61 (r/w/o: 1719.00/431.60/303.00) lat (ms,95%): 186.54 err/s: 1.40 reconn/s: 0.00[ 360s ] thds: 16 tps: 97.90 qps: 1984.20 (r/w/o: 1389.00/355.00/240.20) lat (ms,95%): 240.02 err/s: 0.80 reconn/s: 0.00[ 370s ] thds: 16 tps: 129.20 qps: 2606.49 (r/w/o: 1825.59/462.80/318.10) lat (ms,95%): 186.54 err/s: 1.20 reconn/s: 0.00[ 380s ] thds: 16 tps: 131.80 qps: 2661.20 (r/w/o: 1863.70/474.90/322.60) lat (ms,95%): 176.73 err/s: 1.40 reconn/s: 0.00[ 390s ] thds: 16 tps: 128.90 qps: 2601.28 (r/w/o: 1823.18/462.00/316.10) lat (ms,95%): 186.54 err/s: 1.40 reconn/s: 0.00[ 400s ] thds: 16 tps: 129.10 qps: 2603.14 (r/w/o: 1821.53/462.71/318.90) lat (ms,95%): 179.94 err/s: 0.90 reconn/s: 0.00[ 410s ] thds: 16 tps: 127.50 qps: 2571.98 (r/w/o: 1801.39/460.60/310.00) lat (ms,95%): 189.93 err/s: 1.20 reconn/s: 0.00[ 420s ] thds: 16 tps: 136.10 qps: 2740.94 (r/w/o: 1919.36/486.39/335.19) lat (ms,95%): 183.21 err/s: 1.00 reconn/s: 0.00[ 430s ] thds: 16 tps: 123.80 qps: 2509.35 (r/w/o: 1758.03/449.21/302.11) lat (ms,95%): 179.94 err/s: 1.70 reconn/s: 0.00[ 440s ] thds: 16 tps: 116.00 qps: 2335.22 (r/w/o: 1635.22/416.60/283.40) lat (ms,95%): 207.82 err/s: 0.80 reconn/s: 0.00[ 450s ] thds: 16 tps: 121.60 qps: 2449.29 (r/w/o: 1714.99/436.50/297.80) lat (ms,95%): 193.38 err/s: 0.90 reconn/s: 0.00[ 460s ] thds: 16 tps: 129.40 qps: 2625.49 (r/w/o: 1839.59/468.10/317.80) lat (ms,95%): 170.48 err/s: 2.00 reconn/s: 0.00[ 470s ] thds: 16 tps: 118.90 qps: 2389.42 (r/w/o: 1673.11/424.90/291.40) lat (ms,95%): 183.21 err/s: 0.70 reconn/s: 0.00[ 480s ] thds: 16 tps: 111.90 qps: 2261.00 (r/w/o: 1583.30/404.20/273.50) lat (ms,95%): 200.47 err/s: 1.10 reconn/s: 0.00[ 490s ] thds: 16 tps: 117.50 qps: 2361.81 (r/w/o: 1653.80/420.80/287.20) lat (ms,95%): 211.60 err/s: 1.10 reconn/s: 0.00[ 500s ] thds: 16 tps: 129.20 qps: 2604.07 (r/w/o: 1823.68/461.89/318.50) lat (ms,95%): 186.54 err/s: 0.70 reconn/s: 0.00[ 510s ] thds: 16 tps: 110.20 qps: 2217.82 (r/w/o: 1554.11/398.10/265.60) lat (ms,95%): 196.89 err/s: 0.90 reconn/s: 0.00[ 520s ] thds: 16 tps: 118.70 qps: 2398.30 (r/w/o: 1678.60/430.10/289.60) lat (ms,95%): 204.11 err/s: 1.10 reconn/s: 0.00[ 530s ] thds: 16 tps: 122.70 qps: 2464.09 (r/w/o: 1725.39/438.10/300.60) lat (ms,95%): 183.21 err/s: 0.50 reconn/s: 0.00[ 540s ] thds: 16 tps: 129.30 qps: 2615.68 (r/w/o: 1831.99/468.20/315.50) lat (ms,95%): 183.21 err/s: 1.50 reconn/s: 0.00[ 550s ] thds: 16 tps: 124.60 qps: 2511.38 (r/w/o: 1758.99/449.10/303.30) lat (ms,95%): 186.54 err/s: 1.20 reconn/s: 0.00[ 560s ] thds: 16 tps: 139.80 qps: 2824.05 (r/w/o: 1977.74/498.31/348.01) lat (ms,95%): 167.44 err/s: 1.50 reconn/s: 0.00[ 570s ] thds: 16 tps: 129.10 qps: 2604.89 (r/w/o: 1824.09/465.90/314.90) lat (ms,95%): 173.58 err/s: 1.00 reconn/s: 0.00[ 580s ] thds: 16 tps: 42.10 qps: 845.88 (r/w/o: 592.79/148.90/104.20) lat (ms,95%): 1903.57 err/s: 0.30 reconn/s: 0.00[ 590s ] thds: 16 tps: 113.60 qps: 2296.66 (r/w/o: 1608.04/408.51/280.11) lat (ms,95%): 211.60 err/s: 1.20 reconn/s: 0.00[ 600s ] thds: 16 tps: 140.80 qps: 2843.70 (r/w/o: 1992.20/509.10/342.40) lat (ms,95%): 186.54 err/s: 1.50 reconn/s: 0.00SQL statistics:queries performed:read: 1084860write: 267127other: 197084total: 1549071transactions: 76792 (127.97 per sec.)queries: 1549071 (2581.39 per sec.)ignored errors: 698 (1.16 per sec.)reconnects: 0 (0.00 per sec.)General statistics:total time: 600.0904stotal number of events: 76792Latency (ms):min: 21.94avg: 125.02max: 2436.6795th percentile: 183.21sum: 9600345.70Threads fairness:events (avg/stddev): 4799.5000/6.06execution time (avg/stddev): 600.0216/0.03
- build2:
jan@galapq:~$ sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/mariadb-server/mysql-test/var/tmp/mysqld.2.sock --table-size=1000000 --mysql-user=root runsysbench 1.0.18 (using system LuaJIT 2.1.0-beta3)Running the test with following options:Number of threads: 16Report intermediate results every 10 second(s)Initializing random number generator from current timeInitializing worker threads...Threads started![ 10s ] thds: 16 tps: 146.36 qps: 2968.75 (r/w/o: 2079.88/333.22/555.66) lat (ms,95%): 193.38 err/s: 0.60 reconn/s: 0.00[ 20s ] thds: 16 tps: 206.91 qps: 4154.85 (r/w/o: 2909.37/535.93/709.54) lat (ms,95%): 112.67 err/s: 0.90 reconn/s: 0.00[ 30s ] thds: 16 tps: 221.60 qps: 4462.11 (r/w/o: 3124.81/652.30/685.00) lat (ms,95%): 108.68 err/s: 1.70 reconn/s: 0.00[ 40s ] thds: 16 tps: 236.70 qps: 4762.84 (r/w/o: 3334.76/754.39/673.69) lat (ms,95%): 99.33 err/s: 1.40 reconn/s: 0.00[ 50s ] thds: 16 tps: 231.40 qps: 4676.36 (r/w/o: 3275.54/768.81/632.01) lat (ms,95%): 106.75 err/s: 2.60 reconn/s: 0.00[ 60s ] thds: 16 tps: 234.10 qps: 4723.63 (r/w/o: 3308.65/792.49/622.49) lat (ms,95%): 104.84 err/s: 2.30 reconn/s: 0.00[ 70s ] thds: 16 tps: 213.20 qps: 4307.82 (r/w/o: 3017.81/742.40/547.60) lat (ms,95%): 112.67 err/s: 2.50 reconn/s: 0.00[ 80s ] thds: 16 tps: 210.90 qps: 4263.35 (r/w/o: 2984.83/737.81/540.71) lat (ms,95%): 116.80 err/s: 2.10 reconn/s: 0.00[ 90s ] thds: 16 tps: 186.00 qps: 3751.39 (r/w/o: 2627.00/654.70/469.70) lat (ms,95%): 116.80 err/s: 1.60 reconn/s: 0.00[ 100s ] thds: 16 tps: 195.90 qps: 3951.09 (r/w/o: 2767.30/692.60/491.20) lat (ms,95%): 114.72 err/s: 1.80 reconn/s: 0.00[ 110s ] thds: 16 tps: 194.30 qps: 3934.37 (r/w/o: 2755.68/690.39/488.30) lat (ms,95%): 121.08 err/s: 2.50 reconn/s: 0.00[ 120s ] thds: 16 tps: 110.60 qps: 2229.02 (r/w/o: 1561.02/393.60/274.40) lat (ms,95%): 434.83 err/s: 0.90 reconn/s: 0.00[ 130s ] thds: 16 tps: 228.10 qps: 4591.86 (r/w/o: 3215.77/813.49/562.60) lat (ms,95%): 104.84 err/s: 1.60 reconn/s: 0.00[ 140s ] thds: 16 tps: 218.10 qps: 4404.94 (r/w/o: 3085.36/778.19/541.39) lat (ms,95%): 116.80 err/s: 2.30 reconn/s: 0.00[ 150s ] thds: 16 tps: 218.00 qps: 4422.69 (r/w/o: 3097.86/786.22/538.61) lat (ms,95%): 118.92 err/s: 3.40 reconn/s: 0.00[ 160s ] thds: 16 tps: 266.80 qps: 5378.46 (r/w/o: 3766.57/953.99/657.89) lat (ms,95%): 101.13 err/s: 2.10 reconn/s: 0.00[ 170s ] thds: 16 tps: 222.30 qps: 4484.04 (r/w/o: 3140.23/792.31/551.50) lat (ms,95%): 116.80 err/s: 2.00 reconn/s: 0.00[ 180s ] thds: 16 tps: 191.60 qps: 3872.36 (r/w/o: 2712.67/683.19/476.49) lat (ms,95%): 118.92 err/s: 2.30 reconn/s: 0.00[ 190s ] thds: 16 tps: 220.40 qps: 4443.70 (r/w/o: 3111.30/793.10/539.30) lat (ms,95%): 110.66 err/s: 1.80 reconn/s: 0.00[ 200s ] thds: 16 tps: 205.40 qps: 4157.32 (r/w/o: 2912.11/739.70/505.50) lat (ms,95%): 118.92 err/s: 2.60 reconn/s: 0.00[ 210s ] thds: 16 tps: 193.40 qps: 3911.02 (r/w/o: 2739.71/694.00/477.30) lat (ms,95%): 134.90 err/s: 2.20 reconn/s: 0.00[ 220s ] thds: 16 tps: 174.10 qps: 3520.51 (r/w/o: 2465.41/624.30/430.80) lat (ms,95%): 155.80 err/s: 2.00 reconn/s: 0.00[ 230s ] thds: 16 tps: 224.00 qps: 4527.30 (r/w/o: 3171.00/800.90/555.40) lat (ms,95%): 108.68 err/s: 2.50 reconn/s: 0.00[ 240s ] thds: 16 tps: 246.30 qps: 4957.98 (r/w/o: 3471.99/884.50/601.50) lat (ms,95%): 106.75 err/s: 1.70 reconn/s: 0.00[ 250s ] thds: 16 tps: 246.40 qps: 4960.70 (r/w/o: 3473.40/879.40/607.90) lat (ms,95%): 104.84 err/s: 1.80 reconn/s: 0.00[ 260s ] thds: 16 tps: 214.50 qps: 4317.08 (r/w/o: 3023.88/764.90/528.30) lat (ms,95%): 125.52 err/s: 1.40 reconn/s: 0.00[ 270s ] thds: 16 tps: 124.70 qps: 2500.28 (r/w/o: 1750.29/446.40/303.60) lat (ms,95%): 520.62 err/s: 0.40 reconn/s: 0.00[ 280s ] thds: 16 tps: 213.80 qps: 4316.53 (r/w/o: 3023.02/768.21/525.30) lat (ms,95%): 114.72 err/s: 2.10 reconn/s: 0.00[ 290s ] thds: 16 tps: 217.90 qps: 4385.11 (r/w/o: 3071.01/780.40/533.70) lat (ms,95%): 116.80 err/s: 1.40 reconn/s: 0.00[ 300s ] thds: 16 tps: 216.70 qps: 4368.85 (r/w/o: 3059.27/780.19/529.39) lat (ms,95%): 110.66 err/s: 1.90 reconn/s: 0.00[ 310s ] thds: 16 tps: 210.60 qps: 4244.68 (r/w/o: 2972.06/757.41/515.21) lat (ms,95%): 110.66 err/s: 1.80 reconn/s: 0.00[ 320s ] thds: 16 tps: 224.90 qps: 4528.80 (r/w/o: 3172.30/802.30/554.20) lat (ms,95%): 110.66 err/s: 1.50 reconn/s: 0.00[ 330s ] thds: 16 tps: 273.90 qps: 5519.28 (r/w/o: 3864.49/982.70/672.10) lat (ms,95%): 102.97 err/s: 2.20 reconn/s: 0.00[ 340s ] thds: 16 tps: 232.70 qps: 4697.49 (r/w/o: 3289.49/841.10/566.90) lat (ms,95%): 110.66 err/s: 2.20 reconn/s: 0.00[ 350s ] thds: 16 tps: 208.10 qps: 4196.32 (r/w/o: 2938.61/748.80/508.90) lat (ms,95%): 125.52 err/s: 1.90 reconn/s: 0.00[ 360s ] thds: 16 tps: 218.00 qps: 4392.17 (r/w/o: 3075.78/787.70/528.70) lat (ms,95%): 108.68 err/s: 1.60 reconn/s: 0.00[ 370s ] thds: 16 tps: 215.10 qps: 4353.27 (r/w/o: 3049.18/782.70/521.40) lat (ms,95%): 123.28 err/s: 2.70 reconn/s: 0.00[ 380s ] thds: 16 tps: 228.60 qps: 4606.73 (r/w/o: 3226.32/824.11/556.30) lat (ms,95%): 102.97 err/s: 2.00 reconn/s: 0.00[ 390s ] thds: 16 tps: 228.90 qps: 4611.72 (r/w/o: 3229.32/829.40/553.00) lat (ms,95%): 108.68 err/s: 1.70 reconn/s: 0.00[ 400s ] thds: 16 tps: 219.90 qps: 4428.03 (r/w/o: 3100.75/797.49/529.79) lat (ms,95%): 110.66 err/s: 1.60 reconn/s: 0.00[ 410s ] thds: 16 tps: 195.70 qps: 3951.02 (r/w/o: 2767.62/703.90/479.50) lat (ms,95%): 127.81 err/s: 2.00 reconn/s: 0.00[ 420s ] thds: 16 tps: 206.60 qps: 4163.98 (r/w/o: 2915.89/743.70/504.40) lat (ms,95%): 116.80 err/s: 1.70 reconn/s: 0.00[ 430s ] thds: 16 tps: 220.50 qps: 4454.91 (r/w/o: 3119.71/798.40/536.80) lat (ms,95%): 110.66 err/s: 2.20 reconn/s: 0.00[ 440s ] thds: 16 tps: 187.10 qps: 3770.26 (r/w/o: 2640.87/680.49/448.89) lat (ms,95%): 130.13 err/s: 1.60 reconn/s: 0.00[ 450s ] thds: 16 tps: 191.50 qps: 3859.69 (r/w/o: 2702.96/690.12/466.61) lat (ms,95%): 130.13 err/s: 1.50 reconn/s: 0.00[ 460s ] thds: 16 tps: 220.00 qps: 4436.04 (r/w/o: 3106.56/800.39/529.09) lat (ms,95%): 114.72 err/s: 2.10 reconn/s: 0.00[ 470s ] thds: 16 tps: 205.80 qps: 4152.36 (r/w/o: 2908.14/748.71/495.51) lat (ms,95%): 116.80 err/s: 1.80 reconn/s: 0.00[ 480s ] thds: 16 tps: 219.20 qps: 4422.27 (r/w/o: 3096.48/796.50/529.30) lat (ms,95%): 112.67 err/s: 1.90 reconn/s: 0.00[ 490s ] thds: 16 tps: 221.21 qps: 4477.74 (r/w/o: 3136.87/805.48/535.39) lat (ms,95%): 112.67 err/s: 2.90 reconn/s: 0.00[ 500s ] thds: 16 tps: 217.68 qps: 4389.27 (r/w/o: 3073.77/788.20/527.30) lat (ms,95%): 114.72 err/s: 1.90 reconn/s: 0.00[ 510s ] thds: 16 tps: 195.40 qps: 3953.08 (r/w/o: 2768.96/709.71/474.41) lat (ms,95%): 139.85 err/s: 2.50 reconn/s: 0.00[ 520s ] thds: 16 tps: 174.10 qps: 3526.29 (r/w/o: 2469.59/633.00/423.70) lat (ms,95%): 158.63 err/s: 2.10 reconn/s: 0.00[ 530s ] thds: 16 tps: 209.60 qps: 4239.02 (r/w/o: 2969.41/759.30/510.30) lat (ms,95%): 116.80 err/s: 2.60 reconn/s: 0.00[ 540s ] thds: 16 tps: 216.80 qps: 4365.40 (r/w/o: 3056.30/778.60/530.50) lat (ms,95%): 112.67 err/s: 2.10 reconn/s: 0.00[ 550s ] thds: 16 tps: 191.60 qps: 3888.09 (r/w/o: 2724.30/701.60/462.20) lat (ms,95%): 116.80 err/s: 2.30 reconn/s: 0.00[ 560s ] thds: 16 tps: 226.10 qps: 4561.23 (r/w/o: 3194.75/820.39/546.09) lat (ms,95%): 114.72 err/s: 2.10 reconn/s: 0.00[ 570s ] thds: 16 tps: 212.20 qps: 4285.20 (r/w/o: 3001.60/771.40/512.20) lat (ms,95%): 114.72 err/s: 2.20 reconn/s: 0.00[ 580s ] thds: 16 tps: 222.30 qps: 4488.55 (r/w/o: 3142.97/811.09/534.49) lat (ms,95%): 110.66 err/s: 2.20 reconn/s: 0.00[ 590s ] thds: 16 tps: 250.80 qps: 5043.57 (r/w/o: 3532.05/911.71/599.81) lat (ms,95%): 104.84 err/s: 1.50 reconn/s: 0.00[ 600s ] thds: 16 tps: 256.20 qps: 5153.43 (r/w/o: 3608.45/925.49/619.49) lat (ms,95%): 106.75 err/s: 1.90 reconn/s: 0.00SQL statistics:queries performed:read: 1798986write: 450460other: 319311total: 2568757transactions: 127332 (212.21 per sec.)queries: 2568757 (4281.03 per sec.)ignored errors: 1167 (1.94 per sec.)reconnects: 0 (0.00 per sec.)General statistics:total time: 600.0307stotal number of events: 127332Latency (ms):min: 8.60avg: 75.39max: 2662.3795th percentile: 116.80sum: 9599743.57Threads fairness:events (avg/stddev): 7958.2500/10.00execution time (avg/stddev): 599.9840/0.01
- Analysis: 10.5 still have almost 2x better performance.
I used this time galapq (32-cores) when it was empty from other load. I again compared two builds with some different setting to find out is there some setting based problem. I had 3-node cluster as earlier and two of them are masters i.e. I run sysbench to them and last one as slave.
- git@github.com:codership/mariadb-server.git
- branch: 10.6-
MDEV-24966 - build 1 : commit 930145063595e15350cf082f9d52c211c41e481b (head)
- build 2 : commit 18dc5b01927eae15be98b80717bffdfa9949732b (just before 24915)
In results I add transactions/second from both sysbench instances to one number.
| build | wsrep_sync_wait=0 | wsrep_sync_wait=15 | wsrep_sync_wait=0 && wsrep_slave_threads=16 |
|---|---|---|---|
| build1 | 256 | 257 | 373 |
| build2 | 431 | 430 | 630 |
In my hw there is regression. There is not much a effect what value is used wsrep_sync_wait but adding more slave threads have a effect.
seppo, it looks like some commits of that branch have been merged to MariaDB. PR#1780 still has unresolved review comments and is now also showing conflicts with the 10.6 branch.
- I do see BF aborts (provided outputs)
- 10.6 branch 10.6-
MDEV-24966head - ./mtr galera_3nodes.galera_wsrep_schema -
start-and-exit --mysqld=wsrep-slave-threads=16 --mysqld=-innodb-lock-wait-timeout=1 - sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/mariadb-server/mysql-test/var/tmp/mysqld.1.sock --table-size=100000 --mysql-user=root run
- sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/mariadb-server/mysql-test/var/tmp/mysqld.2.sock --table-size=100000 --mysql-user=root run
- Total hang.
jan@galapq:~$ sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/mariadb-server/mysql-test/var/tmp/mysqld.2.sock --table-size=10000 --mysql-user=root run
|
sysbench 1.0.18 (using system LuaJIT 2.1.0-beta3)
|
|
|
Running the test with following options:
|
Number of threads: 16
|
Report intermediate results every 10 second(s)
|
Initializing random number generator from current time
|
|
|
|
|
Initializing worker threads...
|
|
|
Threads started!
|
|
|
[ 10s ] thds: 16 tps: 947.11 qps: 20781.01 (r/w/o: 14653.55/3818.23/2309.23) lat (ms,95%): 21.11 err/s: 97.98 reconn/s: 0.00
|
[ 20s ] thds: 16 tps: 1349.79 qps: 27226.18 (r/w/o: 19073.45/5289.66/2863.08) lat (ms,95%): 18.95 err/s: 13.60 reconn/s: 0.00
|
[ 30s ] thds: 16 tps: 1360.22 qps: 27401.00 (r/w/o: 19191.65/5404.50/2804.85) lat (ms,95%): 18.28 err/s: 10.00 reconn/s: 0.00
|
[ 40s ] thds: 16 tps: 1179.78 qps: 23981.00 (r/w/o: 16811.55/4745.90/2423.55) lat (ms,95%): 22.69 err/s: 20.90 reconn/s: 0.00
|
[ 50s ] thds: 16 tps: 776.30 qps: 17197.51 (r/w/o: 12131.58/3400.12/1665.81) lat (ms,95%): 31.94 err/s: 90.10 reconn/s: 0.00
|
[ 60s ] thds: 16 tps: 1115.91 qps: 22486.91 (r/w/o: 15752.95/4472.04/2261.92) lat (ms,95%): 26.68 err/s: 9.20 reconn/s: 0.00
|
[ 70s ] thds: 16 tps: 1132.64 qps: 23521.88 (r/w/o: 16512.28/4677.84/2331.77) lat (ms,95%): 19.65 err/s: 47.30 reconn/s: 0.00
|
[ 80s ] thds: 16 tps: 479.42 qps: 9689.30 (r/w/o: 6790.28/1928.08/970.94) lat (ms,95%): 26.68 err/s: 5.30 reconn/s: 0.00
|
[ 90s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
|
[ 100s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
|
[ 110s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
|
[ 120s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
|
[ 130s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
|
[ 140s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
|
[ 150s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
|
[ 160s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
|
[ 170s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
|
[ 180s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
|
[ 190s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
|
[ 200s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
|
[ 210s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
|
[ 220s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00
|
- Provided error logs and query_logs, show processlist output and show engine innodb status output. Too big to add them here.
Tested again after latest review fixes and rebase. Head build 665 trx/s by calculating sysbench result from both nodes and commit 18dc5b01 1004 trx/s. I used ./mtr galera_3nodes.galera_wsrep_schema -start-and-exit --mysqld=wsrep-slave-threads=8 --mysqld=-innodb-lock-wait-timeout=1. I could not see total hangs even with 10k database size when testing head build.
jplindst, I assume that by "head build" you are referring to some version of pull request 1780. Why are you testing with innodb_lock_wait_timeout=1? My understanding is that it would work around the lack of conflict resolution. I think that you should test both revisions with a longer lock wait timeout. The default is 50 seconds.
I agree with Marko here, innodb_lock_wait_timeout=1 is precisely what should not be configured for benchmarking.
BF conflict resolving should never fall back to lock wait expiration, and better option would be to set innodb_lock_wait_timeout to very high value to force this potential problem to surface in testing.
Side note: innodb_lock_wait_timeout=1 used to mean roughly 2 seconds until MDEV-24671 replaced lock_wait_suspend_thread() and lock_wait_timeout_task with lock_wait() and condition variables. Now we would technically be able to set shorter timeout if the high-level interface were adjusted.
| Assignee | Seppo Jaakola [ seppo ] | Jan Lindström [ jplindst ] |
- builds done with: cmake . -DBUILD_CONFIG=mysql_release
- build1
- branch : 10.6-
MDEV-24966 - commit : b0a71cf6c
- branch : 10.6-
- build2
- branch : 10.5
- commit : 7f75acc05
- server start : ./mtr galera_3nodes.galera_wsrep_schema -
start-and-exit --mysqld=-wsrep-slave-threads=3 - sysbench prepare/run : sysbench /usr/share/sysbench/oltp_read_write.lua --threads=10 --report-interval=10 --time=600 --mysql-socket=/home/jan/mariadb-server/mysql-test/var/tmp/mysqld.1.sock --table-size=<table_size> --mysql-user=root run
- sysbench2 run: sysbench /usr/share/sysbench/oltp_read_write.lua --threads=10 --report-interval=10 --time=600 --mysql-socket=/home/jan/mariadb-server/mysql-test/var/tmp/mysqld.2.sock --table-size=<table_size> --mysql-user=root run
| build | table_size | trx/s |
|---|---|---|
| build1 | 1M | 377 |
| build2 | 1M | 533 |
| build1 | 10k | 1120 |
| build2 | 10k | 953 |
| Link |
This issue is blocked by |
- Machine galapq (32 cores)
- builds done with : cmake . -DBUILD_CONFIG=mysql_release
- build1 : branch 10.6-
MDEV-24966commit c5ad9890 - build2: branch 10.5 commit 7f75acc05
- server start : ./mtr galera_3nodes.galera_wsrep_schema -
start-and-exit --mysqld=-wsrep-slave-threads=8 - sysbench1 : sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/10.6/mysql-test/var/tmp/mysqld.1.sock --table-size=1000000 --mysql-user=root run
- sysbench2: sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/10.6/mysql-test/var/tmp/mysqld.2.sock --table-size=1000000 --mysql-user=root run
| buid | node1 trx/s | node1 avg lat ms | node2 trx/s | node2 avg lat ms |
|---|---|---|---|---|
| build1 | 193 | 82 | 260 | 61 |
| build1 | 193 | 82 | 252 | 63 |
| build1 | 195 | 81 | 255 | 62 |
| build1 | 197 | 81 | 263 | 60 |
| build2 | 261 | 61 | 313 | 51 |
| build2 | 261 | 61 | 320 | 49 |
| build2 | 270 | 59 | 312 | 51 |
| build2 | 261 | 61 | 318 | 50 |
| build2 | 270 | 59 | 312 | 51 |
I run one extra for both builds to get wsrep status. Here is results from build1:
jan@galapq:~/10.6/mysql-test$ /usr/local/mysql/bin/mysql -u root -S /home/jan/10.6/mysql-test/var/tmp/mysqld.1.sock
|
Welcome to the MariaDB monitor. Commands end with ; or \g.
|
Your MariaDB connection id is 49
|
Server version: 10.6.0-MariaDB-log MariaDB Server
|
|
|
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
|
|
|
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
|
|
|
MariaDB [(none)]> show status like 'wsrep%';
|
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
|
| Variable_name | Value |
|
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
|
| wsrep_local_state_uuid | accf69c8-9aca-11eb-962e-6728f851f042 |
|
| wsrep_protocol_version | 10 |
|
| wsrep_last_committed | 274237 |
|
| wsrep_replicated | 119499 |
|
| wsrep_replicated_bytes | 621841000 |
|
| wsrep_repl_keys | 1554159 |
|
| wsrep_repl_keys_bytes | 15304224 |
|
| wsrep_repl_data_bytes | 598280386 |
|
| wsrep_repl_other_bytes | 0 |
|
| wsrep_received | 160599 |
|
| wsrep_received_bytes | 299050532 |
|
| wsrep_local_commits | 119046 |
|
| wsrep_local_cert_failures | 197 |
|
| wsrep_local_replays | 5 |
|
| wsrep_local_send_queue | 0 |
|
| wsrep_local_send_queue_max | 17 |
|
| wsrep_local_send_queue_min | 0 |
|
| wsrep_local_send_queue_avg | 0.622991 |
|
| wsrep_local_recv_queue | 0 |
|
| wsrep_local_recv_queue_max | 43 |
|
| wsrep_local_recv_queue_min | 0 |
|
| wsrep_local_recv_queue_avg | 7.39473 |
|
| wsrep_local_cached_downto | 207353 |
|
| wsrep_flow_control_paused_ns | 90625348124 |
|
| wsrep_flow_control_paused | 0.0556871 |
|
| wsrep_flow_control_sent | 2608 |
|
| wsrep_flow_control_recv | 7859 |
|
| wsrep_flow_control_active | false |
|
| wsrep_flow_control_requested | false |
|
| wsrep_cert_deps_distance | 129.89 |
|
| wsrep_apply_oooe | 0.976622 |
|
| wsrep_apply_oool | 0.234074 |
|
| wsrep_apply_window | 10.6306 |
|
| wsrep_commit_oooe | 0 |
|
| wsrep_commit_oool | 0 |
|
| wsrep_commit_window | 7.2205 |
|
| wsrep_local_state | 4 |
|
| wsrep_local_state_comment | Synced |
|
| wsrep_cert_index_size | 390 |
|
| wsrep_causal_reads | 120073 |
|
| wsrep_cert_interval | 12.1642 |
|
| wsrep_open_transactions | 0 |
|
| wsrep_open_connections | 0 |
|
| wsrep_incoming_addresses | 127.0.0.1:16000,127.0.0.1:16001,127.0.0.1:16002 |
|
| wsrep_cluster_weight | 3 |
|
| wsrep_debug_sync_waiters | |
|
| wsrep_desync_count | 0 |
|
| wsrep_evs_delayed | |
|
| wsrep_evs_evict_list | |
|
| wsrep_evs_repl_latency | 0/0/0/0/0 |
|
| wsrep_evs_state | OPERATIONAL |
|
| wsrep_gcomm_uuid | acce5bd2-9aca-11eb-ac53-c78357a8c01d |
|
| wsrep_gmcast_segment | 0 |
|
| wsrep_applier_thread_count | 8 |
|
| wsrep_cluster_capabilities | |
|
| wsrep_cluster_conf_id | 3 |
|
| wsrep_cluster_size | 3 |
|
| wsrep_cluster_state_uuid | accf69c8-9aca-11eb-962e-6728f851f042 |
|
| wsrep_cluster_status | Primary |
|
| wsrep_connected | ON |
|
| wsrep_local_bf_aborts | 384 |
|
| wsrep_local_index | 0 |
|
| wsrep_provider_capabilities | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: |
|
| wsrep_provider_name | Galera |
|
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
|
| wsrep_provider_version | 26.4.6(r1d8d67c8) |
|
| wsrep_ready | ON |
|
| wsrep_rollbacker_thread_count | 1 |
|
| wsrep_thread_count | 9 |
|
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
|
69 rows in set (0.01 sec)
|
|
|
MariaDB [(none)]> exit
|
Bye
|
jan@galapq:~/10.6/mysql-test$ /usr/local/mysql/bin/mysql -u root -S /home/jan/10.6/mysql-test/var/tmp/mysqld.2.sock
|
Welcome to the MariaDB monitor. Commands end with ; or \g.
|
Your MariaDB connection id is 32
|
Server version: 10.6.0-MariaDB-log MariaDB Server
|
|
|
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
|
|
|
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
|
|
|
MariaDB [(none)]> show status like 'wsrep%';
|
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
|
| Variable_name | Value |
|
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
|
| wsrep_local_state_uuid | accf69c8-9aca-11eb-962e-6728f851f042 |
|
| wsrep_protocol_version | 10 |
|
| wsrep_last_committed | 274237 |
|
| wsrep_replicated | 154735 |
|
| wsrep_replicated_bytes | 299002824 |
|
| wsrep_repl_keys | 721155 |
|
| wsrep_repl_keys_bytes | 9482880 |
|
| wsrep_repl_data_bytes | 278832869 |
|
| wsrep_repl_other_bytes | 0 |
|
| wsrep_received | 125357 |
|
| wsrep_received_bytes | 621888328 |
|
| wsrep_local_commits | 154408 |
|
| wsrep_local_cert_failures | 173 |
|
| wsrep_local_replays | 2 |
|
| wsrep_local_send_queue | 0 |
|
| wsrep_local_send_queue_max | 17 |
|
| wsrep_local_send_queue_min | 0 |
|
| wsrep_local_send_queue_avg | 0.689901 |
|
| wsrep_local_recv_queue | 0 |
|
| wsrep_local_recv_queue_max | 42 |
|
| wsrep_local_recv_queue_min | 0 |
|
| wsrep_local_recv_queue_avg | 3.64854 |
|
| wsrep_local_cached_downto | 207353 |
|
| wsrep_flow_control_paused_ns | 90589584407 |
|
| wsrep_flow_control_paused | 0.0549812 |
|
| wsrep_flow_control_sent | 873 |
|
| wsrep_flow_control_recv | 7859 |
|
| wsrep_flow_control_active | false |
|
| wsrep_flow_control_requested | false |
|
| wsrep_cert_deps_distance | 129.89 |
|
| wsrep_apply_oooe | 0.961981 |
|
| wsrep_apply_oool | 0.242971 |
|
| wsrep_apply_window | 9.43399 |
|
| wsrep_commit_oooe | 0 |
|
| wsrep_commit_oool | 0 |
|
| wsrep_commit_window | 6.90582 |
|
| wsrep_local_state | 4 |
|
| wsrep_local_state_comment | Synced |
|
| wsrep_cert_index_size | 390 |
|
| wsrep_causal_reads | 155172 |
|
| wsrep_cert_interval | 12.1642 |
|
| wsrep_open_transactions | 0 |
|
| wsrep_open_connections | 0 |
|
| wsrep_incoming_addresses | 127.0.0.1:16000,127.0.0.1:16001,127.0.0.1:16002 |
|
| wsrep_cluster_weight | 3 |
|
| wsrep_debug_sync_waiters | |
|
| wsrep_desync_count | 0 |
|
| wsrep_evs_delayed | |
|
| wsrep_evs_evict_list | |
|
| wsrep_evs_repl_latency | 0/0/0/0/0 |
|
| wsrep_evs_state | OPERATIONAL |
|
| wsrep_gcomm_uuid | ad599a90-9aca-11eb-9fd1-7fc99b6ff104 |
|
| wsrep_gmcast_segment | 0 |
|
| wsrep_applier_thread_count | 8 |
|
| wsrep_cluster_capabilities | |
|
| wsrep_cluster_conf_id | 3 |
|
| wsrep_cluster_size | 3 |
|
| wsrep_cluster_state_uuid | accf69c8-9aca-11eb-962e-6728f851f042 |
|
| wsrep_cluster_status | Primary |
|
| wsrep_connected | ON |
|
| wsrep_local_bf_aborts | 263 |
|
| wsrep_local_index | 1 |
|
| wsrep_provider_capabilities | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: |
|
| wsrep_provider_name | Galera |
|
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
|
| wsrep_provider_version | 26.4.6(r1d8d67c8) |
|
| wsrep_ready | ON |
|
| wsrep_rollbacker_thread_count | 1 |
|
| wsrep_thread_count | 9 |
|
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
|
69 rows in set (0.00 sec)
|
And here for build2:
jan@galapq:~/10.5/mysql-test$ /usr/local/mysql/bin/mysql -u root -S /home/jan/10.5/mysql-test/var/tmp/mysqld.1.sock
|
Welcome to the MariaDB monitor. Commands end with ; or \g.
|
Your MariaDB connection id is 49
|
Server version: 10.5.10-MariaDB-log MariaDB Server
|
|
|
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
|
|
|
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
|
|
|
MariaDB [(none)]> show status like 'wsrep%';
|
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
|
| Variable_name | Value |
|
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
|
| wsrep_local_state_uuid | a8915695-9ace-11eb-89da-aae7525999e3 |
|
| wsrep_protocol_version | 10 |
|
| wsrep_last_committed | 351725 |
|
| wsrep_replicated | 164228 |
|
| wsrep_replicated_bytes | 710430584 |
|
| wsrep_repl_keys | 1765096 |
|
| wsrep_repl_keys_bytes | 18065216 |
|
| wsrep_repl_data_bytes | 681010422 |
|
| wsrep_repl_other_bytes | 0 |
|
| wsrep_received | 194595 |
|
| wsrep_received_bytes | 365158420 |
|
| wsrep_local_commits | 163552 |
|
| wsrep_local_cert_failures | 386 |
|
| wsrep_local_replays | 3 |
|
| wsrep_local_send_queue | 0 |
|
| wsrep_local_send_queue_max | 17 |
|
| wsrep_local_send_queue_min | 0 |
|
| wsrep_local_send_queue_avg | 0.19802 |
|
| wsrep_local_recv_queue | 0 |
|
| wsrep_local_recv_queue_max | 44 |
|
| wsrep_local_recv_queue_min | 0 |
|
| wsrep_local_recv_queue_avg | 4.10145 |
|
| wsrep_local_cached_downto | 285155 |
|
| wsrep_flow_control_paused_ns | 15454166764 |
|
| wsrep_flow_control_paused | 0.00297159 |
|
| wsrep_flow_control_sent | 719 |
|
| wsrep_flow_control_recv | 1746 |
|
| wsrep_flow_control_active | false |
|
| wsrep_flow_control_requested | false |
|
| wsrep_cert_deps_distance | 123.297 |
|
| wsrep_apply_oooe | 0.969959 |
|
| wsrep_apply_oool | 0.213529 |
|
| wsrep_apply_window | 9.34882 |
|
| wsrep_commit_oooe | 0 |
|
| wsrep_commit_oool | 0 |
|
| wsrep_commit_window | 6.35635 |
|
| wsrep_local_state | 4 |
|
| wsrep_local_state_comment | Synced |
|
| wsrep_cert_index_size | 308 |
|
| wsrep_causal_reads | 165077 |
|
| wsrep_cert_interval | 12.6078 |
|
| wsrep_open_transactions | 0 |
|
| wsrep_open_connections | 0 |
|
| wsrep_incoming_addresses | 127.0.0.1:16000,127.0.0.1:16001,127.0.0.1:16002 |
|
| wsrep_cluster_weight | 3 |
|
| wsrep_debug_sync_waiters | |
|
| wsrep_desync_count | 0 |
|
| wsrep_evs_delayed | |
|
| wsrep_evs_evict_list | |
|
| wsrep_evs_repl_latency | 0/0/0/0/0 |
|
| wsrep_evs_state | OPERATIONAL |
|
| wsrep_gcomm_uuid | a88f6308-9ace-11eb-a1cf-4f7ba5c17968 |
|
| wsrep_gmcast_segment | 0 |
|
| wsrep_applier_thread_count | 8 |
|
| wsrep_cluster_capabilities | |
|
| wsrep_cluster_conf_id | 3 |
|
| wsrep_cluster_size | 3 |
|
| wsrep_cluster_state_uuid | a8915695-9ace-11eb-89da-aae7525999e3 |
|
| wsrep_cluster_status | Primary |
|
| wsrep_connected | ON |
|
| wsrep_local_bf_aborts | 470 |
|
| wsrep_local_index | 0 |
|
| wsrep_provider_capabilities | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: |
|
| wsrep_provider_name | Galera |
|
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
|
| wsrep_provider_version | 26.4.6(r1d8d67c8) |
|
| wsrep_ready | ON |
|
| wsrep_rollbacker_thread_count | 1 |
|
| wsrep_thread_count | 9 |
|
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
|
69 rows in set (0.00 sec)
|
|
|
MariaDB [(none)]> exit
|
Bye
|
jan@galapq:~/10.5/mysql-test$ /usr/local/mysql/bin/mysql -u root -S /home/jan/10.5/mysql-test/var/tmp/mysqld.2.sock
|
Welcome to the MariaDB monitor. Commands end with ; or \g.
|
Your MariaDB connection id is 32
|
Server version: 10.5.10-MariaDB-log MariaDB Server
|
|
|
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
|
|
|
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
|
|
|
MariaDB [(none)]> show status like 'wsrep%';
|
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
|
| Variable_name | Value |
|
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
|
| wsrep_local_state_uuid | a8915695-9ace-11eb-89da-aae7525999e3 |
|
| wsrep_protocol_version | 10 |
|
| wsrep_last_committed | 351725 |
|
| wsrep_replicated | 187494 |
|
| wsrep_replicated_bytes | 365100816 |
|
| wsrep_repl_keys | 878318 |
|
| wsrep_repl_keys_bytes | 11526400 |
|
| wsrep_repl_data_bytes | 340613754 |
|
| wsrep_repl_other_bytes | 0 |
|
| wsrep_received | 171323 |
|
| wsrep_received_bytes | 710487808 |
|
| wsrep_local_commits | 186902 |
|
| wsrep_local_cert_failures | 379 |
|
| wsrep_local_replays | 5 |
|
| wsrep_local_send_queue | 0 |
|
| wsrep_local_send_queue_max | 17 |
|
| wsrep_local_send_queue_min | 0 |
|
| wsrep_local_send_queue_avg | 0.214928 |
|
| wsrep_local_recv_queue | 0 |
|
| wsrep_local_recv_queue_max | 43 |
|
| wsrep_local_recv_queue_min | 0 |
|
| wsrep_local_recv_queue_avg | 2.57234 |
|
| wsrep_local_cached_downto | 285155 |
|
| wsrep_flow_control_paused_ns | 15450304439 |
|
| wsrep_flow_control_paused | 0.00296741 |
|
| wsrep_flow_control_sent | 329 |
|
| wsrep_flow_control_recv | 1746 |
|
| wsrep_flow_control_active | false |
|
| wsrep_flow_control_requested | false |
|
| wsrep_cert_deps_distance | 123.297 |
|
| wsrep_apply_oooe | 0.960673 |
|
| wsrep_apply_oool | 0.21312 |
|
| wsrep_apply_window | 8.98401 |
|
| wsrep_commit_oooe | 0 |
|
| wsrep_commit_oool | 0 |
|
| wsrep_commit_window | 6.14512 |
|
| wsrep_local_state | 4 |
|
| wsrep_local_state_comment | Synced |
|
| wsrep_cert_index_size | 308 |
|
| wsrep_causal_reads | 188249 |
|
| wsrep_cert_interval | 12.6078 |
|
| wsrep_open_transactions | 0 |
|
| wsrep_open_connections | 0 |
|
| wsrep_incoming_addresses | 127.0.0.1:16000,127.0.0.1:16001,127.0.0.1:16002 |
|
| wsrep_cluster_weight | 3 |
|
| wsrep_debug_sync_waiters | |
|
| wsrep_desync_count | 0 |
|
| wsrep_evs_delayed | |
|
| wsrep_evs_evict_list | |
|
| wsrep_evs_repl_latency | 0/0/0/0/0 |
|
| wsrep_evs_state | OPERATIONAL |
|
| wsrep_gcomm_uuid | a8fef041-9ace-11eb-8e11-4bd5058b9c2d |
|
| wsrep_gmcast_segment | 0 |
|
| wsrep_applier_thread_count | 8 |
|
| wsrep_cluster_capabilities | |
|
| wsrep_cluster_conf_id | 3 |
|
| wsrep_cluster_size | 3 |
|
| wsrep_cluster_state_uuid | a8915695-9ace-11eb-89da-aae7525999e3 |
|
| wsrep_cluster_status | Primary |
|
| wsrep_connected | ON |
|
| wsrep_local_bf_aborts | 381 |
|
| wsrep_local_index | 1 |
|
| wsrep_provider_capabilities | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: |
|
| wsrep_provider_name | Galera |
|
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
|
| wsrep_provider_version | 26.4.6(r1d8d67c8) |
|
| wsrep_ready | ON |
|
| wsrep_rollbacker_thread_count | 1 |
|
| wsrep_thread_count | 9 |
|
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
|
69 rows in set (0.00 sec)
|
- Analysis:
- Both performance metrics i.e. transactions in second and average latency seem to be quite similar between runs.
- In build1 average transactions in second was 358 and with build2 it was 581. This means that build2 was 223 transactions/second or 62% faster than build1 in average.
- In build1 average latency was 71ms and with build2 it was 55ms. This means that average latency in build2 was 16ms smaller or 22% smaller.
- Local BF aborts in build1 was 384 in node1 and 263 in node2 while with build2 it was 470 in node1 and 381 in node2. Thus, build2 had more BF aborts.
- Flow control paused in build2 was 0.0557 in node1 and 0.0550 in node2 while with build2 it was 0.00297 in node1 and 0.00297. Thus, build2 had less flow control pauses.
- Questions
- Is this still significant multi-master performance regression?
- Is oltp_read_write good load for multi-master performance testing?
| Status | In Review [ 10002 ] | Stalled [ 10000 ] |
| Assignee | Jan Lindström [ jplindst ] | Seppo Jaakola [ seppo ] |
The total number of BF lock conflicts is quite low here, it probably does not affect to the transaction throughput rates recorded. Unless, there are situations where conflict resolution takes very long time, e.g. due to lock wait timeout expiration. But, I assume innodb_lock_wait_timeout is default 50 secs here, and with that you would observe hang with sysbench 10 sec report interval.
Flow control rates are much higher in build1 execution, and reason for this would be good to find out. A curious observation here is that majority of 'flow control send' signals appear to come from 'node 3', which is the third read only node (7859 FC signals in total, node1: 2608 node2: 873, node3: 4378)
As node3 is in read only role in this test, there should happen no conflict resolution at all. Still this node node is the slowest to apply incoming replication events.
I suggest following further testing:
- running this test against 1 node cluster. i.e. just one node in the cluster. If applying is the bottleneck, I would expect similar transaction throughput rates with this test.
- running the test against two node cluster, but using just one sysbench directed to first node in the cluster. The second node would probably exhaust under the replication load and starts sending flow control stop signals
If the second test shows significantly more flow control waiting in 10.6 cluster, then profiling the reason for this anomaly should be troubleshooted.
Results from single master cluster. See setups from my last above comment. I shut down the nodes 2 and 3 before executing sysbench.
Build1 had 556trx/s and average latency of 28ms while build2 had 605 trx/s and average latency of 26ms. I would say there is similar performance here.
Results from 2 node cluster same setup as my comment above (i.e. 8 slave threads). I shut down the node 3 before executing single sysbench.
- build1:
- Performance: 498 trx/s with 32ms average latency
- Status node1
jan@galapq:~/10.6/mysql-test$ /usr/local/mysql/bin/mysql -u root -S /home/jan/10.6/mysql-test/var/tmp/mysqld.1.sockWelcome to the MariaDB monitor. Commands end with ; or \g.Your MariaDB connection id is 49Server version: 10.6.0-MariaDB-log MariaDB ServerCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]> show status like 'wsrep%';+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| Variable_name | Value |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| wsrep_local_state_uuid | 7ce5d143-9b6d-11eb-940a-6a124c5956f6 || wsrep_protocol_version | 10 || wsrep_last_committed | 299653 || wsrep_replicated | 299649 || wsrep_replicated_bytes | 992392744 || wsrep_repl_keys | 2425631 || wsrep_repl_keys_bytes | 26599600 || wsrep_repl_data_bytes | 945026739 || wsrep_repl_other_bytes | 0 || wsrep_received | 11 || wsrep_received_bytes | 1084 || wsrep_local_commits | 299646 || wsrep_local_cert_failures | 0 || wsrep_local_replays | 0 || wsrep_local_send_queue | 0 || wsrep_local_send_queue_max | 17 || wsrep_local_send_queue_min | 0 || wsrep_local_send_queue_avg | 0.202426 || wsrep_local_recv_queue | 0 || wsrep_local_recv_queue_max | 2 || wsrep_local_recv_queue_min | 0 || wsrep_local_recv_queue_avg | 0.0909091 || wsrep_local_cached_downto | 234559 || wsrep_flow_control_paused_ns | 22206378056 || wsrep_flow_control_paused | 0.029729 || wsrep_flow_control_sent | 0 || wsrep_flow_control_recv | 2694 || wsrep_flow_control_active | false || wsrep_flow_control_requested | false || wsrep_cert_deps_distance | 1831.98 || wsrep_apply_oooe | 0.860614 || wsrep_apply_oool | 0.169488 || wsrep_apply_window | 4.3013 || wsrep_commit_oooe | 0 || wsrep_commit_oool | 0 || wsrep_commit_window | 2.59944 || wsrep_local_state | 4 || wsrep_local_state_comment | Synced || wsrep_cert_index_size | 16035 || wsrep_causal_reads | 299649 || wsrep_cert_interval | 3.61433 || wsrep_open_transactions | 0 || wsrep_open_connections | 0 || wsrep_incoming_addresses | 127.0.0.1:16000,127.0.0.1:16001 || wsrep_cluster_weight | 2 || wsrep_debug_sync_waiters | || wsrep_desync_count | 0 || wsrep_evs_delayed | || wsrep_evs_evict_list | || wsrep_evs_repl_latency | 0.000100169/0.000216768/0.000726967/5.39345e-05/1875 || wsrep_evs_state | OPERATIONAL || wsrep_gcomm_uuid | 7ce4bef8-9b6d-11eb-8669-62fa6683d35b || wsrep_gmcast_segment | 0 || wsrep_applier_thread_count | 8 || wsrep_cluster_capabilities | || wsrep_cluster_conf_id | 4 || wsrep_cluster_size | 2 || wsrep_cluster_state_uuid | 7ce5d143-9b6d-11eb-940a-6a124c5956f6 || wsrep_cluster_status | Primary || wsrep_connected | ON || wsrep_local_bf_aborts | 0 || wsrep_local_index | 0 || wsrep_provider_capabilities | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: || wsrep_provider_name | Galera || wsrep_provider_vendor | Codership Oy <info@codership.com> || wsrep_provider_version | 26.4.6(r1d8d67c8) || wsrep_ready | ON || wsrep_rollbacker_thread_count | 1 || wsrep_thread_count | 9 |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+69 rows in set (0.00 sec) - status node2
jan@galapq:~/10.6/mysql-test$ /usr/local/mysql/bin/mysql -u root -S /home/jan/10.6/mysql-test/var/tmp/mysqld.2.sockWelcome to the MariaDB monitor. Commands end with ; or \g.Your MariaDB connection id is 16Server version: 10.6.0-MariaDB-log MariaDB ServerCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]> show status like 'wsrep%';+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| Variable_name | Value |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| wsrep_local_state_uuid | 7ce5d143-9b6d-11eb-940a-6a124c5956f6 || wsrep_protocol_version | 10 || wsrep_last_committed | 299653 || wsrep_replicated | 0 || wsrep_replicated_bytes | 0 || wsrep_repl_keys | 0 || wsrep_repl_keys_bytes | 0 || wsrep_repl_data_bytes | 0 || wsrep_repl_other_bytes | 0 || wsrep_received | 299654 || wsrep_received_bytes | 992393448 || wsrep_local_commits | 0 || wsrep_local_cert_failures | 0 || wsrep_local_replays | 0 || wsrep_local_send_queue | 0 || wsrep_local_send_queue_max | 1 || wsrep_local_send_queue_min | 0 || wsrep_local_send_queue_avg | 0 || wsrep_local_recv_queue | 0 || wsrep_local_recv_queue_max | 36 || wsrep_local_recv_queue_min | 0 || wsrep_local_recv_queue_avg | 3.83362 || wsrep_local_cached_downto | 234559 || wsrep_flow_control_paused_ns | 22082368614 || wsrep_flow_control_paused | 0.0292511 || wsrep_flow_control_sent | 2694 || wsrep_flow_control_recv | 2694 || wsrep_flow_control_active | false || wsrep_flow_control_requested | false || wsrep_cert_deps_distance | 1831.98 || wsrep_apply_oooe | 0.959883 || wsrep_apply_oool | 0.208861 || wsrep_apply_window | 6.76603 || wsrep_commit_oooe | 0 || wsrep_commit_oool | 0 || wsrep_commit_window | 3.30589 || wsrep_local_state | 4 || wsrep_local_state_comment | Synced || wsrep_cert_index_size | 16035 || wsrep_causal_reads | 2 || wsrep_cert_interval | 3.61433 || wsrep_open_transactions | 0 || wsrep_open_connections | 0 || wsrep_incoming_addresses | 127.0.0.1:16000,127.0.0.1:16001 || wsrep_cluster_weight | 2 || wsrep_debug_sync_waiters | || wsrep_desync_count | 0 || wsrep_evs_delayed | || wsrep_evs_evict_list | || wsrep_evs_repl_latency | 0.000166258/0.000240889/0.000523022/8.75949e-05/12 || wsrep_evs_state | OPERATIONAL || wsrep_gcomm_uuid | 7d582470-9b6d-11eb-ac4d-620ddf7128a1 || wsrep_gmcast_segment | 0 || wsrep_applier_thread_count | 8 || wsrep_cluster_capabilities | || wsrep_cluster_conf_id | 4 || wsrep_cluster_size | 2 || wsrep_cluster_state_uuid | 7ce5d143-9b6d-11eb-940a-6a124c5956f6 || wsrep_cluster_status | Primary || wsrep_connected | ON || wsrep_local_bf_aborts | 0 || wsrep_local_index | 1 || wsrep_provider_capabilities | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: || wsrep_provider_name | Galera || wsrep_provider_vendor | Codership Oy <info@codership.com> || wsrep_provider_version | 26.4.6(r1d8d67c8) || wsrep_ready | ON || wsrep_rollbacker_thread_count | 1 || wsrep_thread_count | 9 |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+69 rows in set (0.00 sec)
- build2:
- Performance: 561 trx/s with 28ms average latency
- status node1
jan@galapq:~/10.5/mysql-test$ /usr/local/mysql/bin/mysql -u root -S /home/jan/10.5/mysql-test/var/tmp/mysqld.1.sockWelcome to the MariaDB monitor. Commands end with ; or \g.Your MariaDB connection id is 64Server version: 10.5.10-MariaDB-log MariaDB ServerCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]> show status like 'wsrep%';+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| Variable_name | Value |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| wsrep_local_state_uuid | 75fc6de2-9b6f-11eb-bb56-df8dbe17f5db || wsrep_protocol_version | 10 || wsrep_last_committed | 337463 || wsrep_replicated | 337458 || wsrep_replicated_bytes | 1069493912 || wsrep_repl_keys | 2607294 || wsrep_repl_keys_bytes | 28960320 || wsrep_repl_data_bytes | 1017144734 || wsrep_repl_other_bytes | 0 || wsrep_received | 12 || wsrep_received_bytes | 1548 || wsrep_local_commits | 337456 || wsrep_local_cert_failures | 0 || wsrep_local_replays | 0 || wsrep_local_send_queue | 0 || wsrep_local_send_queue_max | 17 || wsrep_local_send_queue_min | 0 || wsrep_local_send_queue_avg | 0.108555 || wsrep_local_recv_queue | 0 || wsrep_local_recv_queue_max | 2 || wsrep_local_recv_queue_min | 0 || wsrep_local_recv_queue_avg | 0.0833333 || wsrep_local_cached_downto | 272474 || wsrep_flow_control_paused_ns | 3414753469 || wsrep_flow_control_paused | 0.00398547 || wsrep_flow_control_sent | 0 || wsrep_flow_control_recv | 387 || wsrep_flow_control_active | false || wsrep_flow_control_requested | false || wsrep_cert_deps_distance | 1827.39 || wsrep_apply_oooe | 0.849718 || wsrep_apply_oool | 0.19464 || wsrep_apply_window | 4.24396 || wsrep_commit_oooe | 0 || wsrep_commit_oool | 0 || wsrep_commit_window | 1.95954 || wsrep_local_state | 4 || wsrep_local_state_comment | Synced || wsrep_cert_index_size | 16072 || wsrep_causal_reads | 337458 || wsrep_cert_interval | 3.56067 || wsrep_open_transactions | 0 || wsrep_open_connections | 0 || wsrep_incoming_addresses | 127.0.0.1:16000,127.0.0.1:16001 || wsrep_cluster_weight | 2 || wsrep_debug_sync_waiters | || wsrep_desync_count | 0 || wsrep_evs_delayed | || wsrep_evs_evict_list | || wsrep_evs_repl_latency | 0/0/0/0/0 || wsrep_evs_state | OPERATIONAL || wsrep_gcomm_uuid | 75fbcb96-9b6f-11eb-b7de-eb5d1c823c34 || wsrep_gmcast_segment | 0 || wsrep_applier_thread_count | 8 || wsrep_cluster_capabilities | || wsrep_cluster_conf_id | 4 || wsrep_cluster_size | 2 || wsrep_cluster_state_uuid | 75fc6de2-9b6f-11eb-bb56-df8dbe17f5db || wsrep_cluster_status | Primary || wsrep_connected | ON || wsrep_local_bf_aborts | 0 || wsrep_local_index | 0 || wsrep_provider_capabilities | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: || wsrep_provider_name | Galera || wsrep_provider_vendor | Codership Oy <info@codership.com> || wsrep_provider_version | 26.4.6(r1d8d67c8) || wsrep_ready | ON || wsrep_rollbacker_thread_count | 1 || wsrep_thread_count | 9 |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+69 rows in set (0.00 sec) - status node2
jan@galapq:~/10.5/mysql-test$ /usr/local/mysql/bin/mysql -u root -S /home/jan/10.5/mysql-test/var/tmp/mysqld.2.sockWelcome to the MariaDB monitor. Commands end with ; or \g.Your MariaDB connection id is 17Server version: 10.5.10-MariaDB-log MariaDB ServerCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]> show status like 'wsrep%';+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| Variable_name | Value |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| wsrep_local_state_uuid | 75fc6de2-9b6f-11eb-bb56-df8dbe17f5db || wsrep_protocol_version | 10 || wsrep_last_committed | 337463 || wsrep_replicated | 1 || wsrep_replicated_bytes | 464 || wsrep_repl_keys | 1 || wsrep_repl_keys_bytes | 32 || wsrep_repl_data_bytes | 361 || wsrep_repl_other_bytes | 0 || wsrep_received | 337463 || wsrep_received_bytes | 1069494616 || wsrep_local_commits | 0 || wsrep_local_cert_failures | 0 || wsrep_local_replays | 0 || wsrep_local_send_queue | 0 || wsrep_local_send_queue_max | 1 || wsrep_local_send_queue_min | 0 || wsrep_local_send_queue_avg | 0 || wsrep_local_recv_queue | 0 || wsrep_local_recv_queue_max | 32 || wsrep_local_recv_queue_min | 0 || wsrep_local_recv_queue_avg | 1.46815 || wsrep_local_cached_downto | 272474 || wsrep_flow_control_paused_ns | 3397453977 || wsrep_flow_control_paused | 0.00393239 || wsrep_flow_control_sent | 387 || wsrep_flow_control_recv | 387 || wsrep_flow_control_active | false || wsrep_flow_control_requested | false || wsrep_cert_deps_distance | 1827.39 || wsrep_apply_oooe | 0.881941 || wsrep_apply_oool | 0.192204 || wsrep_apply_window | 4.66429 || wsrep_commit_oooe | 0 || wsrep_commit_oool | 0 || wsrep_commit_window | 2.45095 || wsrep_local_state | 4 || wsrep_local_state_comment | Synced || wsrep_cert_index_size | 16072 || wsrep_causal_reads | 3 || wsrep_cert_interval | 3.56067 || wsrep_open_transactions | 0 || wsrep_open_connections | 0 || wsrep_incoming_addresses | 127.0.0.1:16000,127.0.0.1:16001 || wsrep_cluster_weight | 2 || wsrep_debug_sync_waiters | || wsrep_desync_count | 0 || wsrep_evs_delayed | || wsrep_evs_evict_list | || wsrep_evs_repl_latency | 0/0/0/0/0 || wsrep_evs_state | OPERATIONAL || wsrep_gcomm_uuid | 76604c70-9b6f-11eb-abbe-a3da79cefbfa || wsrep_gmcast_segment | 0 || wsrep_applier_thread_count | 8 || wsrep_cluster_capabilities | || wsrep_cluster_conf_id | 4 || wsrep_cluster_size | 2 || wsrep_cluster_state_uuid | 75fc6de2-9b6f-11eb-bb56-df8dbe17f5db || wsrep_cluster_status | Primary || wsrep_connected | ON || wsrep_local_bf_aborts | 0 || wsrep_local_index | 1 || wsrep_provider_capabilities | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: || wsrep_provider_name | Galera || wsrep_provider_vendor | Codership Oy <info@codership.com> || wsrep_provider_version | 26.4.6(r1d8d67c8) || wsrep_ready | ON || wsrep_rollbacker_thread_count | 1 || wsrep_thread_count | 9 |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+69 rows in set (0.01 sec)
- Analysis: 10.5 seems to have somewhat smaller average latency and transactions/s difference seems similar to single node case.
Results from 2 node cluster same setup as my comment above except using wsrep_slave_threads=1 (Note that it seems 2 is real minimum). I shut down the node 3 before executing single sysbench.
- build1
- Performance : 286 transactions/s with average latency 55ms
- status node1
jan@galapq:~/10.6/mysql-test$ /usr/local/mysql/bin/mysql -u root -S /home/jan/10.6/mysql-test/var/tmp/mysqld.1.sockWelcome to the MariaDB monitor. Commands end with ; or \g.Your MariaDB connection id is 42Server version: 10.6.0-MariaDB-log MariaDB ServerCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]> show status like 'wsrep%';+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| Variable_name | Value |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| wsrep_local_state_uuid | e3916b2b-9b72-11eb-9943-cfc3fcebb9a8 || wsrep_protocol_version | 10 || wsrep_last_committed | 172062 || wsrep_replicated | 172058 || wsrep_replicated_bytes | 732912368 || wsrep_repl_keys | 1812994 || wsrep_repl_keys_bytes | 18636320 || wsrep_repl_data_bytes | 702363364 || wsrep_repl_other_bytes | 0 || wsrep_received | 11 || wsrep_received_bytes | 1084 || wsrep_local_commits | 172055 || wsrep_local_cert_failures | 0 || wsrep_local_replays | 0 || wsrep_local_send_queue | 0 || wsrep_local_send_queue_max | 17 || wsrep_local_send_queue_min | 0 || wsrep_local_send_queue_avg | 10.712 || wsrep_local_recv_queue | 0 || wsrep_local_recv_queue_max | 2 || wsrep_local_recv_queue_min | 0 || wsrep_local_recv_queue_avg | 0.0909091 || wsrep_local_cached_downto | 106470 || wsrep_flow_control_paused_ns | 578413407575 || wsrep_flow_control_paused | 0.769999 || wsrep_flow_control_sent | 0 || wsrep_flow_control_recv | 30297 || wsrep_flow_control_active | false || wsrep_flow_control_requested | false || wsrep_cert_deps_distance | 1836.06 || wsrep_apply_oooe | 0.821653 || wsrep_apply_oool | 0.20861 || wsrep_apply_window | 3.50828 || wsrep_commit_oooe | 0 || wsrep_commit_oool | 0 || wsrep_commit_window | 2.25736 || wsrep_local_state | 4 || wsrep_local_state_comment | Synced || wsrep_cert_index_size | 15711 || wsrep_causal_reads | 172058 || wsrep_cert_interval | 11.4219 || wsrep_open_transactions | 0 || wsrep_open_connections | 0 || wsrep_incoming_addresses | 127.0.0.1:16000,127.0.0.1:16001 || wsrep_cluster_weight | 2 || wsrep_debug_sync_waiters | || wsrep_desync_count | 0 || wsrep_evs_delayed | || wsrep_evs_evict_list | || wsrep_evs_repl_latency | 8.8969e-05/0.000156964/0.000352985/4.07357e-05/370 || wsrep_evs_state | OPERATIONAL || wsrep_gcomm_uuid | e390cea5-9b72-11eb-b28e-fe1aaf12968e || wsrep_gmcast_segment | 0 || wsrep_applier_thread_count | 1 || wsrep_cluster_capabilities | || wsrep_cluster_conf_id | 4 || wsrep_cluster_size | 2 || wsrep_cluster_state_uuid | e3916b2b-9b72-11eb-9943-cfc3fcebb9a8 || wsrep_cluster_status | Primary || wsrep_connected | ON || wsrep_local_bf_aborts | 0 || wsrep_local_index | 0 || wsrep_provider_capabilities | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: || wsrep_provider_name | Galera || wsrep_provider_vendor | Codership Oy <info@codership.com> || wsrep_provider_version | 26.4.6(r1d8d67c8) || wsrep_ready | ON || wsrep_rollbacker_thread_count | 1 || wsrep_thread_count | 2 |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+69 rows in set (0.00 sec) - status node2
jan@galapq:~/10.6/mysql-test$ /usr/local/mysql/bin/mysql -u root -S /home/jan/10.6/mysql-test/var/tmp/mysqld.2.sockWelcome to the MariaDB monitor. Commands end with ; or \g.Your MariaDB connection id is 9Server version: 10.6.0-MariaDB-log MariaDB ServerCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]> show status like 'wsrep%';+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| Variable_name | Value |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| wsrep_local_state_uuid | e3916b2b-9b72-11eb-9943-cfc3fcebb9a8 || wsrep_protocol_version | 10 || wsrep_last_committed | 172062 || wsrep_replicated | 0 || wsrep_replicated_bytes | 0 || wsrep_repl_keys | 0 || wsrep_repl_keys_bytes | 0 || wsrep_repl_data_bytes | 0 || wsrep_repl_other_bytes | 0 || wsrep_received | 172063 || wsrep_received_bytes | 732913072 || wsrep_local_commits | 0 || wsrep_local_cert_failures | 0 || wsrep_local_replays | 0 || wsrep_local_send_queue | 0 || wsrep_local_send_queue_max | 1 || wsrep_local_send_queue_min | 0 || wsrep_local_send_queue_avg | 0 || wsrep_local_recv_queue | 0 || wsrep_local_recv_queue_max | 39 || wsrep_local_recv_queue_min | 0 || wsrep_local_recv_queue_avg | 25.4044 || wsrep_local_cached_downto | 106469 || wsrep_flow_control_paused_ns | 575674914625 || wsrep_flow_control_paused | 0.760511 || wsrep_flow_control_sent | 30297 || wsrep_flow_control_recv | 30297 || wsrep_flow_control_active | false || wsrep_flow_control_requested | false || wsrep_cert_deps_distance | 1836.06 || wsrep_apply_oooe | 0 || wsrep_apply_oool | 0 || wsrep_apply_window | 1 || wsrep_commit_oooe | 0 || wsrep_commit_oool | 0 || wsrep_commit_window | 1 || wsrep_local_state | 4 || wsrep_local_state_comment | Synced || wsrep_cert_index_size | 15711 || wsrep_causal_reads | 2 || wsrep_cert_interval | 11.4219 || wsrep_open_transactions | 0 || wsrep_open_connections | 0 || wsrep_incoming_addresses | 127.0.0.1:16000,127.0.0.1:16001 || wsrep_cluster_weight | 2 || wsrep_debug_sync_waiters | || wsrep_desync_count | 0 || wsrep_evs_delayed | || wsrep_evs_evict_list | || wsrep_evs_repl_latency | 0.000112886/0.000210886/0.00030977/5.14985e-05/54 || wsrep_evs_state | OPERATIONAL || wsrep_gcomm_uuid | e417e757-9b72-11eb-a504-6a88be729c0d || wsrep_gmcast_segment | 0 || wsrep_applier_thread_count | 1 || wsrep_cluster_capabilities | || wsrep_cluster_conf_id | 4 || wsrep_cluster_size | 2 || wsrep_cluster_state_uuid | e3916b2b-9b72-11eb-9943-cfc3fcebb9a8 || wsrep_cluster_status | Primary || wsrep_connected | ON || wsrep_local_bf_aborts | 0 || wsrep_local_index | 1 || wsrep_provider_capabilities | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: || wsrep_provider_name | Galera || wsrep_provider_vendor | Codership Oy <info@codership.com> || wsrep_provider_version | 26.4.6(r1d8d67c8) || wsrep_ready | ON || wsrep_rollbacker_thread_count | 1 || wsrep_thread_count | 2 |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
- build2
- Performance: 462 transactions/s with average latency 34ms
- status node1
jan@galapq:~/10.5/mysql-test$ /usr/local/mysql/bin/mysql -u root -S /home/jan/10.5/mysql-test/var/tmp/mysqld.1.sockWelcome to the MariaDB monitor. Commands end with ; or \g.Your MariaDB connection id is 42Server version: 10.5.10-MariaDB-log MariaDB ServerCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]> show status like 'wsrep%';+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| Variable_name | Value |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| wsrep_local_state_uuid | d2168114-9b74-11eb-ba1a-ea6d858f6fcc || wsrep_protocol_version | 10 || wsrep_last_committed | 277968 || wsrep_replicated | 277964 || wsrep_replicated_bytes | 948291960 || wsrep_repl_keys | 2321370 || wsrep_repl_keys_bytes | 25245072 || wsrep_repl_data_bytes | 903785859 || wsrep_repl_other_bytes | 0 || wsrep_received | 11 || wsrep_received_bytes | 1084 || wsrep_local_commits | 277961 || wsrep_local_cert_failures | 0 || wsrep_local_replays | 0 || wsrep_local_send_queue | 0 || wsrep_local_send_queue_max | 17 || wsrep_local_send_queue_min | 0 || wsrep_local_send_queue_avg | 7.31723 || wsrep_local_recv_queue | 0 || wsrep_local_recv_queue_max | 2 || wsrep_local_recv_queue_min | 0 || wsrep_local_recv_queue_avg | 0.0909091 || wsrep_local_cached_downto | 212797 || wsrep_flow_control_paused_ns | 537219218791 || wsrep_flow_control_paused | 0.701206 || wsrep_flow_control_sent | 0 || wsrep_flow_control_recv | 49948 || wsrep_flow_control_active | false || wsrep_flow_control_requested | false || wsrep_cert_deps_distance | 1827.97 || wsrep_apply_oooe | 0.826758 || wsrep_apply_oool | 0.224166 || wsrep_apply_window | 3.67321 || wsrep_commit_oooe | 0 || wsrep_commit_oool | 0 || wsrep_commit_window | 1.90864 || wsrep_local_state | 4 || wsrep_local_state_comment | Synced || wsrep_cert_index_size | 15991 || wsrep_causal_reads | 277964 || wsrep_cert_interval | 9.11445 || wsrep_open_transactions | 0 || wsrep_open_connections | 0 || wsrep_incoming_addresses | 127.0.0.1:16000,127.0.0.1:16001 || wsrep_cluster_weight | 2 || wsrep_debug_sync_waiters | || wsrep_desync_count | 0 || wsrep_evs_delayed | || wsrep_evs_evict_list | || wsrep_evs_repl_latency | 0/0/0/0/0 || wsrep_evs_state | OPERATIONAL || wsrep_gcomm_uuid | d215e1e8-9b74-11eb-bfdf-7ef29766db9e || wsrep_gmcast_segment | 0 || wsrep_applier_thread_count | 1 || wsrep_cluster_capabilities | || wsrep_cluster_conf_id | 4 || wsrep_cluster_size | 2 || wsrep_cluster_state_uuid | d2168114-9b74-11eb-ba1a-ea6d858f6fcc || wsrep_cluster_status | Primary || wsrep_connected | ON || wsrep_local_bf_aborts | 0 || wsrep_local_index | 0 || wsrep_provider_capabilities | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: || wsrep_provider_name | Galera || wsrep_provider_vendor | Codership Oy <info@codership.com> || wsrep_provider_version | 26.4.6(r1d8d67c8) || wsrep_ready | ON || wsrep_rollbacker_thread_count | 1 || wsrep_thread_count | 2 |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+69 rows in set (0.00 sec) - status node2
jan@galapq:~/10.5/mysql-test$ /usr/local/mysql/bin/mysql -u root -S /home/jan/10.5/mysql-test/var/tmp/mysqld.2.sockWelcome to the MariaDB monitor. Commands end with ; or \g.Your MariaDB connection id is 9Server version: 10.5.10-MariaDB-log MariaDB ServerCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]> show status like 'wsrep%';+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| Variable_name | Value |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| wsrep_local_state_uuid | d2168114-9b74-11eb-ba1a-ea6d858f6fcc || wsrep_protocol_version | 10 || wsrep_last_committed | 277968 || wsrep_replicated | 0 || wsrep_replicated_bytes | 0 || wsrep_repl_keys | 0 || wsrep_repl_keys_bytes | 0 || wsrep_repl_data_bytes | 0 || wsrep_repl_other_bytes | 0 || wsrep_received | 277969 || wsrep_received_bytes | 948292664 || wsrep_local_commits | 0 || wsrep_local_cert_failures | 0 || wsrep_local_replays | 0 || wsrep_local_send_queue | 0 || wsrep_local_send_queue_max | 1 || wsrep_local_send_queue_min | 0 || wsrep_local_send_queue_avg | 0 || wsrep_local_recv_queue | 0 || wsrep_local_recv_queue_max | 39 || wsrep_local_recv_queue_min | 0 || wsrep_local_recv_queue_avg | 25.0316 || wsrep_local_cached_downto | 212797 || wsrep_flow_control_paused_ns | 533158161910 || wsrep_flow_control_paused | 0.691526 || wsrep_flow_control_sent | 49948 || wsrep_flow_control_recv | 49948 || wsrep_flow_control_active | false || wsrep_flow_control_requested | false || wsrep_cert_deps_distance | 1827.97 || wsrep_apply_oooe | 0 || wsrep_apply_oool | 0 || wsrep_apply_window | 1 || wsrep_commit_oooe | 0 || wsrep_commit_oool | 0 || wsrep_commit_window | 1 || wsrep_local_state | 4 || wsrep_local_state_comment | Synced || wsrep_cert_index_size | 15991 || wsrep_causal_reads | 2 || wsrep_cert_interval | 9.11445 || wsrep_open_transactions | 0 || wsrep_open_connections | 0 || wsrep_incoming_addresses | 127.0.0.1:16000,127.0.0.1:16001 || wsrep_cluster_weight | 2 || wsrep_debug_sync_waiters | || wsrep_desync_count | 0 || wsrep_evs_delayed | || wsrep_evs_evict_list | || wsrep_evs_repl_latency | 0/0/0/0/0 || wsrep_evs_state | OPERATIONAL || wsrep_gcomm_uuid | d2787969-9b74-11eb-b7e2-aacc735cd03e || wsrep_gmcast_segment | 0 || wsrep_applier_thread_count | 1 || wsrep_cluster_capabilities | || wsrep_cluster_conf_id | 4 || wsrep_cluster_size | 2 || wsrep_cluster_state_uuid | d2168114-9b74-11eb-ba1a-ea6d858f6fcc || wsrep_cluster_status | Primary || wsrep_connected | ON || wsrep_local_bf_aborts | 0 || wsrep_local_index | 1 || wsrep_provider_capabilities | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: || wsrep_provider_name | Galera || wsrep_provider_vendor | Codership Oy <info@codership.com> || wsrep_provider_version | 26.4.6(r1d8d67c8) || wsrep_ready | ON || wsrep_rollbacker_thread_count | 1 || wsrep_thread_count | 2 |+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+69 rows in set (0.00 sec)
- Analysis: Here we see clear performance regression in 10.6 in both transactions/s and average latency.
My performance test with a two node cluster, where sysbench load is targeted to first node only, and configuring only one replication applier thread, gives following transaction throughput rates:
- 10.6-
MDEV-24966: 318.56 per sec - 10.5: 287.54 per sec
so, my figures are in par with Jan's observations for 10.6-MDEV-24966 build, but my 10.5 build shows a lot lower throughput.
I tested normal master-slave async replication using row based replication with slave_parallel_threads=16 using same sysbench setting as previously where load is targeted to master only.
- 10.6-
MDEV-24966: 595 per sec with avg latency 26ms - 10.5 : 577 per sec with avg latency 27 ms
- I do not know how to finish performance test only when slave node has applied all replication events (this same applies to Galera at least when wsrep_sync_wait=0 i.e. default)
- So at least on my tests I do not see issues on async replication
Read-only regression between 10.5 and 10.6 is caused by https://jira.mariadb.org/browse/MDEV-25404 and similar regression exist on read-write loads also as noted. Pull request solves multi-master hang and other clear issues.
| issue.field.resolutiondate | 2021-04-14 05:02:44.0 | 2021-04-14 05:02:44.445 |
| Fix Version/s | 10.6.0 [ 24431 ] | |
| Fix Version/s | 10.6 [ 24028 ] | |
| Resolution | Fixed [ 1 ] | |
| Status | Stalled [ 10000 ] | Closed [ 6 ] |
| Workflow | MariaDB v3 [ 119459 ] | MariaDB v4 [ 158948 ] |
I tested with 3 node cluster using galapq so that I run sysbench oltp_read_write load. Exact numbers are not interesting here as I used default settings. Instead we are only interested clear regressions. I tested two builds
MDEV-24915)When I run sysbench only to node_1 and nodes 2 and 3 as slave i.e. single-master I received following results:
jan@galapq:~/10.6-bug/mysql-test$ sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/10.6-bug/mysql-test/var/tmp/mysqld.1.sock --table-size=1000000 --mysql-user=root runsysbench 1.0.18 (using system LuaJIT 2.1.0-beta3)Running the test with following options:Number of threads: 16Report intermediate results every 10 second(s)Initializing random number generator from current timeInitializing worker threads...Threads started![ 10s ] thds: 16 tps: 251.54 qps: 5050.48 (r/w/o: 3536.75/546.57/967.17) lat (ms,95%): 102.97 err/s: 0.00 reconn/s: 0.00[ 20s ] thds: 16 tps: 307.91 qps: 6168.88 (r/w/o: 4317.93/772.32/1078.63) lat (ms,95%): 80.03 err/s: 0.00 reconn/s: 0.00[ 30s ] thds: 16 tps: 301.81 qps: 6036.11 (r/w/o: 4225.28/836.12/974.72) lat (ms,95%): 73.13 err/s: 0.00 reconn/s: 0.00[ 40s ] thds: 16 tps: 273.00 qps: 5452.92 (r/w/o: 3817.71/805.00/830.20) lat (ms,95%): 92.42 err/s: 0.00 reconn/s: 0.00[ 50s ] thds: 16 tps: 165.80 qps: 3323.11 (r/w/o: 2325.51/514.10/483.50) lat (ms,95%): 92.42 err/s: 0.00 reconn/s: 0.00[ 60s ] thds: 16 tps: 240.10 qps: 4789.90 (r/w/o: 3352.90/753.30/683.70) lat (ms,95%): 99.33 err/s: 0.00 reconn/s: 0.00[ 70s ] thds: 16 tps: 311.30 qps: 6237.83 (r/w/o: 4366.65/1003.89/867.29) lat (ms,95%): 77.19 err/s: 0.00 reconn/s: 0.00[ 80s ] thds: 16 tps: 283.50 qps: 5669.09 (r/w/o: 3968.89/936.40/763.80) lat (ms,95%): 78.60 err/s: 0.00 reconn/s: 0.00[ 90s ] thds: 16 tps: 287.60 qps: 5752.22 (r/w/o: 4026.42/969.90/755.90) lat (ms,95%): 80.03 err/s: 0.00 reconn/s: 0.00[ 100s ] thds: 16 tps: 287.20 qps: 5744.95 (r/w/o: 4020.93/983.11/740.91) lat (ms,95%): 82.96 err/s: 0.00 reconn/s: 0.00[ 110s ] thds: 16 tps: 273.10 qps: 5458.74 (r/w/o: 3822.46/932.29/703.99) lat (ms,95%): 86.00 err/s: 0.00 reconn/s: 0.00[ 120s ] thds: 16 tps: 271.90 qps: 5441.21 (r/w/o: 3807.51/945.70/688.00) lat (ms,95%): 84.47 err/s: 0.00 reconn/s: 0.00[ 130s ] thds: 16 tps: 265.10 qps: 5301.90 (r/w/o: 3711.40/922.50/668.00) lat (ms,95%): 90.78 err/s: 0.00 reconn/s: 0.00[ 140s ] thds: 16 tps: 269.40 qps: 5378.41 (r/w/o: 3765.50/937.20/675.70) lat (ms,95%): 86.00 err/s: 0.00 reconn/s: 0.00[ 150s ] thds: 16 tps: 271.60 qps: 5440.88 (r/w/o: 3808.49/955.90/676.50) lat (ms,95%): 89.16 err/s: 0.00 reconn/s: 0.00[ 160s ] thds: 16 tps: 264.89 qps: 5298.56 (r/w/o: 3708.43/934.46/655.67) lat (ms,95%): 89.16 err/s: 0.00 reconn/s: 0.00[ 170s ] thds: 16 tps: 264.01 qps: 5275.24 (r/w/o: 3693.57/928.64/653.03) lat (ms,95%): 90.78 err/s: 0.00 reconn/s: 0.00[ 180s ] thds: 16 tps: 272.50 qps: 5450.32 (r/w/o: 3814.11/965.40/670.80) lat (ms,95%): 86.00 err/s: 0.00 reconn/s: 0.00[ 190s ] thds: 16 tps: 251.79 qps: 5040.57 (r/w/o: 3528.61/890.58/621.38) lat (ms,95%): 87.56 err/s: 0.00 reconn/s: 0.00[ 200s ] thds: 16 tps: 266.11 qps: 5310.32 (r/w/o: 3716.89/940.72/652.72) lat (ms,95%): 89.16 err/s: 0.00 reconn/s: 0.00[ 210s ] thds: 16 tps: 268.00 qps: 5371.85 (r/w/o: 3760.63/950.01/661.21) lat (ms,95%): 90.78 err/s: 0.00 reconn/s: 0.00[ 220s ] thds: 16 tps: 272.70 qps: 5445.91 (r/w/o: 3811.74/958.68/675.49) lat (ms,95%): 87.56 err/s: 0.00 reconn/s: 0.00[ 230s ] thds: 16 tps: 165.50 qps: 3318.04 (r/w/o: 2323.03/585.11/409.91) lat (ms,95%): 110.66 err/s: 0.00 reconn/s: 0.00[ 240s ] thds: 16 tps: 254.10 qps: 5081.70 (r/w/o: 3557.40/902.00/622.30) lat (ms,95%): 89.16 err/s: 0.00 reconn/s: 0.00[ 250s ] thds: 16 tps: 266.59 qps: 5331.67 (r/w/o: 3732.31/947.58/651.78) lat (ms,95%): 90.78 err/s: 0.00 reconn/s: 0.00[ 260s ] thds: 16 tps: 242.81 qps: 4847.82 (r/w/o: 3394.88/856.32/596.61) lat (ms,95%): 95.81 err/s: 0.00 reconn/s: 0.00[ 270s ] thds: 16 tps: 253.10 qps: 5057.94 (r/w/o: 3539.66/895.09/623.19) lat (ms,95%): 92.42 err/s: 0.00 reconn/s: 0.00[ 280s ] thds: 16 tps: 265.00 qps: 5312.84 (r/w/o: 3718.12/943.81/650.90) lat (ms,95%): 87.56 err/s: 0.00 reconn/s: 0.00[ 290s ] thds: 16 tps: 266.40 qps: 5325.03 (r/w/o: 3728.62/939.70/656.70) lat (ms,95%): 89.16 err/s: 0.00 reconn/s: 0.00[ 300s ] thds: 16 tps: 261.70 qps: 5236.60 (r/w/o: 3664.73/937.48/634.39) lat (ms,95%): 90.78 err/s: 0.00 reconn/s: 0.00[ 310s ] thds: 16 tps: 233.60 qps: 4669.68 (r/w/o: 3269.39/824.60/575.70) lat (ms,95%): 90.78 err/s: 0.00 reconn/s: 0.00[ 320s ] thds: 16 tps: 121.30 qps: 2419.83 (r/w/o: 1693.82/428.91/297.10) lat (ms,95%): 219.36 err/s: 0.00 reconn/s: 0.00[ 330s ] thds: 16 tps: 143.50 qps: 2877.53 (r/w/o: 2014.02/516.00/347.50) lat (ms,95%): 223.34 err/s: 0.00 reconn/s: 0.00[ 340s ] thds: 16 tps: 212.70 qps: 4249.95 (r/w/o: 2975.26/759.69/514.99) lat (ms,95%): 110.66 err/s: 0.00 reconn/s: 0.00[ 350s ] thds: 16 tps: 195.90 qps: 3923.12 (r/w/o: 2745.51/696.00/481.60) lat (ms,95%): 130.13 err/s: 0.00 reconn/s: 0.00[ 360s ] thds: 16 tps: 173.80 qps: 3475.78 (r/w/o: 2433.19/618.60/424.00) lat (ms,95%): 137.35 err/s: 0.00 reconn/s: 0.00[ 370s ] thds: 16 tps: 257.30 qps: 5141.38 (r/w/o: 3599.59/914.70/627.10) lat (ms,95%): 94.10 err/s: 0.00 reconn/s: 0.00[ 380s ] thds: 16 tps: 262.80 qps: 5261.07 (r/w/o: 3681.85/936.11/643.11) lat (ms,95%): 92.42 err/s: 0.00 reconn/s: 0.00[ 390s ] thds: 16 tps: 248.70 qps: 4966.09 (r/w/o: 3477.09/881.20/607.80) lat (ms,95%): 90.78 err/s: 0.00 reconn/s: 0.00[ 400s ] thds: 16 tps: 248.98 qps: 4984.93 (r/w/o: 3489.87/888.86/606.20) lat (ms,95%): 95.81 err/s: 0.00 reconn/s: 0.00[ 410s ] thds: 16 tps: 257.43 qps: 5148.67 (r/w/o: 3604.10/912.76/631.82) lat (ms,95%): 89.16 err/s: 0.00 reconn/s: 0.00[ 420s ] thds: 16 tps: 247.50 qps: 4950.27 (r/w/o: 3464.48/874.60/611.20) lat (ms,95%): 90.78 err/s: 0.00 reconn/s: 0.00[ 430s ] thds: 16 tps: 229.70 qps: 4587.79 (r/w/o: 3210.49/815.10/562.20) lat (ms,95%): 99.33 err/s: 0.00 reconn/s: 0.00[ 440s ] thds: 16 tps: 242.50 qps: 4858.35 (r/w/o: 3401.24/863.41/593.71) lat (ms,95%): 95.81 err/s: 0.00 reconn/s: 0.00[ 450s ] thds: 16 tps: 240.40 qps: 4802.77 (r/w/o: 3363.58/851.39/587.80) lat (ms,95%): 99.33 err/s: 0.00 reconn/s: 0.00[ 460s ] thds: 16 tps: 103.00 qps: 2065.22 (r/w/o: 1444.01/373.20/248.00) lat (ms,95%): 419.45 err/s: 0.00 reconn/s: 0.00[ 470s ] thds: 16 tps: 197.00 qps: 3931.99 (r/w/o: 2752.79/696.50/482.70) lat (ms,95%): 130.13 err/s: 0.00 reconn/s: 0.00[ 480s ] thds: 16 tps: 231.30 qps: 4629.34 (r/w/o: 3241.26/830.29/557.79) lat (ms,95%): 108.68 err/s: 0.00 reconn/s: 0.00[ 490s ] thds: 16 tps: 231.70 qps: 4636.82 (r/w/o: 3245.71/828.90/562.20) lat (ms,95%): 99.33 err/s: 0.00 reconn/s: 0.00[ 500s ] thds: 16 tps: 172.30 qps: 3445.12 (r/w/o: 2411.12/614.20/419.80) lat (ms,95%): 110.66 err/s: 0.00 reconn/s: 0.00[ 510s ] thds: 16 tps: 146.10 qps: 2922.60 (r/w/o: 2046.40/520.50/355.70) lat (ms,95%): 186.54 err/s: 0.00 reconn/s: 0.00[ 520s ] thds: 16 tps: 218.40 qps: 4370.05 (r/w/o: 3057.86/779.69/532.49) lat (ms,95%): 108.68 err/s: 0.00 reconn/s: 0.00[ 530s ] thds: 16 tps: 227.30 qps: 4542.54 (r/w/o: 3179.93/808.61/554.01) lat (ms,95%): 108.68 err/s: 0.00 reconn/s: 0.00[ 540s ] thds: 16 tps: 182.70 qps: 3655.83 (r/w/o: 2558.85/647.49/449.49) lat (ms,95%): 110.66 err/s: 0.00 reconn/s: 0.00[ 550s ] thds: 16 tps: 173.80 qps: 3472.65 (r/w/o: 2431.83/621.81/419.01) lat (ms,95%): 110.66 err/s: 0.00 reconn/s: 0.00[ 560s ] thds: 16 tps: 149.20 qps: 2988.63 (r/w/o: 2091.42/535.11/362.10) lat (ms,95%): 170.48 err/s: 0.00 reconn/s: 0.00[ 570s ] thds: 16 tps: 89.30 qps: 1786.20 (r/w/o: 1250.20/319.90/216.10) lat (ms,95%): 977.74 err/s: 0.00 reconn/s: 0.00[ 580s ] thds: 16 tps: 187.70 qps: 3748.93 (r/w/o: 2626.72/670.40/451.80) lat (ms,95%): 134.90 err/s: 0.00 reconn/s: 0.00[ 590s ] thds: 16 tps: 222.00 qps: 4445.29 (r/w/o: 3109.09/792.30/543.90) lat (ms,95%): 108.68 err/s: 0.00 reconn/s: 0.00[ 600s ] thds: 16 tps: 238.90 qps: 4771.86 (r/w/o: 3341.27/851.99/578.60) lat (ms,95%): 104.84 err/s: 0.00 reconn/s: 0.00SQL statistics:queries performed:read: 1957830write: 480654other: 358416total: 2796900transactions: 139845 (233.05 per sec.)queries: 2796900 (4660.90 per sec.)ignored errors: 0 (0.00 per sec.)reconnects: 0 (0.00 per sec.)General statistics:total time: 600.0750stotal number of events: 139845Latency (ms):min: 7.91avg: 68.65max: 4289.3495th percentile: 104.84sum: 9600278.25Threads fairness:events (avg/stddev): 8740.3125/13.61execution time (avg/stddev): 600.0174/0.02jan@galapq:~/10.6-bug/mysql-test$ sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/10.6-bug/mysql-test/var/tmp/mysqld.1.sock --table-size=1000000 --mysql-user=root runsysbench 1.0.18 (using system LuaJIT 2.1.0-beta3)Running the test with following options:Number of threads: 16Report intermediate results every 10 second(s)Initializing random number generator from current timeInitializing worker threads...Threads started![ 10s ] thds: 16 tps: 330.42 qps: 6638.79 (r/w/o: 4648.27/1191.31/799.21) lat (ms,95%): 82.96 err/s: 0.00 reconn/s: 0.00[ 20s ] thds: 16 tps: 390.32 qps: 7803.53 (r/w/o: 5464.03/1404.86/934.64) lat (ms,95%): 66.84 err/s: 0.00 reconn/s: 0.00[ 30s ] thds: 16 tps: 429.80 qps: 8592.66 (r/w/o: 6014.24/1546.11/1032.31) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 40s ] thds: 16 tps: 409.19 qps: 8187.60 (r/w/o: 5731.36/1464.66/991.58) lat (ms,95%): 62.19 err/s: 0.00 reconn/s: 0.00[ 50s ] thds: 16 tps: 413.71 qps: 8270.81 (r/w/o: 5790.48/1485.32/995.01) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 60s ] thds: 16 tps: 447.70 qps: 8956.98 (r/w/o: 6268.78/1615.70/1072.50) lat (ms,95%): 54.83 err/s: 0.00 reconn/s: 0.00[ 70s ] thds: 16 tps: 278.20 qps: 5562.51 (r/w/o: 3894.61/1000.80/667.10) lat (ms,95%): 207.82 err/s: 0.00 reconn/s: 0.00[ 80s ] thds: 16 tps: 447.71 qps: 8948.04 (r/w/o: 6264.40/1616.62/1067.02) lat (ms,95%): 57.87 err/s: 0.00 reconn/s: 0.00[ 90s ] thds: 16 tps: 450.29 qps: 9011.05 (r/w/o: 6306.20/1629.97/1074.88) lat (ms,95%): 58.92 err/s: 0.00 reconn/s: 0.00[ 100s ] thds: 16 tps: 385.10 qps: 7696.93 (r/w/o: 5388.65/1389.79/918.49) lat (ms,95%): 68.05 err/s: 0.00 reconn/s: 0.00[ 110s ] thds: 16 tps: 345.70 qps: 6918.80 (r/w/o: 4841.07/1250.22/827.51) lat (ms,95%): 71.83 err/s: 0.00 reconn/s: 0.00[ 120s ] thds: 16 tps: 311.90 qps: 6241.51 (r/w/o: 4369.41/1126.90/745.20) lat (ms,95%): 77.19 err/s: 0.00 reconn/s: 0.00[ 130s ] thds: 16 tps: 265.20 qps: 5301.62 (r/w/o: 3711.91/960.60/629.10) lat (ms,95%): 112.67 err/s: 0.00 reconn/s: 0.00[ 140s ] thds: 16 tps: 433.00 qps: 8664.33 (r/w/o: 6064.32/1559.01/1041.00) lat (ms,95%): 62.19 err/s: 0.00 reconn/s: 0.00[ 150s ] thds: 16 tps: 438.30 qps: 8750.53 (r/w/o: 6124.72/1581.51/1044.30) lat (ms,95%): 58.92 err/s: 0.00 reconn/s: 0.00[ 160s ] thds: 16 tps: 357.89 qps: 7173.07 (r/w/o: 5022.01/1296.88/854.18) lat (ms,95%): 69.29 err/s: 0.00 reconn/s: 0.00[ 170s ] thds: 16 tps: 291.70 qps: 5819.47 (r/w/o: 4073.18/1054.99/691.30) lat (ms,95%): 78.60 err/s: 0.00 reconn/s: 0.00[ 180s ] thds: 16 tps: 216.40 qps: 4332.43 (r/w/o: 3033.02/785.71/513.70) lat (ms,95%): 282.25 err/s: 0.00 reconn/s: 0.00[ 190s ] thds: 16 tps: 445.90 qps: 8912.59 (r/w/o: 6239.36/1609.32/1063.91) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00[ 200s ] thds: 16 tps: 455.10 qps: 9117.59 (r/w/o: 6381.89/1652.40/1083.30) lat (ms,95%): 56.84 err/s: 0.00 reconn/s: 0.00[ 210s ] thds: 16 tps: 457.70 qps: 9142.30 (r/w/o: 6400.33/1652.18/1089.79) lat (ms,95%): 57.87 err/s: 0.00 reconn/s: 0.00[ 220s ] thds: 16 tps: 292.70 qps: 5865.98 (r/w/o: 4105.26/1064.61/696.11) lat (ms,95%): 74.46 err/s: 0.00 reconn/s: 0.00[ 230s ] thds: 16 tps: 293.70 qps: 5873.93 (r/w/o: 4111.75/1062.29/699.89) lat (ms,95%): 82.96 err/s: 0.00 reconn/s: 0.00[ 240s ] thds: 16 tps: 179.80 qps: 3594.21 (r/w/o: 2515.81/652.10/426.30) lat (ms,95%): 114.72 err/s: 0.00 reconn/s: 0.00[ 250s ] thds: 16 tps: 150.20 qps: 3005.60 (r/w/o: 2104.20/545.30/356.10) lat (ms,95%): 502.20 err/s: 0.00 reconn/s: 0.00[ 260s ] thds: 16 tps: 384.10 qps: 7675.57 (r/w/o: 5376.18/1383.29/916.10) lat (ms,95%): 74.46 err/s: 0.00 reconn/s: 0.00[ 270s ] thds: 16 tps: 401.70 qps: 8040.28 (r/w/o: 5625.05/1459.81/955.41) lat (ms,95%): 62.19 err/s: 0.00 reconn/s: 0.00[ 280s ] thds: 16 tps: 433.30 qps: 8663.64 (r/w/o: 6065.66/1572.79/1025.19) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00[ 290s ] thds: 16 tps: 448.50 qps: 8967.64 (r/w/o: 6276.16/1626.79/1064.69) lat (ms,95%): 57.87 err/s: 0.00 reconn/s: 0.00[ 300s ] thds: 16 tps: 342.91 qps: 6862.11 (r/w/o: 4803.98/1252.02/806.11) lat (ms,95%): 74.46 err/s: 0.00 reconn/s: 0.00[ 310s ] thds: 16 tps: 363.30 qps: 7262.79 (r/w/o: 5083.69/1323.20/855.90) lat (ms,95%): 66.84 err/s: 0.00 reconn/s: 0.00[ 320s ] thds: 16 tps: 309.20 qps: 6188.17 (r/w/o: 4331.28/1124.79/732.10) lat (ms,95%): 78.60 err/s: 0.00 reconn/s: 0.00[ 330s ] thds: 16 tps: 398.30 qps: 7965.80 (r/w/o: 5576.20/1446.00/943.60) lat (ms,95%): 70.55 err/s: 0.00 reconn/s: 0.00[ 340s ] thds: 16 tps: 438.40 qps: 8765.69 (r/w/o: 6137.49/1589.20/1039.00) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00[ 350s ] thds: 16 tps: 391.50 qps: 7816.94 (r/w/o: 5470.66/1420.59/925.69) lat (ms,95%): 64.47 err/s: 0.00 reconn/s: 0.00[ 360s ] thds: 16 tps: 346.90 qps: 6953.60 (r/w/o: 4867.07/1267.32/819.21) lat (ms,95%): 66.84 err/s: 0.00 reconn/s: 0.00[ 370s ] thds: 16 tps: 305.50 qps: 6100.14 (r/w/o: 4271.96/1108.69/719.49) lat (ms,95%): 70.55 err/s: 0.00 reconn/s: 0.00[ 380s ] thds: 16 tps: 449.01 qps: 8989.94 (r/w/o: 6291.10/1636.52/1062.32) lat (ms,95%): 58.92 err/s: 0.00 reconn/s: 0.00[ 390s ] thds: 16 tps: 440.79 qps: 8802.79 (r/w/o: 6161.62/1606.68/1034.49) lat (ms,95%): 58.92 err/s: 0.00 reconn/s: 0.00[ 400s ] thds: 16 tps: 450.50 qps: 9016.45 (r/w/o: 6311.64/1634.91/1069.91) lat (ms,95%): 55.82 err/s: 0.00 reconn/s: 0.00[ 410s ] thds: 16 tps: 449.69 qps: 9000.57 (r/w/o: 6300.61/1637.58/1062.38) lat (ms,95%): 57.87 err/s: 0.00 reconn/s: 0.00[ 420s ] thds: 16 tps: 462.61 qps: 9248.18 (r/w/o: 6474.23/1685.23/1088.72) lat (ms,95%): 56.84 err/s: 0.00 reconn/s: 0.00[ 430s ] thds: 16 tps: 452.90 qps: 9056.18 (r/w/o: 6339.09/1648.60/1068.50) lat (ms,95%): 57.87 err/s: 0.00 reconn/s: 0.00[ 440s ] thds: 16 tps: 460.49 qps: 9215.66 (r/w/o: 6450.70/1681.37/1083.58) lat (ms,95%): 57.87 err/s: 0.00 reconn/s: 0.00[ 450s ] thds: 16 tps: 429.21 qps: 8575.06 (r/w/o: 6002.61/1567.03/1005.42) lat (ms,95%): 61.08 err/s: 0.00 reconn/s: 0.00[ 460s ] thds: 16 tps: 180.10 qps: 3601.70 (r/w/o: 2521.10/658.20/422.40) lat (ms,95%): 325.98 err/s: 0.00 reconn/s: 0.00[ 470s ] thds: 16 tps: 311.39 qps: 6237.20 (r/w/o: 4366.06/1135.36/735.78) lat (ms,95%): 95.81 err/s: 0.00 reconn/s: 0.00[ 480s ] thds: 16 tps: 370.41 qps: 7397.14 (r/w/o: 5178.20/1350.23/868.72) lat (ms,95%): 65.65 err/s: 0.00 reconn/s: 0.00[ 490s ] thds: 16 tps: 431.60 qps: 8636.85 (r/w/o: 6047.64/1574.71/1014.51) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00[ 500s ] thds: 16 tps: 406.69 qps: 8136.47 (r/w/o: 5694.64/1481.36/960.47) lat (ms,95%): 63.32 err/s: 0.00 reconn/s: 0.00[ 510s ] thds: 16 tps: 447.01 qps: 8938.00 (r/w/o: 6257.51/1632.25/1048.23) lat (ms,95%): 58.92 err/s: 0.00 reconn/s: 0.00[ 520s ] thds: 16 tps: 450.39 qps: 9010.00 (r/w/o: 6306.43/1650.08/1053.49) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00[ 530s ] thds: 16 tps: 440.60 qps: 8805.98 (r/w/o: 6162.56/1615.11/1028.31) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00[ 540s ] thds: 16 tps: 388.30 qps: 7762.00 (r/w/o: 5432.63/1425.48/903.89) lat (ms,95%): 70.55 err/s: 0.00 reconn/s: 0.00[ 550s ] thds: 16 tps: 354.70 qps: 7104.96 (r/w/o: 4975.24/1304.81/824.91) lat (ms,95%): 73.13 err/s: 0.00 reconn/s: 0.00[ 560s ] thds: 16 tps: 327.50 qps: 6541.80 (r/w/o: 4577.90/1197.60/766.30) lat (ms,95%): 69.29 err/s: 0.00 reconn/s: 0.00[ 570s ] thds: 16 tps: 341.30 qps: 6837.00 (r/w/o: 4786.40/1254.00/796.60) lat (ms,95%): 65.65 err/s: 0.00 reconn/s: 0.00[ 580s ] thds: 16 tps: 144.00 qps: 2868.01 (r/w/o: 2007.21/523.20/337.60) lat (ms,95%): 646.19 err/s: 0.00 reconn/s: 0.00[ 590s ] thds: 16 tps: 379.60 qps: 7592.94 (r/w/o: 5315.86/1388.19/888.89) lat (ms,95%): 70.55 err/s: 0.00 reconn/s: 0.00[ 600s ] thds: 16 tps: 430.50 qps: 8621.07 (r/w/o: 6034.35/1584.11/1002.61) lat (ms,95%): 59.99 err/s: 0.00 reconn/s: 0.00SQL statistics:queries performed:read: 3147508write: 816465other: 532467total: 4496440transactions: 224822 (374.68 per sec.)queries: 4496440 (7493.57 per sec.)ignored errors: 0 (0.00 per sec.)reconnects: 0 (0.00 per sec.)General statistics:total time: 600.0382stotal number of events: 224822Latency (ms):min: 3.94avg: 42.70max: 3315.1395th percentile: 68.05sum: 9599490.64Threads fairness:events (avg/stddev): 14051.3750/45.29execution time (avg/stddev): 599.9682/0.01When I run two sysbench instances one to node_1 and second to node 2 and leave node3 as slave i.e. multi-master I received following results:
jan@galapq:~/10.6-bug/mysql-test$ sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/10.6-bug/mysql-test/var/tmp/mysqld.1.sock --table-size=1000000 --mysql-user=root runsysbench 1.0.18 (using system LuaJIT 2.1.0-beta3)Running the test with following options:Number of threads: 16Report intermediate results every 10 second(s)Initializing random number generator from current timeInitializing worker threads...Threads started![ 10s ] thds: 16 tps: 93.08 qps: 1896.92 (r/w/o: 1329.36/337.71/229.84) lat (ms,95%): 267.41 err/s: 0.30 reconn/s: 0.00[ 20s ] thds: 16 tps: 98.11 qps: 1972.51 (r/w/o: 1381.98/353.92/236.61) lat (ms,95%): 235.74 err/s: 0.70 reconn/s: 0.00[ 30s ] thds: 16 tps: 105.20 qps: 2140.21 (r/w/o: 1499.11/382.50/258.60) lat (ms,95%): 231.53 err/s: 1.90 reconn/s: 0.00[ 40s ] thds: 16 tps: 86.90 qps: 1752.40 (r/w/o: 1227.90/313.70/210.80) lat (ms,95%): 282.25 err/s: 0.70 reconn/s: 0.00[ 50s ] thds: 16 tps: 94.50 qps: 1903.30 (r/w/o: 1331.80/342.80/228.70) lat (ms,95%): 262.64 err/s: 0.60 reconn/s: 0.00[ 60s ] thds: 16 tps: 76.80 qps: 1550.78 (r/w/o: 1086.39/276.40/188.00) lat (ms,95%): 669.89 err/s: 0.80 reconn/s: 0.00[ 70s ] thds: 16 tps: 52.00 qps: 1044.21 (r/w/o: 730.81/186.60/126.80) lat (ms,95%): 200.47 err/s: 0.20 reconn/s: 0.00[ 80s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 90s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 100s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 110s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 120s ] thds: 16 tps: 44.10 qps: 889.79 (r/w/o: 623.69/161.30/104.80) lat (ms,95%): 493.24 err/s: 0.70 reconn/s: 0.00[ 130s ] thds: 16 tps: 81.40 qps: 1637.81 (r/w/o: 1147.11/294.00/196.70) lat (ms,95%): 287.38 err/s: 0.30 reconn/s: 0.00[ 140s ] thds: 16 tps: 107.50 qps: 2164.32 (r/w/o: 1515.01/387.60/261.70) lat (ms,95%): 223.34 err/s: 0.70 reconn/s: 0.00[ 150s ] thds: 16 tps: 118.50 qps: 2390.92 (r/w/o: 1674.71/431.70/284.50) lat (ms,95%): 204.11 err/s: 1.20 reconn/s: 0.00[ 160s ] thds: 16 tps: 101.40 qps: 2046.40 (r/w/o: 1433.30/366.40/246.70) lat (ms,95%): 189.93 err/s: 0.90 reconn/s: 0.00[ 170s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 180s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 190s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 200s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 210s ] thds: 16 tps: 6.90 qps: 136.40 (r/w/o: 95.80/23.20/17.40) lat (ms,95%): 50446.94 err/s: 0.40 reconn/s: 0.00[ 220s ] thds: 16 tps: 89.10 qps: 1798.39 (r/w/o: 1259.40/322.00/217.00) lat (ms,95%): 282.25 err/s: 0.40 reconn/s: 0.00[ 230s ] thds: 16 tps: 60.60 qps: 1227.67 (r/w/o: 859.98/220.69/147.00) lat (ms,95%): 539.71 err/s: 0.90 reconn/s: 0.00[ 240s ] thds: 16 tps: 88.00 qps: 1776.95 (r/w/o: 1244.24/319.21/213.51) lat (ms,95%): 314.45 err/s: 0.80 reconn/s: 0.00[ 250s ] thds: 16 tps: 108.80 qps: 2196.93 (r/w/o: 1538.55/393.89/264.49) lat (ms,95%): 227.40 err/s: 1.20 reconn/s: 0.00[ 260s ] thds: 16 tps: 119.70 qps: 2416.60 (r/w/o: 1692.60/434.50/289.50) lat (ms,95%): 207.82 err/s: 1.10 reconn/s: 0.00[ 270s ] thds: 16 tps: 119.90 qps: 2420.82 (r/w/o: 1695.41/435.80/289.60) lat (ms,95%): 204.11 err/s: 1.20 reconn/s: 0.00[ 280s ] thds: 16 tps: 123.20 qps: 2477.54 (r/w/o: 1735.13/446.41/296.01) lat (ms,95%): 196.89 err/s: 0.80 reconn/s: 0.00[ 290s ] thds: 16 tps: 105.50 qps: 2117.40 (r/w/o: 1482.10/376.70/258.60) lat (ms,95%): 186.54 err/s: 0.30 reconn/s: 0.00[ 300s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 310s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 320s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 330s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 340s ] thds: 16 tps: 7.60 qps: 148.80 (r/w/o: 104.40/25.80/18.60) lat (ms,95%): 50446.94 err/s: 0.00 reconn/s: 0.00[ 350s ] thds: 16 tps: 44.10 qps: 895.38 (r/w/o: 627.78/161.90/105.70) lat (ms,95%): 831.46 err/s: 0.60 reconn/s: 0.00[ 360s ] thds: 16 tps: 101.10 qps: 2048.35 (r/w/o: 1434.13/366.41/247.81) lat (ms,95%): 240.02 err/s: 1.40 reconn/s: 0.00[ 370s ] thds: 16 tps: 110.60 qps: 2233.56 (r/w/o: 1564.57/397.89/271.10) lat (ms,95%): 235.74 err/s: 1.20 reconn/s: 0.00[ 380s ] thds: 16 tps: 124.59 qps: 2510.36 (r/w/o: 1758.40/449.38/302.58) lat (ms,95%): 189.93 err/s: 0.90 reconn/s: 0.00[ 390s ] thds: 16 tps: 123.78 qps: 2485.39 (r/w/o: 1739.78/444.84/300.76) lat (ms,95%): 196.89 err/s: 0.60 reconn/s: 0.00[ 400s ] thds: 16 tps: 126.42 qps: 2552.04 (r/w/o: 1787.51/456.68/307.85) lat (ms,95%): 193.38 err/s: 1.20 reconn/s: 0.00[ 410s ] thds: 16 tps: 125.59 qps: 2526.75 (r/w/o: 1769.30/449.87/307.58) lat (ms,95%): 189.93 err/s: 0.90 reconn/s: 0.00[ 420s ] thds: 16 tps: 122.51 qps: 2468.35 (r/w/o: 1728.30/445.03/295.02) lat (ms,95%): 207.82 err/s: 0.90 reconn/s: 0.00[ 430s ] thds: 16 tps: 119.50 qps: 2401.12 (r/w/o: 1681.61/431.10/288.40) lat (ms,95%): 200.47 err/s: 0.70 reconn/s: 0.00[ 440s ] thds: 16 tps: 44.50 qps: 898.71 (r/w/o: 628.81/164.50/105.40) lat (ms,95%): 2238.47 err/s: 0.20 reconn/s: 0.00[ 450s ] thds: 16 tps: 68.80 qps: 1386.88 (r/w/o: 971.49/249.00/166.40) lat (ms,95%): 467.30 err/s: 0.60 reconn/s: 0.00[ 460s ] thds: 16 tps: 100.50 qps: 2028.69 (r/w/o: 1421.09/367.20/240.40) lat (ms,95%): 240.02 err/s: 1.00 reconn/s: 0.00[ 470s ] thds: 16 tps: 106.30 qps: 2136.13 (r/w/o: 1495.22/385.51/255.40) lat (ms,95%): 223.34 err/s: 0.50 reconn/s: 0.00[ 480s ] thds: 16 tps: 120.90 qps: 2447.13 (r/w/o: 1714.45/438.39/294.29) lat (ms,95%): 200.47 err/s: 1.60 reconn/s: 0.00[ 490s ] thds: 16 tps: 119.70 qps: 2427.48 (r/w/o: 1700.15/436.11/291.21) lat (ms,95%): 200.47 err/s: 1.70 reconn/s: 0.00[ 500s ] thds: 16 tps: 128.20 qps: 2584.87 (r/w/o: 1810.18/465.69/309.00) lat (ms,95%): 183.21 err/s: 1.10 reconn/s: 0.00[ 510s ] thds: 16 tps: 120.70 qps: 2436.87 (r/w/o: 1706.88/438.99/291.00) lat (ms,95%): 196.89 err/s: 1.30 reconn/s: 0.00[ 520s ] thds: 16 tps: 116.20 qps: 2341.26 (r/w/o: 1640.04/420.81/280.41) lat (ms,95%): 186.54 err/s: 1.00 reconn/s: 0.00[ 530s ] thds: 16 tps: 42.00 qps: 844.79 (r/w/o: 591.09/154.00/99.70) lat (ms,95%): 1149.76 err/s: 0.30 reconn/s: 0.00[ 540s ] thds: 16 tps: 92.00 qps: 1866.72 (r/w/o: 1307.82/336.60/222.30) lat (ms,95%): 277.21 err/s: 1.20 reconn/s: 0.00[ 550s ] thds: 16 tps: 109.40 qps: 2205.70 (r/w/o: 1545.10/396.60/264.00) lat (ms,95%): 223.34 err/s: 1.00 reconn/s: 0.00[ 560s ] thds: 16 tps: 112.40 qps: 2277.73 (r/w/o: 1595.12/408.70/273.90) lat (ms,95%): 215.44 err/s: 1.50 reconn/s: 0.00[ 570s ] thds: 16 tps: 124.60 qps: 2509.09 (r/w/o: 1756.89/452.50/299.70) lat (ms,95%): 186.54 err/s: 1.00 reconn/s: 0.00[ 580s ] thds: 16 tps: 119.70 qps: 2421.71 (r/w/o: 1696.20/435.80/289.70) lat (ms,95%): 211.60 err/s: 1.40 reconn/s: 0.00[ 590s ] thds: 16 tps: 86.80 qps: 1753.79 (r/w/o: 1228.39/317.30/208.10) lat (ms,95%): 189.93 err/s: 0.90 reconn/s: 0.00[ 600s ] thds: 16 tps: 55.60 qps: 1123.78 (r/w/o: 787.89/202.30/133.60) lat (ms,95%): 1739.68 err/s: 0.70 reconn/s: 0.00SQL statistics:queries performed:read: 643776write: 165074other: 110386total: 919236transactions: 45569 (75.93 per sec.)queries: 919236 (1531.71 per sec.)ignored errors: 415 (0.69 per sec.)reconnects: 0 (0.00 per sec.)General statistics:total time: 600.1337stotal number of events: 45569Latency (ms):min: 22.61avg: 210.70max: 51326.5595th percentile: 248.83sum: 9601279.79Threads fairness:events (avg/stddev): 2848.0625/6.92execution time (avg/stddev): 600.0800/0.03jan@galapq:~$ sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/10.6-bug/mysql-test/var/tmp/mysqld.2.sock --table-size=1000000 --mysql-user=root runsysbench 1.0.18 (using system LuaJIT 2.1.0-beta3)Running the test with following options:Number of threads: 16Report intermediate results every 10 second(s)Initializing random number generator from current timeInitializing worker threads...Threads started![ 10s ] thds: 16 tps: 87.98 qps: 1804.78 (r/w/o: 1265.30/318.93/220.55) lat (ms,95%): 292.60 err/s: 0.80 reconn/s: 0.00[ 20s ] thds: 16 tps: 102.90 qps: 2078.68 (r/w/o: 1455.96/370.41/252.31) lat (ms,95%): 267.41 err/s: 1.20 reconn/s: 0.00[ 30s ] thds: 16 tps: 110.20 qps: 2223.51 (r/w/o: 1556.90/399.40/267.20) lat (ms,95%): 219.36 err/s: 0.90 reconn/s: 0.00[ 40s ] thds: 16 tps: 93.10 qps: 1871.99 (r/w/o: 1310.79/333.90/227.30) lat (ms,95%): 267.41 err/s: 0.60 reconn/s: 0.00[ 50s ] thds: 16 tps: 103.30 qps: 2086.37 (r/w/o: 1461.18/369.50/255.70) lat (ms,95%): 262.64 err/s: 1.00 reconn/s: 0.00[ 60s ] thds: 16 tps: 74.00 qps: 1491.31 (r/w/o: 1044.41/265.90/181.00) lat (ms,95%): 746.32 err/s: 0.70 reconn/s: 0.00[ 70s ] thds: 16 tps: 24.10 qps: 489.50 (r/w/o: 343.00/87.40/59.10) lat (ms,95%): 183.21 err/s: 0.30 reconn/s: 0.00[ 80s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 90s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 100s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 110s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 120s ] thds: 16 tps: 59.70 qps: 1203.20 (r/w/o: 842.90/211.90/148.40) lat (ms,95%): 502.20 err/s: 0.60 reconn/s: 0.00[ 130s ] thds: 16 tps: 76.50 qps: 1551.21 (r/w/o: 1086.81/277.30/187.10) lat (ms,95%): 634.66 err/s: 1.10 reconn/s: 0.00[ 140s ] thds: 16 tps: 116.90 qps: 2361.97 (r/w/o: 1654.28/424.19/283.50) lat (ms,95%): 204.11 err/s: 1.20 reconn/s: 0.00[ 150s ] thds: 16 tps: 128.20 qps: 2588.42 (r/w/o: 1813.02/467.60/307.80) lat (ms,95%): 179.94 err/s: 1.30 reconn/s: 0.00[ 160s ] thds: 16 tps: 77.20 qps: 1571.40 (r/w/o: 1100.40/281.80/189.20) lat (ms,95%): 200.47 err/s: 1.40 reconn/s: 0.00[ 170s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 180s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 190s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 200s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 210s ] thds: 16 tps: 27.80 qps: 563.60 (r/w/o: 394.80/101.10/67.70) lat (ms,95%): 49546.69 err/s: 0.40 reconn/s: 0.00[ 220s ] thds: 16 tps: 85.10 qps: 1711.53 (r/w/o: 1198.42/303.21/209.90) lat (ms,95%): 297.92 err/s: 0.50 reconn/s: 0.00[ 230s ] thds: 16 tps: 55.20 qps: 1108.38 (r/w/o: 776.29/198.10/134.00) lat (ms,95%): 1304.21 err/s: 0.40 reconn/s: 0.00[ 240s ] thds: 16 tps: 100.00 qps: 2016.49 (r/w/o: 1411.90/364.00/240.60) lat (ms,95%): 240.02 err/s: 0.70 reconn/s: 0.00[ 250s ] thds: 16 tps: 114.10 qps: 2299.10 (r/w/o: 1610.00/415.10/274.00) lat (ms,95%): 207.82 err/s: 0.90 reconn/s: 0.00[ 260s ] thds: 16 tps: 123.60 qps: 2498.52 (r/w/o: 1750.22/447.90/300.40) lat (ms,95%): 193.38 err/s: 1.50 reconn/s: 0.00[ 270s ] thds: 16 tps: 125.10 qps: 2522.68 (r/w/o: 1767.49/452.50/302.70) lat (ms,95%): 207.82 err/s: 1.10 reconn/s: 0.00[ 280s ] thds: 16 tps: 135.50 qps: 2738.11 (r/w/o: 1917.11/489.10/331.90) lat (ms,95%): 176.73 err/s: 1.40 reconn/s: 0.00[ 290s ] thds: 16 tps: 77.10 qps: 1559.21 (r/w/o: 1092.01/277.30/189.90) lat (ms,95%): 179.94 err/s: 0.90 reconn/s: 0.00[ 300s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 310s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 320s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 330s ] thds: 16 tps: 0.00 qps: 0.00 (r/w/o: 0.00/0.00/0.00) lat (ms,95%): 0.00 err/s: 0.00 reconn/s: 0.00[ 340s ] thds: 16 tps: 17.20 qps: 349.00 (r/w/o: 245.40/61.30/42.30) lat (ms,95%): 50446.94 err/s: 0.40 reconn/s: 0.00[ 350s ] thds: 16 tps: 49.10 qps: 994.12 (r/w/o: 695.41/179.90/118.80) lat (ms,95%): 1258.08 err/s: 0.50 reconn/s: 0.00[ 360s ] thds: 16 tps: 110.90 qps: 2238.91 (r/w/o: 1568.01/398.90/272.00) lat (ms,95%): 219.36 err/s: 1.10 reconn/s: 0.00[ 370s ] thds: 16 tps: 115.50 qps: 2322.27 (r/w/o: 1626.38/413.29/282.60) lat (ms,95%): 207.82 err/s: 0.90 reconn/s: 0.00[ 380s ] thds: 16 tps: 122.50 qps: 2480.93 (r/w/o: 1738.02/442.81/300.10) lat (ms,95%): 204.11 err/s: 1.50 reconn/s: 0.00[ 390s ] thds: 16 tps: 126.58 qps: 2560.26 (r/w/o: 1792.96/459.24/308.06) lat (ms,95%): 186.54 err/s: 1.40 reconn/s: 0.00[ 400s ] thds: 16 tps: 128.12 qps: 2578.32 (r/w/o: 1806.02/463.86/308.44) lat (ms,95%): 186.54 err/s: 1.10 reconn/s: 0.00[ 410s ] thds: 16 tps: 124.59 qps: 2513.87 (r/w/o: 1759.91/452.58/301.38) lat (ms,95%): 189.93 err/s: 0.90 reconn/s: 0.00[ 420s ] thds: 16 tps: 117.91 qps: 2375.63 (r/w/o: 1663.59/421.72/290.32) lat (ms,95%): 196.89 err/s: 1.00 reconn/s: 0.00[ 430s ] thds: 16 tps: 133.28 qps: 2686.19 (r/w/o: 1881.09/481.45/323.66) lat (ms,95%): 173.58 err/s: 1.00 reconn/s: 0.00[ 440s ] thds: 16 tps: 32.90 qps: 665.67 (r/w/o: 466.25/120.31/79.11) lat (ms,95%): 2539.17 err/s: 0.40 reconn/s: 0.00[ 450s ] thds: 16 tps: 76.00 qps: 1540.89 (r/w/o: 1079.40/276.80/184.70) lat (ms,95%): 350.33 err/s: 1.10 reconn/s: 0.00[ 460s ] thds: 16 tps: 112.70 qps: 2274.52 (r/w/o: 1593.21/407.10/274.20) lat (ms,95%): 211.60 err/s: 1.10 reconn/s: 0.00[ 470s ] thds: 16 tps: 110.30 qps: 2226.79 (r/w/o: 1559.59/398.80/268.40) lat (ms,95%): 215.44 err/s: 1.20 reconn/s: 0.00[ 480s ] thds: 16 tps: 119.10 qps: 2399.00 (r/w/o: 1680.70/431.40/286.90) lat (ms,95%): 204.11 err/s: 1.10 reconn/s: 0.00[ 490s ] thds: 16 tps: 123.90 qps: 2505.21 (r/w/o: 1753.51/452.70/299.00) lat (ms,95%): 196.89 err/s: 1.10 reconn/s: 0.00[ 500s ] thds: 16 tps: 127.10 qps: 2564.60 (r/w/o: 1796.20/460.30/308.10) lat (ms,95%): 173.58 err/s: 1.20 reconn/s: 0.00[ 510s ] thds: 16 tps: 126.60 qps: 2554.27 (r/w/o: 1789.78/454.79/309.70) lat (ms,95%): 183.21 err/s: 1.50 reconn/s: 0.00[ 520s ] thds: 16 tps: 97.90 qps: 1979.48 (r/w/o: 1385.39/358.10/236.00) lat (ms,95%): 189.93 err/s: 0.80 reconn/s: 0.00[ 530s ] thds: 16 tps: 55.20 qps: 1114.63 (r/w/o: 780.62/202.30/131.70) lat (ms,95%): 893.56 err/s: 0.80 reconn/s: 0.00[ 540s ] thds: 16 tps: 101.90 qps: 2053.30 (r/w/o: 1438.70/367.80/246.80) lat (ms,95%): 231.53 err/s: 0.80 reconn/s: 0.00[ 550s ] thds: 16 tps: 115.90 qps: 2359.49 (r/w/o: 1653.09/424.10/282.30) lat (ms,95%): 207.82 err/s: 2.00 reconn/s: 0.00[ 560s ] thds: 16 tps: 125.40 qps: 2526.81 (r/w/o: 1768.61/450.90/307.30) lat (ms,95%): 186.54 err/s: 1.20 reconn/s: 0.00[ 570s ] thds: 16 tps: 121.60 qps: 2448.81 (r/w/o: 1715.31/442.60/290.90) lat (ms,95%): 193.38 err/s: 0.80 reconn/s: 0.00[ 580s ] thds: 16 tps: 132.50 qps: 2668.79 (r/w/o: 1868.99/476.40/323.40) lat (ms,95%): 176.73 err/s: 1.00 reconn/s: 0.00[ 590s ] thds: 16 tps: 64.80 qps: 1303.86 (r/w/o: 912.47/234.19/157.19) lat (ms,95%): 559.50 err/s: 0.40 reconn/s: 0.00[ 600s ] thds: 16 tps: 109.90 qps: 2205.98 (r/w/o: 1545.75/396.31/263.91) lat (ms,95%): 634.66 err/s: 0.40 reconn/s: 0.00SQL statistics:queries performed:read: 664188write: 169885other: 114293total: 948366transactions: 46986 (78.30 per sec.)queries: 948366 (1580.43 per sec.)ignored errors: 456 (0.76 per sec.)reconnects: 0 (0.00 per sec.)General statistics:total time: 600.0657stotal number of events: 46986Latency (ms):min: 21.48avg: 204.33max: 51132.7895th percentile: 244.38sum: 9600455.13Threads fairness:events (avg/stddev): 2936.6250/8.56execution time (avg/stddev): 600.0284/0.02jan@galapq:~/10.6-bug/mysql-test$ sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/10.6-bug/mysql-test/var/tmp/mysqld.1.sock --table-size=1000000 --mysql-user=root runsysbench 1.0.18 (using system LuaJIT 2.1.0-beta3)Running the test with following options:Number of threads: 16Report intermediate results every 10 second(s)Initializing random number generator from current timeInitializing worker threads...Threads started![ 10s ] thds: 16 tps: 76.38 qps: 1556.60 (r/w/o: 1091.82/160.56/304.22) lat (ms,95%): 977.74 err/s: 0.10 reconn/s: 0.00[ 20s ] thds: 16 tps: 65.50 qps: 1320.59 (r/w/o: 923.86/149.11/247.62) lat (ms,95%): 1213.57 err/s: 0.50 reconn/s: 0.00[ 30s ] thds: 16 tps: 207.00 qps: 4163.29 (r/w/o: 2914.89/536.10/712.30) lat (ms,95%): 116.80 err/s: 1.20 reconn/s: 0.00[ 40s ] thds: 16 tps: 225.50 qps: 4546.02 (r/w/o: 3183.61/659.60/702.80) lat (ms,95%): 104.84 err/s: 1.80 reconn/s: 0.00[ 50s ] thds: 16 tps: 229.00 qps: 4618.74 (r/w/o: 3235.96/719.59/663.19) lat (ms,95%): 104.84 err/s: 2.20 reconn/s: 0.00[ 60s ] thds: 16 tps: 165.70 qps: 3349.02 (r/w/o: 2345.02/543.70/460.30) lat (ms,95%): 132.49 err/s: 1.90 reconn/s: 0.00[ 70s ] thds: 16 tps: 138.90 qps: 2808.31 (r/w/o: 1966.40/473.50/368.40) lat (ms,95%): 419.45 err/s: 1.40 reconn/s: 0.00[ 80s ] thds: 16 tps: 215.70 qps: 4356.82 (r/w/o: 3051.61/735.60/569.60) lat (ms,95%): 112.67 err/s: 2.50 reconn/s: 0.00[ 90s ] thds: 16 tps: 223.80 qps: 4520.39 (r/w/o: 3165.89/774.90/579.60) lat (ms,95%): 106.75 err/s: 2.20 reconn/s: 0.00[ 100s ] thds: 16 tps: 212.20 qps: 4289.00 (r/w/o: 3004.10/753.60/531.30) lat (ms,95%): 114.72 err/s: 2.30 reconn/s: 0.00[ 110s ] thds: 16 tps: 212.20 qps: 4280.86 (r/w/o: 2997.57/747.99/535.29) lat (ms,95%): 114.72 err/s: 1.90 reconn/s: 0.00[ 120s ] thds: 16 tps: 228.70 qps: 4616.73 (r/w/o: 3233.42/805.61/577.70) lat (ms,95%): 139.85 err/s: 2.30 reconn/s: 0.00[ 130s ] thds: 16 tps: 216.70 qps: 4366.21 (r/w/o: 3058.34/766.49/541.39) lat (ms,95%): 118.92 err/s: 2.10 reconn/s: 0.00[ 140s ] thds: 16 tps: 205.50 qps: 4163.95 (r/w/o: 2916.03/738.11/509.81) lat (ms,95%): 118.92 err/s: 2.40 reconn/s: 0.00[ 150s ] thds: 16 tps: 208.90 qps: 4217.23 (r/w/o: 2953.92/739.51/523.80) lat (ms,95%): 116.80 err/s: 2.30 reconn/s: 0.00[ 160s ] thds: 16 tps: 177.70 qps: 3585.08 (r/w/o: 2510.29/641.20/433.60) lat (ms,95%): 112.67 err/s: 1.40 reconn/s: 0.00[ 170s ] thds: 16 tps: 202.00 qps: 4065.58 (r/w/o: 2846.98/717.90/500.70) lat (ms,95%): 125.52 err/s: 1.40 reconn/s: 0.00[ 180s ] thds: 16 tps: 216.10 qps: 4345.64 (r/w/o: 3043.33/771.81/530.50) lat (ms,95%): 114.72 err/s: 1.30 reconn/s: 0.00[ 190s ] thds: 16 tps: 218.20 qps: 4410.71 (r/w/o: 3089.30/779.70/541.70) lat (ms,95%): 110.66 err/s: 2.40 reconn/s: 0.00[ 200s ] thds: 16 tps: 214.10 qps: 4319.51 (r/w/o: 3025.21/769.30/525.00) lat (ms,95%): 110.66 err/s: 2.00 reconn/s: 0.00[ 210s ] thds: 16 tps: 126.30 qps: 2553.58 (r/w/o: 1787.99/452.90/312.70) lat (ms,95%): 292.60 err/s: 1.40 reconn/s: 0.00[ 220s ] thds: 16 tps: 124.40 qps: 2508.43 (r/w/o: 1756.95/445.89/305.59) lat (ms,95%): 559.50 err/s: 1.10 reconn/s: 0.00[ 230s ] thds: 16 tps: 193.51 qps: 3904.35 (r/w/o: 2734.31/697.73/472.32) lat (ms,95%): 116.80 err/s: 1.80 reconn/s: 0.00[ 240s ] thds: 16 tps: 205.30 qps: 4136.38 (r/w/o: 2896.59/733.30/506.50) lat (ms,95%): 123.28 err/s: 1.60 reconn/s: 0.00[ 250s ] thds: 16 tps: 209.40 qps: 4228.79 (r/w/o: 2962.19/750.60/516.00) lat (ms,95%): 112.67 err/s: 2.20 reconn/s: 0.00[ 260s ] thds: 16 tps: 156.10 qps: 3153.42 (r/w/o: 2208.61/562.60/382.20) lat (ms,95%): 118.92 err/s: 1.70 reconn/s: 0.00[ 270s ] thds: 16 tps: 145.40 qps: 2944.09 (r/w/o: 2061.60/522.90/359.60) lat (ms,95%): 244.38 err/s: 1.80 reconn/s: 0.00[ 280s ] thds: 16 tps: 206.40 qps: 4171.01 (r/w/o: 2921.54/740.08/509.39) lat (ms,95%): 116.80 err/s: 2.30 reconn/s: 0.00[ 290s ] thds: 16 tps: 213.30 qps: 4295.45 (r/w/o: 3008.43/766.01/521.01) lat (ms,95%): 112.67 err/s: 1.80 reconn/s: 0.00[ 300s ] thds: 16 tps: 215.70 qps: 4360.29 (r/w/o: 3053.50/775.50/531.30) lat (ms,95%): 121.08 err/s: 2.20 reconn/s: 0.00[ 310s ] thds: 16 tps: 201.40 qps: 4061.89 (r/w/o: 2844.79/723.10/494.00) lat (ms,95%): 123.28 err/s: 1.90 reconn/s: 0.00[ 320s ] thds: 16 tps: 171.90 qps: 3467.94 (r/w/o: 2428.33/617.41/422.20) lat (ms,95%): 150.29 err/s: 1.50 reconn/s: 0.00[ 330s ] thds: 16 tps: 169.60 qps: 3429.40 (r/w/o: 2402.00/608.60/418.80) lat (ms,95%): 150.29 err/s: 1.90 reconn/s: 0.00[ 340s ] thds: 16 tps: 168.90 qps: 3410.31 (r/w/o: 2388.41/607.40/414.50) lat (ms,95%): 125.52 err/s: 1.70 reconn/s: 0.00[ 350s ] thds: 16 tps: 53.30 qps: 1079.30 (r/w/o: 756.00/193.20/130.10) lat (ms,95%): 995.51 err/s: 0.70 reconn/s: 0.00[ 360s ] thds: 16 tps: 127.90 qps: 2567.58 (r/w/o: 1797.58/462.90/307.10) lat (ms,95%): 200.47 err/s: 0.60 reconn/s: 0.00[ 370s ] thds: 16 tps: 186.20 qps: 3758.60 (r/w/o: 2632.30/671.70/454.60) lat (ms,95%): 137.35 err/s: 1.90 reconn/s: 0.00[ 380s ] thds: 16 tps: 211.60 qps: 4276.64 (r/w/o: 2995.62/764.41/516.60) lat (ms,95%): 110.66 err/s: 2.20 reconn/s: 0.00[ 390s ] thds: 16 tps: 214.60 qps: 4326.53 (r/w/o: 3030.45/767.29/528.79) lat (ms,95%): 110.66 err/s: 2.10 reconn/s: 0.00[ 400s ] thds: 16 tps: 204.70 qps: 4135.67 (r/w/o: 2895.85/741.01/498.81) lat (ms,95%): 112.67 err/s: 2.00 reconn/s: 0.00[ 410s ] thds: 16 tps: 211.80 qps: 4264.80 (r/w/o: 2986.20/759.80/518.80) lat (ms,95%): 112.67 err/s: 1.40 reconn/s: 0.00[ 420s ] thds: 16 tps: 218.20 qps: 4392.69 (r/w/o: 3075.79/785.50/531.40) lat (ms,95%): 106.75 err/s: 1.70 reconn/s: 0.00[ 430s ] thds: 16 tps: 208.50 qps: 4227.10 (r/w/o: 2961.70/756.60/508.80) lat (ms,95%): 112.67 err/s: 2.90 reconn/s: 0.00[ 440s ] thds: 16 tps: 215.38 qps: 4361.95 (r/w/o: 3055.25/782.74/523.96) lat (ms,95%): 108.68 err/s: 2.80 reconn/s: 0.00[ 450s ] thds: 16 tps: 201.52 qps: 4067.03 (r/w/o: 2848.43/728.66/489.94) lat (ms,95%): 130.13 err/s: 2.30 reconn/s: 0.00[ 460s ] thds: 16 tps: 150.90 qps: 3042.11 (r/w/o: 2130.20/548.10/363.80) lat (ms,95%): 176.73 err/s: 0.90 reconn/s: 0.00[ 470s ] thds: 16 tps: 213.90 qps: 4317.46 (r/w/o: 3023.97/770.29/523.19) lat (ms,95%): 110.66 err/s: 2.10 reconn/s: 0.00[ 480s ] thds: 16 tps: 214.70 qps: 4333.44 (r/w/o: 3035.23/782.41/515.81) lat (ms,95%): 112.67 err/s: 2.10 reconn/s: 0.00[ 490s ] thds: 16 tps: 153.00 qps: 3097.51 (r/w/o: 2170.91/556.10/370.50) lat (ms,95%): 139.85 err/s: 2.20 reconn/s: 0.00[ 500s ] thds: 16 tps: 217.90 qps: 4402.00 (r/w/o: 3081.83/789.88/530.29) lat (ms,95%): 112.67 err/s: 2.10 reconn/s: 0.00[ 510s ] thds: 16 tps: 215.70 qps: 4352.16 (r/w/o: 3047.84/783.71/520.61) lat (ms,95%): 110.66 err/s: 2.00 reconn/s: 0.00[ 520s ] thds: 16 tps: 111.50 qps: 2251.40 (r/w/o: 1576.40/405.70/269.30) lat (ms,95%): 746.32 err/s: 1.10 reconn/s: 0.00[ 530s ] thds: 16 tps: 226.30 qps: 4571.32 (r/w/o: 3201.81/825.60/543.90) lat (ms,95%): 110.66 err/s: 2.40 reconn/s: 0.00[ 540s ] thds: 16 tps: 217.70 qps: 4399.07 (r/w/o: 3081.18/787.80/530.10) lat (ms,95%): 108.68 err/s: 2.50 reconn/s: 0.00[ 550s ] thds: 16 tps: 216.20 qps: 4348.31 (r/w/o: 3044.71/781.20/522.40) lat (ms,95%): 112.67 err/s: 1.30 reconn/s: 0.00[ 560s ] thds: 16 tps: 172.40 qps: 3482.80 (r/w/o: 2439.30/625.40/418.10) lat (ms,95%): 123.28 err/s: 1.70 reconn/s: 0.00[ 570s ] thds: 16 tps: 198.20 qps: 3995.65 (r/w/o: 2798.26/723.19/474.19) lat (ms,95%): 116.80 err/s: 1.70 reconn/s: 0.00[ 580s ] thds: 16 tps: 216.20 qps: 4366.73 (r/w/o: 3057.92/785.11/523.70) lat (ms,95%): 116.80 err/s: 2.30 reconn/s: 0.00[ 590s ] thds: 16 tps: 79.20 qps: 1597.21 (r/w/o: 1118.61/285.90/192.70) lat (ms,95%): 802.05 err/s: 0.60 reconn/s: 0.00[ 600s ] thds: 16 tps: 144.50 qps: 2919.85 (r/w/o: 2045.27/528.59/345.99) lat (ms,95%): 158.63 err/s: 1.60 reconn/s: 0.00SQL statistics:queries performed:read: 1579018write: 393810other: 281802total: 2254630transactions: 111710 (186.16 per sec.)queries: 2254630 (3757.31 per sec.)ignored errors: 1077 (1.79 per sec.)reconnects: 0 (0.00 per sec.)General statistics:total time: 600.0633stotal number of events: 111710Latency (ms):min: 10.60avg: 85.94max: 2669.5895th percentile: 127.81sum: 9600035.41Threads fairness:events (avg/stddev): 6981.8750/10.97execution time (avg/stddev): 600.0022/0.02jan@galapq:~$ sysbench /usr/share/sysbench/oltp_read_write.lua --threads=16 --report-interval=10 --time=600 --mysql-socket=/home/jan/10.6-bug/mysql-test/var/tmp/mysqld.2.sock --table-size=1000000 --mysql-user=root runsysbench 1.0.18 (using system LuaJIT 2.1.0-beta3)Running the test with following options:Number of threads: 16Report intermediate results every 10 second(s)Initializing random number generator from current timeInitializing worker threads...Threads started![ 10s ] thds: 16 tps: 61.28 qps: 1261.78 (r/w/o: 884.57/134.47/242.74) lat (ms,95%): 1089.30 err/s: 0.30 reconn/s: 0.00[ 20s ] thds: 16 tps: 122.70 qps: 2467.49 (r/w/o: 1727.66/293.61/446.22) lat (ms,95%): 155.80 err/s: 0.80 reconn/s: 0.00[ 30s ] thds: 16 tps: 214.10 qps: 4327.23 (r/w/o: 3031.02/586.90/709.30) lat (ms,95%): 116.80 err/s: 2.30 reconn/s: 0.00[ 40s ] thds: 16 tps: 230.60 qps: 4652.11 (r/w/o: 3257.81/701.90/692.40) lat (ms,95%): 106.75 err/s: 2.10 reconn/s: 0.00[ 50s ] thds: 16 tps: 235.50 qps: 4743.30 (r/w/o: 3321.90/754.40/667.00) lat (ms,95%): 104.84 err/s: 1.80 reconn/s: 0.00[ 60s ] thds: 16 tps: 169.40 qps: 3415.32 (r/w/o: 2391.45/571.89/451.99) lat (ms,95%): 125.52 err/s: 1.40 reconn/s: 0.00[ 70s ] thds: 16 tps: 133.50 qps: 2690.63 (r/w/o: 1884.82/446.51/359.30) lat (ms,95%): 484.44 err/s: 1.20 reconn/s: 0.00[ 80s ] thds: 16 tps: 224.10 qps: 4529.75 (r/w/o: 3171.97/775.39/582.39) lat (ms,95%): 112.67 err/s: 2.40 reconn/s: 0.00[ 90s ] thds: 16 tps: 217.80 qps: 4395.74 (r/w/o: 3079.92/764.81/551.00) lat (ms,95%): 108.68 err/s: 2.30 reconn/s: 0.00[ 100s ] thds: 16 tps: 216.80 qps: 4391.34 (r/w/o: 3074.53/772.11/544.70) lat (ms,95%): 108.68 err/s: 2.70 reconn/s: 0.00[ 110s ] thds: 16 tps: 167.30 qps: 3389.32 (r/w/o: 2374.32/593.70/421.30) lat (ms,95%): 158.63 err/s: 2.40 reconn/s: 0.00[ 120s ] thds: 16 tps: 245.10 qps: 4940.29 (r/w/o: 3459.80/867.00/613.50) lat (ms,95%): 112.67 err/s: 2.10 reconn/s: 0.00[ 130s ] thds: 16 tps: 221.50 qps: 4480.54 (r/w/o: 3138.26/786.99/555.29) lat (ms,95%): 104.84 err/s: 2.50 reconn/s: 0.00[ 140s ] thds: 16 tps: 210.80 qps: 4252.35 (r/w/o: 2977.93/750.61/523.81) lat (ms,95%): 116.80 err/s: 2.00 reconn/s: 0.00[ 150s ] thds: 16 tps: 212.90 qps: 4305.09 (r/w/o: 3015.30/761.60/528.20) lat (ms,95%): 112.67 err/s: 2.40 reconn/s: 0.00[ 160s ] thds: 16 tps: 173.00 qps: 3493.10 (r/w/o: 2446.20/616.10/430.80) lat (ms,95%): 132.49 err/s: 1.70 reconn/s: 0.00[ 170s ] thds: 16 tps: 213.90 qps: 4315.53 (r/w/o: 3022.55/766.69/526.29) lat (ms,95%): 112.67 err/s: 2.00 reconn/s: 0.00[ 180s ] thds: 16 tps: 224.00 qps: 4520.31 (r/w/o: 3165.41/801.00/553.90) lat (ms,95%): 108.68 err/s: 2.10 reconn/s: 0.00[ 190s ] thds: 16 tps: 220.10 qps: 4467.58 (r/w/o: 3130.26/794.91/542.41) lat (ms,95%): 110.66 err/s: 3.60 reconn/s: 0.00[ 200s ] thds: 16 tps: 227.30 qps: 4603.12 (r/w/o: 3224.34/820.09/558.69) lat (ms,95%): 106.75 err/s: 2.90 reconn/s: 0.00[ 210s ] thds: 16 tps: 74.30 qps: 1493.12 (r/w/o: 1045.31/263.60/184.20) lat (ms,95%): 926.33 err/s: 0.50 reconn/s: 0.00[ 220s ] thds: 16 tps: 161.70 qps: 3254.92 (r/w/o: 2279.31/572.20/403.40) lat (ms,95%): 139.85 err/s: 1.00 reconn/s: 0.00[ 230s ] thds: 16 tps: 217.90 qps: 4396.67 (r/w/o: 3078.88/787.09/530.70) lat (ms,95%): 108.68 err/s: 2.00 reconn/s: 0.00[ 240s ] thds: 16 tps: 202.90 qps: 4090.01 (r/w/o: 2864.21/725.80/500.00) lat (ms,95%): 121.08 err/s: 1.70 reconn/s: 0.00[ 250s ] thds: 16 tps: 211.90 qps: 4288.32 (r/w/o: 3003.74/760.19/524.39) lat (ms,95%): 116.80 err/s: 3.00 reconn/s: 0.00[ 260s ] thds: 16 tps: 113.30 qps: 2292.33 (r/w/o: 1605.52/409.30/277.50) lat (ms,95%): 539.71 err/s: 1.10 reconn/s: 0.00[ 270s ] thds: 16 tps: 196.50 qps: 3960.92 (r/w/o: 2774.02/705.70/481.20) lat (ms,95%): 116.80 err/s: 1.90 reconn/s: 0.00[ 280s ] thds: 16 tps: 226.00 qps: 4561.43 (r/w/o: 3194.85/814.39/552.19) lat (ms,95%): 104.84 err/s: 2.00 reconn/s: 0.00[ 290s ] thds: 16 tps: 222.40 qps: 4488.30 (r/w/o: 3143.30/799.60/545.40) lat (ms,95%): 110.66 err/s: 2.00 reconn/s: 0.00[ 300s ] thds: 16 tps: 229.70 qps: 4634.10 (r/w/o: 3245.97/827.52/560.61) lat (ms,95%): 112.67 err/s: 2.20 reconn/s: 0.00[ 310s ] thds: 16 tps: 195.90 qps: 3960.36 (r/w/o: 2772.87/710.19/477.30) lat (ms,95%): 132.49 err/s: 2.10 reconn/s: 0.00[ 320s ] thds: 16 tps: 150.80 qps: 3065.33 (r/w/o: 2147.62/543.91/373.80) lat (ms,95%): 161.51 err/s: 2.60 reconn/s: 0.00[ 330s ] thds: 16 tps: 199.20 qps: 4017.50 (r/w/o: 2813.70/716.10/487.70) lat (ms,95%): 125.52 err/s: 2.00 reconn/s: 0.00[ 340s ] thds: 16 tps: 94.70 qps: 1910.09 (r/w/o: 1337.29/340.00/232.80) lat (ms,95%): 669.89 err/s: 0.60 reconn/s: 0.00[ 350s ] thds: 16 tps: 101.20 qps: 2039.21 (r/w/o: 1428.01/362.30/248.90) lat (ms,95%): 257.95 err/s: 0.80 reconn/s: 0.00[ 360s ] thds: 16 tps: 147.00 qps: 2951.19 (r/w/o: 2066.80/527.60/356.80) lat (ms,95%): 176.73 err/s: 0.80 reconn/s: 0.00[ 370s ] thds: 16 tps: 199.20 qps: 4033.56 (r/w/o: 2824.77/719.29/489.50) lat (ms,95%): 125.52 err/s: 2.40 reconn/s: 0.00[ 380s ] thds: 16 tps: 210.20 qps: 4257.22 (r/w/o: 2982.01/760.50/514.70) lat (ms,95%): 114.72 err/s: 2.80 reconn/s: 0.00[ 390s ] thds: 16 tps: 206.80 qps: 4160.53 (r/w/o: 2913.42/740.90/506.20) lat (ms,95%): 112.67 err/s: 1.30 reconn/s: 0.00[ 400s ] thds: 16 tps: 213.00 qps: 4297.16 (r/w/o: 3009.97/761.49/525.70) lat (ms,95%): 116.80 err/s: 2.10 reconn/s: 0.00[ 410s ] thds: 16 tps: 225.40 qps: 4548.48 (r/w/o: 3184.98/809.80/553.70) lat (ms,95%): 106.75 err/s: 2.00 reconn/s: 0.00[ 420s ] thds: 16 tps: 219.50 qps: 4416.85 (r/w/o: 3093.24/793.21/530.41) lat (ms,95%): 110.66 err/s: 1.60 reconn/s: 0.00[ 430s ] thds: 16 tps: 209.50 qps: 4221.42 (r/w/o: 2956.01/754.40/511.00) lat (ms,95%): 106.75 err/s: 1.50 reconn/s: 0.00[ 440s ] thds: 16 tps: 226.70 qps: 4568.79 (r/w/o: 3199.19/819.50/550.10) lat (ms,95%): 104.84 err/s: 1.80 reconn/s: 0.00[ 450s ] thds: 16 tps: 164.20 qps: 3310.29 (r/w/o: 2318.39/592.60/399.30) lat (ms,95%): 132.49 err/s: 1.40 reconn/s: 0.00[ 460s ] thds: 16 tps: 205.30 qps: 4148.12 (r/w/o: 2905.01/741.00/502.10) lat (ms,95%): 121.08 err/s: 2.20 reconn/s: 0.00[ 470s ] thds: 16 tps: 215.00 qps: 4340.37 (r/w/o: 3040.28/775.50/524.60) lat (ms,95%): 110.66 err/s: 2.20 reconn/s: 0.00[ 480s ] thds: 16 tps: 217.10 qps: 4382.92 (r/w/o: 3069.31/787.50/526.10) lat (ms,95%): 116.80 err/s: 2.10 reconn/s: 0.00[ 490s ] thds: 16 tps: 158.20 qps: 3187.79 (r/w/o: 2232.70/571.40/383.70) lat (ms,95%): 475.79 err/s: 1.30 reconn/s: 0.00[ 500s ] thds: 16 tps: 219.40 qps: 4439.09 (r/w/o: 3109.39/797.50/532.20) lat (ms,95%): 110.66 err/s: 2.70 reconn/s: 0.00[ 510s ] thds: 16 tps: 220.60 qps: 4452.73 (r/w/o: 3118.12/798.61/536.00) lat (ms,95%): 106.75 err/s: 2.10 reconn/s: 0.00[ 520s ] thds: 16 tps: 95.20 qps: 1925.59 (r/w/o: 1349.19/343.00/233.40) lat (ms,95%): 733.00 err/s: 1.20 reconn/s: 0.00[ 530s ] thds: 16 tps: 221.10 qps: 4458.76 (r/w/o: 3122.27/802.19/534.30) lat (ms,95%): 110.66 err/s: 1.90 reconn/s: 0.00[ 540s ] thds: 16 tps: 223.70 qps: 4510.48 (r/w/o: 3158.49/808.60/543.40) lat (ms,95%): 112.67 err/s: 2.00 reconn/s: 0.00[ 550s ] thds: 16 tps: 224.70 qps: 4540.18 (r/w/o: 3180.26/811.61/548.31) lat (ms,95%): 108.68 err/s: 2.40 reconn/s: 0.00[ 560s ] thds: 16 tps: 159.60 qps: 3221.48 (r/w/o: 2255.99/578.60/386.90) lat (ms,95%): 127.81 err/s: 1.50 reconn/s: 0.00[ 570s ] thds: 16 tps: 206.90 qps: 4181.62 (r/w/o: 2928.74/752.09/500.79) lat (ms,95%): 112.67 err/s: 2.30 reconn/s: 0.00[ 580s ] thds: 16 tps: 149.90 qps: 3024.66 (r/w/o: 2118.25/546.51/359.91) lat (ms,95%): 227.40 err/s: 1.40 reconn/s: 0.00[ 590s ] thds: 16 tps: 125.80 qps: 2552.10 (r/w/o: 1787.80/458.90/305.40) lat (ms,95%): 539.71 err/s: 1.90 reconn/s: 0.00[ 600s ] thds: 16 tps: 239.10 qps: 4799.83 (r/w/o: 3361.25/865.49/573.09) lat (ms,95%): 110.66 err/s: 1.00 reconn/s: 0.00SQL statistics:queries performed:read: 1618008write: 404173other: 288107total: 2310288transactions: 114448 (190.73 per sec.)queries: 2310288 (3850.17 per sec.)ignored errors: 1124 (1.87 per sec.)reconnects: 0 (0.00 per sec.)General statistics:total time: 600.0463stotal number of events: 114448Latency (ms):min: 6.96avg: 83.88max: 3259.0195th percentile: 125.52sum: 9599981.11Threads fairness:events (avg/stddev): 7153.0000/13.83execution time (avg/stddev): 599.9988/0.01