Hetnzer EX42-NVMe bare-metal servers, Ubuntu 18.04.3 LTS, docker 19.03.4~3-0~ubuntu-bionic

after 24 hours

crocodylus Do you mean by both are getting the same amount of traffic (load balancing), and the memory consumption differs more than twice and is slowly growing that the memory consumption of one of the instances is twice that of the other, or that the memory consumption of both has doubled since they were started?

I have added fresh screenshots. For the first instance the memory consumption is 77Mb->95Mb->113Mb, for the second one is 135Mb-152Mb->170Mb.

From the MaxScale log, what does the line

2019-11-27 12:50:20   notice : Using up to ... of memory for query classifier cache

say? Note that the amount is only what could be used, not what is used.

Please execute

maxctrl show threads

What kind of numbers do you have for QC cache size?

first node:

2019-11-19 22:25:24 notice : Using up to 9.40GiB of memory for query classifier cache
2019-11-19 22:25:24 notice : Service cache: /var/cache/maxscale
2019-11-19 22:25:24 notice : Query classification results are cached and reused. Memory used per thread: 2.35GiB

Id 0
Accepts 1645034
Reads 112712855
Writes 119046802
Hangups 10840
Errors 8
Avg event queue length 1
Max event queue length 14
Max exec time 381
Max queue time 180
Current FDs 3
Total FDs 6560407
Load (1s) 2
Load (1m) 1
Load (1h) 1
QC cache size 20172239
QC cache inserts 18634
QC cache hits 32724723
QC cache misses 18634
QC cache evictions 0

Id 1
Accepts 1614514
Reads 111084909
Writes 117876932
Hangups 0
Errors 9
Avg event queue length 1
Max event queue length 15
Max exec time 180
Max queue time 130
Current FDs 3
Total FDs 6425083
Load (1s) 1
Load (1m) 1
Load (1h) 1
QC cache size 19840836
QC cache inserts 18380
QC cache hits 32324991
QC cache misses 18380
QC cache evictions 0

Id 2
Accepts 1601042
Reads 110987546
Writes 117032490
Hangups 1
Errors 4
Avg event queue length 1
Max event queue length 17
Max exec time 286
Max queue time 13
Current FDs 3
Total FDs 6370876
Load (1s) 2
Load (1m) 2
Load (1h) 1
QC cache size 19996137
QC cache inserts 18246
QC cache hits 32546346
QC cache misses 18246
QC cache evictions 0

Id 3
Accepts 1626569
Reads 112543957
Writes 118878595
Hangups 0
Errors 5
Avg event queue length 1
Max event queue length 14
Max exec time 435
Max queue time 50
Current FDs 11
Total FDs 6473494
Load (1s) 0
Load (1m) 1
Load (1h) 1
QC cache size 19888425
QC cache inserts 18598
QC cache hits 32914698
QC cache misses 18598
QC cache evictions 0

second node:

2019-11-19 22:25:24 notice : Using up to 9.41GiB of memory for query classifier cache
2019-11-19 22:25:24 notice : Service cache: /var/cache/maxscale
2019-11-19 22:25:24 notice : Query classification results are cached and reused. Memory used per thread: 2.35GiB

Id 0
Accepts 1834286
Reads 130179539
Writes 137911341
Hangups 10849
Errors 9
Avg event queue length 1
Max event queue length 12
Max exec time 410
Max queue time 51
Current FDs 5
Total FDs 7314805
Load (1s) 0
Load (1m) 0
Load (1h) 0
QC cache size 31550901
QC cache inserts 26273
QC cache hits 38074574
QC cache misses 26273
QC cache evictions 0

Id 1
Accepts 1850676
Reads 130731761
Writes 138528883
Hangups 1
Errors 3
Avg event queue length 1
Max event queue length 10
Max exec time 280
Max queue time 1
Current FDs 3
Total FDs 7367657
Load (1s) 1
Load (1m) 0
Load (1h) 0
QC cache size 31802936
QC cache inserts 26282
QC cache hits 38085342
QC cache misses 26282
QC cache evictions 0

Id 2
Accepts 1771115
Reads 124758208
Writes 132208587
Hangups 0
Errors 3
Avg event queue length 1
Max event queue length 10
Max exec time 410
Max queue time 3
Current FDs 7
Total FDs 7048079
Load (1s) 1
Load (1m) 0
Load (1h) 0
QC cache size 31093934
QC cache inserts 26001
QC cache hits 36369674
QC cache misses 26001
QC cache evictions 0

Id 3
Accepts 1826410
Reads 127804303
Writes 135496347
Hangups 2
Errors 4
Avg event queue length 1
Max event queue length 11
Max exec time 380
Max queue time 70
Current FDs 3
Total FDs 7269271
Load (1s) 3
Load (1m) 0
Load (1h) 0
QC cache size 31154303
QC cache inserts 26463
QC cache hits 36994154
QC cache misses 26463
QC cache evictions 0

Ok, so it seems that in the case of the first node, the Query Classifier Cache uses in total (across all threads) some 60MB of memory and in the case of the second node, some 120MB of memory.

The number of unique canonical statements differ also significantly; in the case of the first node there are some 18-thousand and in the case of the second node there are some 25-thousand.

For instance, the unique statements

SELECT * FROM t WHERE f > 3;

SELECT * FROM t WHERE f > 4;

SELECT * FROM t WHERE f > 5;

correspond all to the one canonical statement

SELECT * FROM t WHERE f > ?;

You can put a cap on the amount of memory used for the query classifier cache as explained here: https://mariadb.com/kb/en/mariadb-maxscale-24-mariadb-maxscale-configuration-guide/#query_classifier_cache_size

If the memory growth doesn't stop even when the query_classifier_cache_size is set to a lower value (e.g. 100MB), please let us know.

Closing as Cannot Reproduce.