Sergey Vojtovich added a comment - 2018-05-18 17:11 You're right, we need to concentrate on original report. Which I couldn't reproduce locally, most probably due to some known issues with Samsung SSD being slow on heavy fsync. I will try to look into ramdisk performance more closely, with longer run and more concurrent connections (like 1000).

Marko Mäkelä added a comment - 2018-05-21 07:08 I agree that we must address the I/O issue, but at the same time it could be worth looking for CPU regressions, that is, approach repeatable regressions from multiple angles. For what it is worth, I repeated some 180-second ramdisk benchmarks today: revision threads=40 6c279ad6a71c63cb595fde7c951aadb31c3dbebc 130213.08tps 6c279ad6a71c63cb595fde7c951aadb31c3dbebc (patch 1) 134518.90tps 7ad9b3588f53672b3d2d9f345ec9ab7335b50206 104287.79tps 7ad9b3588f53672b3d2d9f345ec9ab7335b50206 (patch 2) 112871.50tps 7ad9b3588f53672b3d2d9f345ec9ab7335b50206 (patch 1m+2) 109021.72tps 7ad9b3588f53672b3d2d9f345ec9ab7335b50206 (patch 1+2) 114488.47tps The baseline was 123927.89tps before the weekend, and now it is 130213.08tps, so there indeed is some turbulence in the figures even after I switched to always do a full rebuild. Patch 1 reverts some struct  or class  member variables from size_t to a narrower type, such as uint  or uint32 . A minimal version of the patch (1m) is below: diff --git a/sql/sql_cache.h b/sql/sql_cache.h index 0ed45a9ed81..ca2c3a59ee7 100644 --- a/sql/sql_cache.h +++ b/sql/sql_cache.h @@ -350,7 +350,7 @@ class Query_cache size_t min_allocation_unit, min_result_data_size; uint def_query_hash_size, def_table_hash_size; - size_t mem_bin_num, mem_bin_steps; // See at init_cache & find_bin + uint mem_bin_num, mem_bin_steps; // See at init_cache & find_bin bool initialized; In my test resuls above, this minimal patch appeared to cause a slight performance regression, but I think that we can explain it by saying that it falls within the 5% or 10% of fluctuation. Patch 2 removes st_mem_root::name and the name  parameter of init_alloc_root() : diff --git a/include/my_alloc.h b/include/my_alloc.h index 3ba8a02fc48..c9879125d96 100644 --- a/include/my_alloc.h +++ b/include/my_alloc.h @@ -52,7 +52,9 @@ typedef struct st_mem_root unsigned int first_block_usage; void (*error_handler)(void); +#ifndef DBUG_OFF const char *name; +#endif } MEM_ROOT; #ifdef __cplusplus diff --git a/include/my_sys.h b/include/my_sys.h index 13ab7b12320..fb18cd92e8a 100644 --- a/include/my_sys.h +++ b/include/my_sys.h @@ -888,9 +888,16 @@ extern void my_free_lock(void *ptr); #define alloc_root_inited(A) ((A)->min_malloc != 0) #define ALLOC_ROOT_MIN_BLOCK_SIZE (MALLOC_OVERHEAD + sizeof(USED_MEM) + 8) #define clear_alloc_root(A) do { (A)->free= (A)->used= (A)->pre_alloc= 0; (A)->min_malloc=0;} while(0) +#ifndef DBUG_OFF extern void init_alloc_root(MEM_ROOT *mem_root, const char *name, size_t block_size, size_t pre_alloc_size, myf my_flags); +#else +extern void init_alloc_Root(MEM_ROOT *mem_root, + size_t block_size, size_t pre_alloc_size, + myf my_flags); +# define init_alloc_root(m,n,b,p,f) init_alloc_Root(m,b,p,f) +#endif extern void *alloc_root(MEM_ROOT *mem_root, size_t Size); extern void *multi_alloc_root(MEM_ROOT *mem_root, ...); extern void free_root(MEM_ROOT *root, myf MyFLAGS); diff --git a/mysys/my_alloc.c b/mysys/my_alloc.c index 7139466be17..fddc9118312 100644 --- a/mysys/my_alloc.c +++ b/mysys/my_alloc.c @@ -54,9 +54,15 @@ Because of this, we store in MY_THREAD_SPECIFIC as bit 1 in block_size */ +#ifndef DBUG_OFF void init_alloc_root(MEM_ROOT *mem_root, const char *name, size_t block_size, size_t pre_alloc_size __attribute__((unused)), myf my_flags) +#else +void init_alloc_Root(MEM_ROOT *mem_root, size_t block_size, + size_t pre_alloc_size __attribute__((unused)), + myf my_flags) +#endif { DBUG_ENTER("init_alloc_root"); DBUG_PRINT("enter",("root: %p name: %s prealloc: %zu", mem_root, @@ -72,7 +78,9 @@ void init_alloc_root(MEM_ROOT *mem_root, const char *name, size_t block_size, mem_root->block_num= 4; /* We shift this with >>2 */ mem_root->first_block_usage= 0; mem_root->total_alloc= 0; +#ifndef DBUG_OFF mem_root->name= name; +#endif #if !(defined(HAVE_valgrind) && defined(EXTRA_DEBUG)) if (pre_alloc_size) I would suggest to include this patch in 10.3 in some form. With GCC 7.3.0 and the 3-minute CPU-bound ramdisk benchmark, we still appear to have a significant regression somewhere between 10.3.4 and 10.3.5. I will keep trying to trace it down while Svoj can concentrate on the originally reported regression.

Marko Mäkelä added a comment - 2018-05-21 13:10 I can still repeat the huge drop caused by commit 6c279ad6a71c63cb595fde7c951aadb31c3dbebc: dropping from 207119.69tps to 130029.74tps. I narrowed it down to the sql_class.h and sql_class.cc changes in that commit, but when I reverted that part of the patch in the latest 10.3, I did not see any improvement. In my benchmark, I would see contention in lock_sys.mutex and log_sys.mutex . The situation should be improved by MDEV-16232 and MDEV-14425 . Also, at low concurrency, it seems that purge is wasting resources and creating unnecessary contention. Perhaps the amount of purge threads should be automatically scaled based on the amount of purgeable records?

Marko Mäkelä added a comment - 2018-08-24 12:24 A bad merge of MDEV-14705 to MariaDB 10.3.6 causes frequent calls to service_manager_extend_timeout() from all purge threads. These calls were supposed to only occur during a slow shutdown. This will be fixed in MDEV-17003 . Given that the performance regression reported in this ticket was reported on Windows, where the systemd interface is not present, this probably does not explain much of the regression. Also, I probably did not have libsystemd-dev installed when compiling code for running benchmarks.

Vladislav Vaintroub added a comment - 2020-11-17 13:34 By now, we are all educated that MDEV-12288 caused that issue. It just came not really announced back then