[MDEV-4682] QUERY CACHE - add STATISTICS per query and show this informations in qc_info plugin Created: 2013-06-19  Updated: 2015-12-07  Resolved: 2015-09-11

Status: Closed
Project: MariaDB Server
Component/s: Plugins, Query Cache
Fix Version/s: 10.1.8

Type: Task Priority: Major
Reporter: roberto spadim Assignee: Sergei Golubchik
Resolution: Fixed Votes: 2
Labels: plugins, qc_info, querycache

Issue Links:
Blocks
is blocked by MDEV-6781 bug with query cache when using views Closed
Duplicate
duplicates MDEV-4571 CLOSE - Query cache like features Closed
duplicates MDEV-4589 CLOSE - query cache expent time searc... Closed
PartOf
includes MDEV-4581 QUERY CACHE - ADD more columns in QC_... Closed
Relates
relates to MDEV-4454 Query Cache Partitioning Open
relates to MDEV-6777 Query cache with inconsistent results... Open
relates to MDEV-6778 Include number of concurrent queries ... Open
Sprint: 10.1.8-1

 Description   

Change sql_cache.cc and sql_cache.h to add hit rates and statistics
usable with qc_info plugin

Statistics:
total hits / total time expent in query cache
hits in three categories:
1) low period category: period between the last hit and the current hit < query expent time
2) high period category: period < query expent time * 10 and > query expent time
3) outlier category: period > 1 second, and period > query expent time * 10
3.1) in a future version instead of * 10, we could use number of competitors (concurrent) when inserting this query at query cache
for example if we have 9 concurrent inserts of the same query at query cache this number *10 is an acceptable value for high period, if we have n concurrent inserts a (n+1) value is acceptable

These categories helps to understand what kind of query we have cached:
1) low period hits: in this case if we lose this query entry, we will have a performace problem, and users will tell "the server is slow now"
2) high period hits: we have a query that is executed with a good frequency, removing it we will have a higher i/o in disks
3) outlier period hits: this query isn't a query that we should cache, or we can loose it and we don't see users reporting "hey the server is slow, or the disks is spinning a lot"

this allow a better query cache control (MDEV-4584) and a better cache hit rate when optimizing queries with SQL_CACHE/SQL_NO_CACHE


a second part could be an variable (query_cache_stats=0/1) to turn off/on the statistics and use a block memory just to statistics instead of one big block sharing memory with queries flags

in this case:
turn on -> off, should clean this stats memory block
turn off -> on, should create stats with 0 value (maybe a problem with lock contention), or clean all query cache and start from zero

changing query cache queries (invalidate, table invalidate, reallocation, etc) should handle the stats block too

tables of this plugin:

query_cache_queries:

QUERY_CACHE_ID STATEMENT_SCHEMA STATEMENT_TEXT RESULT_FOUND_ROWS QUERY_ROWS SELECT_ROWS_READ QUERY_HITS QUERY_HITS_PERIOD_LOW QUERY_HITS_PERIOD_HIGH QUERY_HITS_PERIOD_OUTLIERS QUERY_HITS_TOTAL_TIME_US QUERY_HITS_MEAN_PERIOD_US QUERY_HITS_MEAN_PERIOD_LOW_US QUERY_HITS_MEAN_PERIOD_HIGH_US QUERY_INSERT_TIME QUERY_LAST_HIT_TIME SELECT_EXPEND_TIME_US SELECT_LOCK_TIME_US TABLES_TYPE RESULT_LENGTH RESULT_BLOCKS_COUNT RESULT_BLOCKS_SIZE RESULT_BLOCKS_SIZE_USED FLAGS_CLIENT_LONG_FLAG FLAGS_CLIENT_PROTOCOL_41 FLAGS_PROTOCOL_TYPE FLAGS_MORE_RESULTS_EXISTS FLAGS_IN_TRANS FLAGS_AUTOCOMMIT FLAGS_PKT_NR FLAGS_CHARACTER_SET_CLIENT FLAGS_CHARACTER_SET_RESULTS FLAGS_COLLATION_CONNECTION FLAGS_LIMIT FLAGS_TIME_ZONE FLAGS_SQL_MODE FLAGS_MAX_SORT_LENGTH FLAGS_GROUP_CONCAT_MAX_LEN FLAGS_DIV_PRECISION_INCREMENT FLAGS_DEFAULT_WEEK_FORMAT FLAGS_LC_TIME_NAMES
1 teste select * from t1 3 3 3 0 0 0 0 0       1411247500.713750   232 35 NON TRANSACT 78 1 512 142 1 1 0 0 0 1 1 utf8 utf8 utf8_general_ci -1 SYSTEM PIPES_AS_CONCAT,ALLOW_INVALID_DATES 1024 1024 5 0 en_US

query_cache_queries_tables:

QUERY_CACHE_ID SCHEMA TABLE_NAME TABLE_SUFFIX
1 teste t1 (here we display partition name like P#p0)

query_cache_tables

TABLE_SCHEMA TABLE_NAME TABLE_HASHED TABLE_TYPE QUERIES_IN_CACHE
teste t1 1 NON_TRANSACT 1


 Comments   
Comment by roberto spadim [ 2013-06-19 ]

should use qc_plugin with MDEV-4682 support
this diff_10_0_3_qc_info.cc is the same sent to MDEV_4581 (with statistics support)

Comment by roberto spadim [ 2013-06-19 ]

use the same diff_10_0_3_qc_info.cc

Comment by roberto spadim [ 2013-07-07 ]

sergey, should i put some #ifdef, and make statistic a option when compiling the source?

Comment by roberto spadim [ 2014-06-16 ]

hi guys, now i'm with time to code a bit more, could we consider a review of this MDEV?
i'm using the last code near to one year, and it's running quite ok at production server (ok i shouldn't use 10.0 at production that time, but it's running nice)

about informations, the statistic data, is nice to me, i could search queries that should be SQL_NO_CACHE and queries that could be SQL_CACHE very easy, and check if cache is doing a good job

thanks guys

Comment by roberto spadim [ 2014-07-16 ]

hi, guys, i was thinking about using audit plugin to collect statistic about query cache, could it work? i'm confuse about query cache invalidation, if audit plugin could solve this problem or not, any idea if i will have problems? the 'bad' part of this today implementation patch, is the change of sql_cache.cc and sql_cache.h source code, it change structures of cache code at sql_cache.h

Comment by roberto spadim [ 2014-09-20 ]

sending patchs via pull request at github

Comment by roberto spadim [ 2014-09-20 ]

there's one TODO, use or not use statistics with query cache, since it increase memory use should be nice turn it off/on, at least at compile time, but should be nice if it could be done at runtime, for example, turning statistics on/off release all memory and queries and restart query cache again with a new 'table' (a block of memory) of statistics if it's on, if it's off don't use this 'table'

Comment by Oleksandr Byelkin [ 2014-09-30 ]

Before using try_lock() it is better to check of is_disabled() to avoid delay in case of changing QC state.

Comment by roberto spadim [ 2014-09-30 ]

nice i will include it now

Comment by roberto spadim [ 2014-09-30 ]

updated: https://github.com/rspadim/server/commit/7bca392c82959b7e211822f988b95832f3e2d96a

qc->is_disabled() before all try_lock(thd)

Comment by roberto spadim [ 2014-10-13 ]

i think we are ok to review
i'm using in production without problems, at 10.0.13 and 10.1 (github) both tested, but 10.0.13 i didn't have bazaar, i just commited to github

must check if i done the right work with test files

one doubt is... query_id should be a number from 1......infinte, or there's a hash table value like a primary key or something like it?

Comment by Daniel Black [ 2015-03-19 ]

Looks good. Seems documentation is ahead of code a bit this time - https://mariadb.com/kb/en/mariadb/query_cache_info-plugin/ probably needs a little more work to describe which tables are in which versions.

Comment by Oleksandr Byelkin [ 2015-03-22 ]

It should be mentioned that using this plugin (during fetching data from information schema) lock QC. So one who use it should be aware of it and do not make DoS on his server by frequent requests.

Comment by Daniel Black [ 2015-05-17 ]

https://github.com/MariaDB/server/pull/67/files is now the basic Query Cache Info plugin with decoding of query cache flags and test cases that show most of the query cache flags changed by server variables.

Hoping it is suitable for 10.0 as well as 10.1

Edit: was incomplete. Now isn't.

Comment by roberto spadim [ 2015-12-07 ]

nice, now just need some statistics, flags are ok

Generated at Thu Feb 08 06:58:20 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.