[MDEV-573] mysqlreport updated version that adds AriaDB pagecache stats, binlog cache stats, subquery cache stats, TokuDB stats and bugfixes Created: 2012-10-04  Updated: 2016-02-17  Resolved: 2016-02-17

Status: Closed
Project: MariaDB Server
Component/s: Scripts & Clients
Fix Version/s: 10.1.12

Type: Task Priority: Minor
Reporter: Jean Weisbuch Assignee: Sergey Vojtovich
Resolution: Fixed Votes: 0
Labels: mysqlreport


 Description   

The latest revision can be found on : https://github.com/jb-boin/mariadb/blob/5.5.30/debian/additions/mysqlreport

Read the comments to follow the new revisions additions (such as subquery cache, binary log cache, TokuDB stats, returned rows stats, threadpool usage stats and many small fixes/improvements).


I've made a patch for mysqlreport so it can report actual AriaDB page cache stats like the key buffer cache usage for MyISAM.

Patch is pretty trivial and based on how mysqlreport reports the MyISAM key cache informations.
I would strongly advise to verify if the returned values are correct and if possible, implement a better verification for the presence of AriaDB than checking the version string for "-MariaDB-".

Tested on MySQL 5.1 (no AriaDB infos shown) and on Maria 5.5.27 (shows informations without any error).

Here is an example of the produced output :

__ Aria Pagecache ______________________________________________________
Buffer used 124.95M of 128.00M %Used: 97.61
Current 128.00M %Usage: 100.00
Write hit 53.16%
Read hit 86.87%

ps: there is a warning when launching mysqlreport :

Use of uninitialized value in formline at mysqlreport line 1300

Its an upstream bug (same error encountered on Debian 6 with non MariaDB mysqlreport and MySQL 5.1) as mysqlreport reads "Innodb_buffer_pool_read_ahead_seq" value but it seems that this value has been replaced by "Innodb_buffer_pool_read_ahead" on MySQL/MariaDB for a long time, correcting this could also be a good idea.



 Comments   
Comment by Jean Weisbuch [ 2012-10-09 ]

This is an updated version of the patch which now verify that the ariadb engine is present and active rather than checking that the server version string contains MariaDB.

The "Innodb_buffer_pool_read_ahead value isnt shown anymore to avoid the warning (variable was removed on innodb plugin on 5.1 and not present on 5.5).

Submitting this patch "upstream" could also be a good idea.

Comment by Jean Weisbuch [ 2013-01-14 ]

This new revision adds a subquery cache section with hit rate and hit/miss informations for it.

Another addition is the percentage of disk created temporary tables.

A check has been added to verify if innodb is enabled before showing and computing innodb variables (in case you have skip-innodb you wont end up with many warnings).

Plus some small code cleanups such as the $query that is now local on the functions.

Comment by George L [ 2013-02-01 ]

Jean seems I am having problem applying r6 diff ?

patch mysqlreport -i aria.diff -o mysqlreport_patched.txt
patching file mysqlreport
Hunk #17 succeeded at 1053 (offset -1 lines).
Hunk #18 succeeded at 1063 (offset -1 lines).
Hunk #19 succeeded at 1096 (offset -3 lines).
Hunk #20 succeeded at 1152 (offset -3 lines).
Hunk #21 succeeded at 1257 (offset -3 lines).
Hunk #22 succeeded at 1290 (offset -3 lines).
Hunk #23 succeeded at 1339 (offset -3 lines).
Hunk #24 FAILED at 1410.
1 out of 24 hunks FAILED – saving rejects to file mysqlreport_patched.txt.rej

rej file contents

@@ -1410,3 +1524,22 @@
Updated @>>>>>>>> @>>>>>/s
make_short($stats

{'Innodb_rows_updated'}), t($stats{'Innodb_rows_updated'}

)
.
+
+format PAGECACHE_BUFF_MAX =
+
+__ Aria Pagecache ______________________________________________________
+Buffer used @>>>>>> of @>>>>>> %Used: @>>>>>
+make_short($pagecache_buffer_used, 1), make_short($vars

{'aria_pagecache_buffer_size'}, 1), perc($pagecache_buffer_used, $vars{'aria_pagecache_buffer_size'}

)
+.
+
+format PAGECACHE_BUFF_USAGE =
+ Current @>>>>>> %Usage: @>>>>>
+make_short($pagecache_buffer_usage, 1), perc($pagecache_buffer_usage, $vars

{'aria_pagecache_buffer_size'}

)
+.
+
+format PAGECACHE_RATIOS =
+Write hit @>>>>>%
+$pagecache_write_ratio
+Read hit @>>>>>%
+$pagecache_read_ratio
+.

Comment by Jean Weisbuch [ 2013-02-01 ]

I attached the up-to-date patched mysqlreport.

Comment by Jean Weisbuch [ 2013-03-12 ]

This revision adds the binary log cache hit ratio for both transactionnal and statements (not shown if bin_log is not activated).

eg:
__ Binary Log Cache _____________________________________________________
Disk use
Transactional 0.57%
Non transactional 0.48%

Comment by Jean Weisbuch [ 2013-03-29 ]

I uploaded a new revision (mysqlreport-29mar2013) that corrects a regression added in the 12mar2013 revision, mysqlreport would hit a division by 0 error in case the bin_log would be activated with Binlog_cache_use = 0 or Binlog_stmt_cache_use = 0.

Also, in case both of the ratios = 0.00%, the whole binlog cache section wont be shown.

Comment by Jean Weisbuch [ 2013-05-06 ]

The 6 may revision adds TokuDB engine support : cachetable usage and miss rate are shown.

eg:
__ TokuDB ______________________________________________________________
Cachetable 4.00G of 4.00G %Used: 99.95
Miss 332.57k 0.3/s

Comment by Jean Weisbuch [ 2013-05-13 ]

This revision fixes the per seconds stats that were truncated to 5 chars max on the Subquery Cache, Questions and Query Cache sections (for example 153.6k/s was truncated to 153.6/s).

Comment by Jean Weisbuch [ 2013-05-28 ]

The 28 may release is a small bugfix that supress a warning about an uninitialized variable when ran on MySQL/MariaDB 5.1 which doesn't have the "Binlog_stmt_cache_use" variable.

Comment by George L [ 2013-05-28 ]

Hi Jean would like to say thank you so much for the updates, I find mysqlreport very useful and it's awesome that your contributions are keeping this tool up to date for MariaDB

Comment by Jean Weisbuch [ 2013-05-29 ]

The 29 may 2013 revision adds :

  • TokuDB pagecache evictions stats.
  • It is now written MariaDB at the top of the report (instead of MySQL) if the Aria engine is present.
  • The server version is now truncated after 2 more chars (eg: 5.5.30-tokudb-7.0.1 instead of 5.5.30-tokudb-7.0).

Also, historical comments about MySQL 6.0 have been removed from the code.

ps: thanks George for your support.

Comment by Jean Weisbuch [ 2013-06-07 ]

I put the code on Github as its simpler to manage and have a revision history : https://github.com/jb-boin/mariadb/blob/5.5.30/debian/additions/mysqlreport

The new revision i made today includes informations about row count : the number of rows returned by the server, the number of rows returned using indexes (and its percentage) and the average number of rows returned by question.

eg:
__ Rows ________________________________________________________________
Rows 1.11T 400.0k/s
Using idx 947.99G 341.4k/s %Index: 85.36
Rows/question 289.14

Comment by Jean Weisbuch [ 2013-06-07 ]

TODO:

Upstream bug to fix :
__ Questions ___________________________________________________________
Total 3.84G 1.4k/s
DMS 4.55G 1.6k/s %Total: 118.52
-Unknown 1.90G 682.7/s 49.35
Com_ 1.18G 426.2/s 30.81
COM_QUIT 825.96k 0.3/s 0.02

Comment by Jean Weisbuch [ 2013-07-04 ]

I updated mysqlreport to include threadpool usage stats when used on MariaDB (i don't know how it is implemented on MySQL).

It also don't show the "Cached" line anymore when the thread pool is used as the thread_cache_size variable is not used when activated and the Threads_cached status always return 0 which was resulting on bogus values (often negative percentage of %Cache).

Here is an example of the output on a server with thread_handling=pool-of-threads :
__ Threads _____________________________________________________________
Running 2 of 267
Created 180.28k 0.2/s
Slow 0 0/s
Threadpool 95 of 1200 %Used: 7.92
Running 48 of 1200 %Running: 4
Idle 47 of 1200 %Idle: 3.92

The revision diff : https://github.com/jb-boin/mariadb/commit/b25a7dfc2b7c5fbaf7bf83e2a16498cfdd5e3c10#debian/additions/mysqlreport

Comment by Jean Weisbuch [ 2013-09-17 ]

Threadpool usage calculation doesnt seems to be correct, i got these values on a live server (the threadpool was not full at the moment) :

__ Threads _____________________________________________________________
Running 1 of 1195
Created 365.79k 0.2/s
Slow 0 0/s
Threadpool 1301 of 1200 %Used: 108.42
Running 651 of 1200 %Running: 54.25
Idle 650 of 1200 %Idle: 54.17

------------------------------------------------+

Variable_name Value

------------------------------------------------+

Threadpool_idle_threads 654
Threadpool_threads 656
Threads_cached 0
Threads_connected 1189
Threads_created 365790
Threads_running 418

------------------------------------------------+

Comment by Jean Weisbuch [ 2013-10-02 ]

Today's revision adds the XtraDB flushes LRU stats : https://github.com/jb-boin/mariadb/commit/a9922df1095d2b87d8541c1cce623df4b0f82656

Example of the output :
Flushes 49.65M 15.4/s
LRU 2.54M 0.8/s %Flush: 5.12

Comment by Jean Weisbuch [ 2013-12-22 ]

New revision :https://github.com/jb-boin/mariadb/commit/7598e45acb42b89a9de29600f46066350cf5494d#diff-7552e4de9944e68ad76cc2faaba80034

  • MariaDB detection has been cleaned : was just checking that the Aria engine was enabled, now verify that the version string has -MariaDB on it
  • Thread pool stats should now be shown on Percona Server as well
  • The uptime format was "DAYS H:M:S", now "DAYS+H:M:S" for a better readability
  • %Flush is now shown as %LRU to be consistent with the other stats output
  • Deletion of the temporary file is now done with the Perl "unlink" function instead of executing the system "rm" or "del" command
Comment by Jean Weisbuch [ 2014-11-18 ]

This new revision (3.5-maria15 : https://github.com/jb-boin/mariadb/commit/a612dac0f4e3c7db358db10f740ca0cfd3b5121f) adds support for MariaDB 10.0 :

  • TokuDB variables case changed from 5.5 to 10.0 (eg. Tokudb_CACHETABLE_SIZE_CURRENT => Tokudb_cachetable_size_current) : the $stats array indexes are now forced to have the first char uppercase and the rest lowercase to avoid issues when this kind of changes happens and to keep compat with both versions.
  • (upstream bug) Test to show is the slow query log is enabled was relying on the log_slow_queries deprecated variable that has been replaced by slow_query_log for versions >= 5.1.29 and removed on MariaDB 10.0.

Tested without any errors/warnings issued on MySQL 5.1.72, MariaDB 5.5.40 and MariaDB 10.0.14 with TokuDB and tested both with XtraDB and Innodb plugin.

Comment by Jean Weisbuch [ 2015-01-05 ]

New revision (3.5-maria16 : https://github.com/jb-boin/mariadb/commit/bbd68f13b1cf90279e4713fc163bafc787df32ff) :

  • The query cache section is not shown anymore when have_query_cache=ON, query_cache_size>0 and query_cache_type=OFF.
Comment by Jean Weisbuch [ 2015-10-23 ]

New revision (4.0 : https://github.com/jb-boin/mariadb/blob/0e765c8cdf70cbfc85fbe42c4c6a08f379b6a1a2/debian/additions/mysqlreport) :

  • Compatibility with MySQL versions anterior to 5.0 has been restored : Checks relying on the information_schema are now skipped on versions older than 5.0.0.
Generated at Thu Feb 08 06:29:45 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.