[MDEV-12327] InnoDB status monitor gets enabled after server startup Created: 2017-03-22  Updated: 2018-01-08  Resolved: 2018-01-08

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB, Storage Engine - XtraDB
Affects Version/s: 10.0.30, 10.1.22
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: azurit Assignee: Jan Lindström (Inactive)
Resolution: Duplicate Votes: 1
Labels: None
Environment:

Debian Jessie, 64 bit


Attachments: PNG File server08.png    
Issue Links:
Duplicate
is duplicated by MDEV-14776 InnoDB Monitor output generated by sp... Closed

 Description   

Setting innodb_status_output = 0 in config file has no effect, innodb status is still enabled after server restart (and can be disabled only by executing 'set global innodb_status_output=OFF;').



 Comments   
Comment by Elena Stepanova [ 2017-03-26 ]

innodb_status_output is disabled by default, setting it to 0 in config file shouldn't be necessary. If it's enabled after server restart, it most likely means you have it enabled in some other config file which is read after the one that you edit. Please check.

Comment by azurit [ 2017-03-26 ]

According to documentation ( https://mariadb.com/kb/en/mariadb/xtradb-innodb-monitors/ ), innodb_status_output is also enabled IF table mysql.innodb_monitor exists. This table was created automatically by upgrade script and i don't want to drop it as it will be, probably, created again with next upgrade. I need to disable this feature as it is totaly spamming my logs And no, i'm NOT enabling it in any config file.

Comment by azurit [ 2017-03-26 ]

i checked it again and there's no table innodb_monitor (or other innodb_*_monitor) in any database. i don't understand then why this feature gets enabled with every server restart after the upgrade to 10.0.30/10.1.22.

Comment by Elena Stepanova [ 2017-03-26 ]

According to documentation ( https://mariadb.com/kb/en/mariadb/xtradb-innodb-monitors/ ), innodb_status_output is also enabled IF table mysql.innodb_monitor exists. This table was created automatically by upgrade script and i don't want to drop it as it will be, probably, created again with next upgrade. I need to disable this feature as it is totaly spamming my logs And no, i'm NOT enabling it in any config file.

Actually, the page says that you can run CREATE TABLE to enable the monitor, not to have it created. Even if the table exists, its pure existence doesn't enable the monitor upon server startup (I've double-checked).

Also, it wouldn't be created by upgrade script (unless you have your own upgrade script), it's neither needed nor wanted by the server.

Finally, you can safely drop the table. It is a deprecated method of enabling the monitor. Please drop, restart the server and see if it helps.

Comment by Elena Stepanova [ 2017-03-26 ]

i checked it again and there's no table innodb_monitor (or other innodb_*_monitor) in any database. i don't understand then why this feature gets enabled with every server restart after the upgrade to 10.0.30/10.1.22.

If you have 10.1 handy, please run

select * from information_schema.system_variables where variable_name = 'innodb_status_output' \G

Comment by azurit [ 2017-03-26 ]

here it is:

*************************** 1. row ***************************
        VARIABLE_NAME: INNODB_STATUS_OUTPUT
        SESSION_VALUE: NULL
         GLOBAL_VALUE: ON
  GLOBAL_VALUE_ORIGIN: CONFIG
        DEFAULT_VALUE: OFF
       VARIABLE_SCOPE: GLOBAL
        VARIABLE_TYPE: BOOLEAN
     VARIABLE_COMMENT: Enable InnoDB monitor output to the error log.
    NUMERIC_MIN_VALUE: NULL
    NUMERIC_MAX_VALUE: NULL
   NUMERIC_BLOCK_SIZE: NULL
      ENUM_VALUE_LIST: OFF,ON
            READ_ONLY: NO
COMMAND_LINE_ARGUMENT: OPTIONAL

again, i'm NOT enabling it in config file:

# grep -R "innodb_status_output" /etc/
/etc/mysql/my.cnf:innodb_status_output = 0

Comment by Elena Stepanova [ 2017-03-26 ]

Please also attach your error log (you might need to gzip it if it's too big). InnoDB might enable status monitor automatically under certain conditions, such as

  • a long semaphore wait
  • cannot find free blocks in the buffer pool
    and some more. The error log will hopefully give us a clue.

Regarding the grep, it is not enough. Your cnf file might be in the home dir, or the option might look like 'innodb-status-output'. The easiest way to make sure is to start the server with --defaults-file=/etc/mysql/my.cnf

Comment by azurit [ 2017-03-26 ]

is this it?

Mar 26 13:02:15 server01 mysqld[22546]: 2017-03-26 13:02:15 7f57e93f9700  InnoDB: Warning: difficult to find free blocks in
Mar 26 13:02:15 server01 mysqld[22546]: InnoDB: the buffer pool (162 search iterations)!
Mar 26 13:02:15 server01 mysqld[22546]: InnoDB: 0 failed attempts to flush a page! Consider
Mar 26 13:02:15 server01 mysqld[22546]: InnoDB: increasing the buffer pool size.
Mar 26 13:02:15 server01 mysqld[22546]: InnoDB: It is also possible that in your Unix version
Mar 26 13:02:15 server01 mysqld[22546]: InnoDB: fsync is very slow, or completely frozen inside
Mar 26 13:02:15 server01 mysqld[22546]: InnoDB: the OS kernel. Then upgrading to a newer version
Mar 26 13:02:15 server01 mysqld[22546]: InnoDB: of your operating system may help. Look at the
Mar 26 13:02:15 server01 mysqld[22546]: InnoDB: number of fsyncs in diagnostic info below.
Mar 26 13:02:15 server01 mysqld[22546]: InnoDB: Pending flushes (fsync) log: 0; buffer pool: 1
Mar 26 13:02:15 server01 mysqld[22546]: InnoDB: 7704258 OS file reads, 3721002 OS file writes, 934909 OS fsyncs
Mar 26 13:02:15 server01 mysqld[22546]: InnoDB: Starting InnoDB Monitor to print further
Mar 26 13:02:15 server01 mysqld[22546]: InnoDB: diagnostics to the standard output.

what does it mean? it started to appear on several different servers after upgrade to 10.1.22 (10.0.30 is probably not affected).

Comment by azurit [ 2017-03-26 ]

innodb_buffer_pool_size = 512M

Comment by Elena Stepanova [ 2017-03-26 ]

Yes, that's it.
I assume you didn't change your InnoDB settings recently? Maybe unintentionally?
Does it start happening right after server startup?
I'll assign it to an InnoDB expert, hopefully he'll figure out what could have caused the difference. He'll probably request more information from you. Meanwhile, I think the output of show variables won't hurt, and the actual output of the status monitor.

Comment by azurit [ 2017-03-26 ]

I have 6 different servers where is this problem occuring but i must say that it's NOT happening on all 10.1.22 servers. I didn't notice it before and i probably would as syslog log files are really full of it (100 MB and more).

I tested it on one server and innodb_status_output is not enabled immediately after server starts.

The message above is occuring multiple times per day on every affected server but i wasn't able to find anything which could trigger it - sometimes i can see bigger server load at time when it's logged (the most visible is it on server graphs which i attached, message was logged 5 times at 4:09, by the way, server backup was running at that time).

Comment by azurit [ 2017-03-27 ]

I didn't change any settings and i was upgrading from 10.1.20 (we skipped 10.1.21 because of MDEV-11842 ).

Comment by azurit [ 2017-04-06 ]

Anything new? We are really considering moving away from MariaDB back to MySQL because of lots of severe bugs. I'm also trying to resolve MDEV-10011 bud i don't see a thing in logs because of this shit! I'm forced to work with 100 MB logs/day!!

Generated at Thu Feb 08 07:56:51 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.