[MCOL-340] Logging fails totally after removing the log files Created: 2016-09-30  Updated: 2023-10-26  Resolved: 2016-10-20

Status: Closed
Project: MariaDB ColumnStore
Component/s: ?
Affects Version/s: 1.0.3
Fix Version/s: Icebox

Type: Bug Priority: Major
Reporter: Daniel Wikman Assignee: David Thompson (Inactive)
Resolution: Not a Bug Votes: 0
Labels: None
Environment:

Centos7.2


 Description   

After removing the log files: debug.log warning.log err.log info.log I can't get the logging to work again. Even after creating the files again.

Before I did anything:

[root@tgw-qxdr3 columnstore]# pwd
/var/log/mariadb/columnstore

[root@tgw-qxdr3 columnstore]# ls -la debug.log warning.log err.log info.log
total 6112
drwxr-xr-x 5 root root 4096 Sep 30 03:30 .
drwxr-x---. 3 mysql mysql 4096 Sep 27 07:54 ..
-rwxr-xr-x 1 root root 2076488 Sep 30 05:08 debug.log
-rwxr-xr-x 1 root root 0 Sep 30 03:30 err.log
-rwxr-xr-x 1 root root 2073658 Sep 30 05:08 info.log
-rwxr-xr-x 1 root root 2072205 Sep 30 05:08 warning.log

Then I removed the files:
[root@tgw-qxdr3 columnstore]# rm -f debug.log warning.log err.log info.log

And restarted columnstore:
[root@tgw-qxdr3 columnstore]# mcsadmin restartSystem y
restartsystem Fri Sep 30 05:56:25 2016

System being restarted now ...
Successful restart of System

Here you would think that the removed log files should have been created again, but that never happened.

[root@tgw-qxdr3 columnstore]# ls -la debug.log warning.log err.log info.log
ls: cannot access debug.log: No such file or directory
ls: cannot access warning.log: No such file or directory
ls: cannot access err.log: No such file or directory
ls: cannot access info.log: No such file or directory

Here's the log configuration:

mscadmin> getLogConfig
getlogconfig Fri Sep 30 05:38:30 2016
MariaDB Columnstore System Log Configuration Data
System Logging Configuration File being used: /etc/rsyslog.d/49-columnstore.conf
Module Configured Log Levels
------ ---------------------------------------
pm1 Critical Error Warning Info Debug

[root@tgw-qxdr3 columnstore]# cat /etc/rsyslog.d/49-columnstore.conf

  1. MariaDb Columnstore Database Platform Logging
    local1.crit -/var/log/mariadb/columnstore/crit.log
    local1.err -/var/log/mariadb/columnstore/err.log
    local1.warning -/var/log/mariadb/columnstore/warning.log
    local1.info -/var/log/mariadb/columnstore/info.log
    local1.debug -/var/log/mariadb/columnstore/debug.log

And even after creating the files manually and a restartSystem, no logging occurs:

[root@tgw-qxdr3 columnstore]# touch err.log warning.log debug.log info.log
[root@tgw-qxdr3 columnstore]# chmod 755 err.log warning.log debug.log info.log

[root@tgw-qxdr3 columnstore]# mcsadmin restartSystem y
restartsystem Fri Sep 30 06:05:02 2016

System being restarted now .............
Successful restart of System

[root@tgw-qxdr3 columnstore]# ls -la debug.log warning.log err.log info.log
-rwxr-xr-x 1 root root 0 Sep 30 06:02 debug.log
-rwxr-xr-x 1 root root 0 Sep 30 06:01 err.log
-rwxr-xr-x 1 root root 0 Sep 30 06:01 info.log
-rwxr-xr-x 1 root root 0 Sep 30 06:01 warning.log

 Comments   
Comment by Daniel Wikman [ 2016-10-03 ]

Somehow the logs started to work again..
Not sure what solved the problem.

Comment by David Thompson (Inactive) [ 2016-10-19 ]

i can reproduce this. A workaround / way to retrigger the logging is to run enablelog which seems to trigger this:
mcsadmin enablelog pm1 critical

I agree that it's surprising that shutting down and restarting the server doesn't resume the logging. I can take a guess that the the reason it fixed itself is probably due to the log archiving / rotation.

This is important to address as operationally sometimes you want to blow away logs in emergency circumstances when disk gets low.

Comment by David Thompson (Inactive) [ 2016-10-19 ]

Can you take a look. Possibly this is legacy infinidb behavior but if possible we should try to fix this in order of preference:

  • the system is resiliant to log files being deleted and recreates them if not existing.
  • the system correctly starts logging after restart.
Comment by David Thompson (Inactive) [ 2016-10-20 ]

Column store uses syslog logging and this is actually an issue with rsyslogd rather than column store. Should you be in this situation, a better fix is to restart the service, e.g.:

systemctl restart rsyslog.service

The install configures this and also configures syslog to do rotation and archiving of the log files. The following command will tell you where the syslog config file is

mcsadmin getLogConfig

log rotation config is in /etc/logrotate.d/columnstore (at least for Centos 7)

Generated at Thu Feb 08 02:20:19 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.