[MDEV-13896] Upgraded to 10.2.8 on Centos 7.4 ibdata error Created: 2017-09-23  Updated: 2021-03-11  Resolved: 2021-03-11

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.2.8
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Jeffrey Smith Assignee: Unassigned
Resolution: Not a Bug Votes: 2
Labels: None
Environment:

CentOS Linux release 7.4.1708 (Core)

Issue Links:
Problem/Incident
is caused by MDEV-10298 Improve systemd service hardening Closed

 Description   

did a yum update to install the latest version and rebooted my machine, now maridb refuses to start.

Sep 23 11:24:13 hornbillsql1 polkitd[759]: Registered Authentication Agent for unix-process:6449:287204 (system bus name :1.74 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, l
 
Sep 23 11:24:13 hornbillsql1 systemd[1]: Starting MariaDB database server...
 
-- Subject: Unit mariadb.service has begun start-up
 
-- Defined-By: systemd
 
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
 
--
 
-- Unit mariadb.service has begun starting up.
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] /usr/sbin/mysqld (mysqld 10.2.8-MariaDB-log) starting as process 6682 ...
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Warning] Can't create test file /home/mysql/hornbillsql1.lower-test
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] mysqld: File '/home/mysql/aria_log_control' not found (Errcode: 30 "Read-only file system")
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] mysqld: Got error 'Can't open file' when trying to use aria control file '/home/mysql/aria_log_control'
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] Plugin 'Aria' init function returned error.
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Uses event mutexes
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Compressed tables use zlib 1.2.7
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Using Linux native AIO
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Number of pools: 1
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Using SSE2 crc32 instructions
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Completed initialization of buffer pool
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140564997007104 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Starting shutdown...
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] Plugin 'InnoDB' init function returned error.
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] Plugin 'FEEDBACK' is disabled.
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] Unknown/unsupported storage engine: InnoDB
 
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] Aborting
 
Sep 23 11:24:14 hornbillsql1 systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
 
Sep 23 11:24:14 hornbillsql1 systemd[1]: Failed to start MariaDB database server.
 
-- Subject: Unit mariadb.service has failed
 
-- Defined-By: systemd
 
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
 
--
 
-- Unit mariadb.service has failed.
 
--
 
-- The result is failed.

Its saying permission are wrong on ibdata1 though they are set as owned by the mysql user.

-rw-rw----. 1 mysql mysql 3667918848 Sep 23 10:16 ibdata1

SELinux is also currently set as permissive so cant be that.

[root@hornbillsql1 mysql]# sestatus
 
SELinux status:                 enabled
 
SELinuxfs mount:                /sys/fs/selinux
 
SELinux root directory:         /etc/selinux
 
Loaded policy name:             targeted
 
Current mode:                   permissive
 
Mode from config file:          permissive
 
Policy MLS status:              enabled
 
Policy deny_unknown status:     allowed
 
Max kernel policy version:      28

also seems I'm not the only one https://stackoverflow.com/questions/46345170/mariadb-ibdata1-file-cannot-be-written#



 Comments   
Comment by Lydie Decoker [ 2017-09-28 ]

I got the exact same issue. After upgrading my Mariadb to 10.2.9-1, I can not restart it.
My mariadb data dir is not in the standard directory but also under /home. Everything was working well before the upgrade.
SELinux is disabled on my server.

If I tried to restart using the normal datadir (I still have my old files there), it does restart.

Another test : If I do start mysql via command line (mysqld -u mysql), mariadb starts.

From my point of view, this is a quite blocking issue as this does crashed everything. Any help / reaction would be appreciated.

Thx!

Comment by Elena Stepanova [ 2017-09-28 ]

jeffsmith82, ldecoker, from which version did you upgrade?

I see that both of you use datadir under /home. By default, home is protected, if you want to use it, you need to remove the flag. See more details in MDEV-10399 and MDEV-10298.

But it's not a recent change, it was made in 10.1.16 (which means over a year ago), so it's not clear why you have only started experiencing it now, unless you upgraded from 5.5 or 10.0 or some very old release of 10.1. Was it the case?

Comment by Lydie Decoker [ 2017-09-28 ]

Elena,

Thx for the reply. The ProtectHome variable was already set to false before the upgrade. As said, everything was working well with the datadir set to /home.

I was already running the 10.2 version before the yum update. So this was a minor upgrade in my case.
What is strange to me is when I start mysqld manually, it works (using the same datadir). So the error saying that the filesystem is read-only does not seem correct to me.

Comment by Lydie Decoker [ 2017-09-28 ]

Elena,

Looks like the problem comes from the ProtectSystem. It was set to "full". I forced it to false and now MariaDb is starting.
but from what I have read, this variable does protect /boot, /usr and /etc. Do not understand why I have to disable it to make it work. Is there any other values for this variable (except full or false)?

Thx!

Comment by Sergey Vojtovich [ 2017-09-28 ]

FYI:

ProtectSystem=
Takes a boolean argument or "full". If true, mounts the /usr and /boot directories read-only for processes invoked by this unit. If set to "full", the /etc directory is mounted read-only, too. This setting ensures that any modification of the vendor-supplied operating system (and optionally its configuration) is prohibited for the service. It is recommended to enable this setting for all long-running services, unless they are involved with system updates or need to modify the operating system in other ways. Note however that processes retaining the CAP_SYS_ADMIN capability can undo the effect of this setting. This setting is hence particularly useful for daemons which have this capability removed, for example with CapabilityBoundingSet=. Defaults to off.

ProtectHome=
Takes a boolean argument or "read-only". If true, the directories /home, /root and /run/user are made inaccessible and empty for processes invoked by this unit. If set to "read-only", the three directories are made read-only instead. It is recommended to enable this setting for all long-running services (in particular network-facing ones), to ensure they cannot get access to private user data, unless the services actually require access to the user's private data. Note however that processes retaining the CAP_SYS_ADMIN capability can undo the effect of this setting. This setting is hence particularly useful for daemons which have this capability removed, for example with CapabilityBoundingSet=. Defaults to off.

Comment by Elena Stepanova [ 2017-11-05 ]

svoj, from your explanation above, it's unclear why ProtectSystem=full would affect server startup (via the service). I can't think what it might need to modify under /etc/. Do you have any ideas on that?

Comment by Sergey Vojtovich [ 2017-11-22 ]

elenst, it is unclear to me either. We probably need to try reproducing this on our side. Might be systemd bug?

Comment by Daniel Black [ 2021-03-11 ]

Looks like a ProtectHome=yes being default issue.

Fix per documentation: https://mariadb.com/kb/en/systemd/#configuring-access-to-home-directories

Generated at Thu Feb 08 08:09:12 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.