Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Not a Bug
-
10.2.8
-
None
-
CentOS Linux release 7.4.1708 (Core)
Description
did a yum update to install the latest version and rebooted my machine, now maridb refuses to start.
Sep 23 11:24:13 hornbillsql1 polkitd[759]: Registered Authentication Agent for unix-process:6449:287204 (system bus name :1.74 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, l |
Sep 23 11:24:13 hornbillsql1 systemd[1]: Starting MariaDB database server... |
-- Subject: Unit mariadb.service has begun start-up
|
-- Defined-By: systemd
|
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel |
--
|
-- Unit mariadb.service has begun starting up.
|
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] /usr/sbin/mysqld (mysqld 10.2.8-MariaDB-log) starting as process 6682 ... |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Warning] Can't create test file /home/mysql/hornbillsql1.lower-test |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] mysqld: File '/home/mysql/aria_log_control' not found (Errcode: 30 "Read-only file system") |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] mysqld: Got error 'Can't open file' when trying to use aria control file '/home/mysql/aria_log_control' |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] Plugin 'Aria' init function returned error. |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed. |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Uses event mutexes |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Compressed tables use zlib 1.2.7 |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Using Linux native AIO |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Number of pools: 1 |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Using SSE2 crc32 instructions |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Completed initialization of buffer pool |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140564997007104 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] InnoDB: Plugin initialization aborted with error Generic error |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] InnoDB: Starting shutdown... |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] Plugin 'InnoDB' init function returned error. |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [Note] Plugin 'FEEDBACK' is disabled. |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] Unknown/unsupported storage engine: InnoDB |
Sep 23 11:24:14 hornbillsql1 mysqld[6682]: 2017-09-23 11:24:14 140567722469504 [ERROR] Aborting |
Sep 23 11:24:14 hornbillsql1 systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE |
Sep 23 11:24:14 hornbillsql1 systemd[1]: Failed to start MariaDB database server. |
-- Subject: Unit mariadb.service has failed
|
-- Defined-By: systemd
|
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel |
--
|
-- Unit mariadb.service has failed.
|
--
|
-- The result is failed.
|
Its saying permission are wrong on ibdata1 though they are set as owned by the mysql user.
-rw-rw----. 1 mysql mysql 3667918848 Sep 23 10:16 ibdata1
|
SELinux is also currently set as permissive so cant be that.
[root@hornbillsql1 mysql]# sestatus
|
SELinux status: enabled
|
SELinuxfs mount: /sys/fs/selinux
|
SELinux root directory: /etc/selinux
|
Loaded policy name: targeted
|
Current mode: permissive
|
Mode from config file: permissive
|
Policy MLS status: enabled
|
Policy deny_unknown status: allowed
|
Max kernel policy version: 28
|
also seems I'm not the only one https://stackoverflow.com/questions/46345170/mariadb-ibdata1-file-cannot-be-written#
Attachments
Issue Links
- is caused by
-
-
- Closed
-
Activity
jeffsmith82, ldecoker, from which version did you upgrade?
I see that both of you use datadir under /home. By default, home is protected, if you want to use it, you need to remove the flag. See more details in MDEV-10399 and MDEV-10298.
But it's not a recent change, it was made in 10.1.16 (which means over a year ago), so it's not clear why you have only started experiencing it now, unless you upgraded from 5.5 or 10.0 or some very old release of 10.1. Was it the case?
Elena,
Thx for the reply. The ProtectHome variable was already set to false before the upgrade. As said, everything was working well with the datadir set to /home.
I was already running the 10.2 version before the yum update. So this was a minor upgrade in my case.
What is strange to me is when I start mysqld manually, it works (using the same datadir). So the error saying that the filesystem is read-only does not seem correct to me.
Elena,
Looks like the problem comes from the ProtectSystem. It was set to "full". I forced it to false and now MariaDb is starting.
but from what I have read, this variable does protect /boot, /usr and /etc. Do not understand why I have to disable it to make it work. Is there any other values for this variable (except full or false)?
Thx!
FYI:
ProtectSystem=
Takes a boolean argument or "full". If true, mounts the /usr and /boot directories read-only for processes invoked by this unit. If set to "full", the /etc directory is mounted read-only, too. This setting ensures that any modification of the vendor-supplied operating system (and optionally its configuration) is prohibited for the service. It is recommended to enable this setting for all long-running services, unless they are involved with system updates or need to modify the operating system in other ways. Note however that processes retaining the CAP_SYS_ADMIN capability can undo the effect of this setting. This setting is hence particularly useful for daemons which have this capability removed, for example with CapabilityBoundingSet=. Defaults to off.
ProtectHome=
Takes a boolean argument or "read-only". If true, the directories /home, /root and /run/user are made inaccessible and empty for processes invoked by this unit. If set to "read-only", the three directories are made read-only instead. It is recommended to enable this setting for all long-running services (in particular network-facing ones), to ensure they cannot get access to private user data, unless the services actually require access to the user's private data. Note however that processes retaining the CAP_SYS_ADMIN capability can undo the effect of this setting. This setting is hence particularly useful for daemons which have this capability removed, for example with CapabilityBoundingSet=. Defaults to off.
svoj, from your explanation above, it's unclear why ProtectSystem=full would affect server startup (via the service). I can't think what it might need to modify under /etc/. Do you have any ideas on that?
elenst, it is unclear to me either. We probably need to try reproducing this on our side. Might be systemd bug?
Looks like a ProtectHome=yes being default issue.
Fix per documentation: https://mariadb.com/kb/en/systemd/#configuring-access-to-home-directories
I got the exact same issue. After upgrading my Mariadb to 10.2.9-1, I can not restart it.
My mariadb data dir is not in the standard directory but also under /home. Everything was working well before the upgrade.
SELinux is disabled on my server.
If I tried to restart using the normal datadir (I still have my old files there), it does restart.
Another test : If I do start mysql via command line (mysqld -u mysql), mariadb starts.
From my point of view, this is a quite blocking issue as this does crashed everything. Any help / reaction would be appreciated.
Thx!