[MDEV-19444] when i change my datadir, the server cannot restart anymore!!! Created: 2019-05-12  Updated: 2019-05-13  Resolved: 2019-05-13

Status: Closed
Project: MariaDB Server
Component/s: Authentication and Privilege System
Affects Version/s: 10.3.14
Fix Version/s: N/A

Type: Bug Priority: Blocker
Reporter: hongzhenwei Assignee: Unassigned
Resolution: Not a Bug Votes: 0
Labels: None
Environment:

CentOS Linux release 7.6.1810 (Core)



 Description   

May 12 20:49:29 bogon systemd[1]: Starting MariaDB 10.3.14 database server...
May 12 20:49:29 bogon mysqld[21369]: 2019-05-12 20:49:29 0 [Note] /usr/sbin/mysqld (mysqld 10.3.14-MariaDB) starting as process 21369 ...
May 12 20:49:29 bogon systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
May 12 20:49:29 bogon systemd[1]: Failed to start MariaDB 10.3.14 database server.
May 12 20:49:29 bogon systemd[1]: Unit mariadb.service entered failed state.
May 12 20:49:29 bogon systemd[1]: mariadb.service failed.
[root@bogon mariadb-10.3.14-rhel-7-x86_64-rpms]# systemctl start mariadb
Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
[root@bogon mariadb-10.3.14-rhel-7-x86_64-rpms]# systemctl status mariadb.service
● mariadb.service - MariaDB 10.3.14 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─dontprotecthome.conf, limitcore.conf, limitnofile.conf, migrated-from-my.cnf-settings.conf
Active: failed (Result: exit-code) since Sun 2019-05-12 20:50:21 CST; 5s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 21424 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 21388 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 21386 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Main PID: 21424 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"

May 12 20:50:20 bogon systemd[1]: Starting MariaDB 10.3.14 database server...
May 12 20:50:21 bogon mysqld[21424]: 2019-05-12 20:50:21 0 [Note] /usr/sbin/mysqld (mysqld 10.3.14-MariaDB) starting as process 21424 ...
May 12 20:50:21 bogon systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
May 12 20:50:21 bogon systemd[1]: Failed to start MariaDB 10.3.14 database server.
May 12 20:50:21 bogon systemd[1]: Unit mariadb.service entered failed state.
May 12 20:50:21 bogon systemd[1]: mariadb.service failed.



 Comments   
Comment by Vicențiu Ciorbaru [ 2019-05-12 ]

Hi Faker,

There isn't much info to go on here, however I suspect that you edited your config file to point to a different data directory that was not initialized before. Before restarting the service, did you run:

mysql_install_db --user=mysql --basedir=<path-to-mariadb-install-directory> --datadir=<your-new-datadir>

basedir is probably /usr in your case, but best to double check where mysqld is.

Comment by hongzhenwei [ 2019-05-12 ]

Hi Vicentiu Ciorbaru,
I am using yum installation,Then, I specified datadir in /etc/my.cnf.d/server.cnf instead of the default. Now, I want to know how to change my datadir. could you give me some advice?
And I also executed the mysql_install_db command, but I got an error.such like this:
------------------------------------------------------------------------------------------------------------------------------------------
[root@mariadb3 mariadb-10.3.14-rhel-7-x86_64-rpms]# mysql_install_db --user=root --basedir=/var/lib/mysql --datadir=/usr/local/data/mariadb

FATAL ERROR: Could not find my_print_defaults

The following directories were searched:

/var/lib/mysql/bin
/var/lib/mysql/extra

If you compiled from source, you need to either run 'make install' to
copy the software into the correct location ready for operation.
If you don't want to do a full install, you can use the --srcdir
option to only install the mysql database and privilege tables

If you are using a binary release, you must either be at the top
level of the extracted archive, or pass the --basedir option
pointing to that location.

The latest information about mysql_install_db is available at
https://mariadb.com/kb/en/installing-system-tables-mysql_install_db
------------------------------------------------------------------------------------------------------------------------------------------

thank you!

Comment by Anel Husakovic [ 2019-05-12 ]

Hi Faker
did you try to specify

--no-defaults

to run as you did above (this is one option you can try) according to documentation.
Second option you can try is to add custom file for your configuration and run

--defaults-file=/path/to/your/.cnf 

to your CLI.

Comment by Daniel Black [ 2019-05-13 ]

By the filename `dontprotecthome.conf`, can I assume your datadir is on the /home directory now? Or /usr/local/data/mariadb (link your `mysql_install_db`?). In addition to the systemd protections on /home, there are probably selinux protections too. By following from What to Do if MariaDB Doesn't Start, to selinux there's instructions for changing the datadir selinux labels.

With `mysql_install_db` and your configuration set, the only option I think is needed is '--user mysql` (datadir is picked up from the configuration), the user the mysql instance is running as.

Comment by hongzhenwei [ 2019-05-13 ]

Hi Daniel Black,
After I successfully executed the command, I generated a new datadir, as follows:
----------------------------------------------------------------------------------------------------------------------------------------
[root@mariadb3 mariadb]# pwd
/usr/local/data/mariadb
[root@mariadb3 mariadb]# ll
total 110620
rw-rw---. 1 root root 16384 May 13 10:23 aria_log.00000001
rw-rw---. 1 root root 52 May 13 10:23 aria_log_control
rw-rw---. 1 root root 976 May 13 10:23 ib_buffer_pool
rw-rw---. 1 root root 12582912 May 13 10:23 ibdata1
rw-rw---. 1 root root 50331648 May 13 10:23 ib_logfile0
rw-rw---. 1 root root 50331648 May 13 09:23 ib_logfile1
drwx------. 2 root root 4096 May 13 09:23 mysql
drwx------. 2 root root 20 May 13 10:23 performance_schema
drwx------. 2 root root 20 May 13 09:23 test
----------------------------------------------------------------------------------------------------------------------------------------
But when I started the service, I still got an error(At this point, I pointed datadir to my new directory in my.cnf instead of the default.), as follows:
----------------------------------------------------------------------------------------------------------------------------------------
May 13 10:10:39 mariadb3 systemd[1]: Unit mariadb.service entered failed state.
May 13 10:10:39 mariadb3 systemd[1]: mariadb.service failed.
May 13 10:10:39 mariadb3 polkitd[6135]: Unregistered Authentication Agent for unix-process:19067:5215237 (system bus name :1.242, object path /org/f
May 13 10:20:01 mariadb3 systemd[1]: Started Session 108 of user root.
– Subject: Unit session-108.scope has finished start-up
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit session-108.scope has finished starting up.

-- The start-up result is done.
May 13 10:20:01 mariadb3 CROND[19104]: (root) CMD (/usr/lib64/sa/sa1 1 1)
May 13 10:24:28 mariadb3 polkitd[6135]: Registered Authentication Agent for unix-process:19192:5298178 (system bus name :1.245 [/usr/bin/pkttyagent
May 13 10:24:28 mariadb3 systemd[1]: Starting MariaDB 10.3.14 database server...
– Subject: Unit mariadb.service has begun start-up
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit mariadb.service has begun starting up.
May 13 10:24:28 mariadb3 mysqld[19221]: 2019-05-13 10:24:28 0 [Note] /usr/sbin/mysqld (mysqld 10.3.14-MariaDB) starting as process 19221 ...
May 13 10:24:28 mariadb3 mysqld[19221]: 2019-05-13 10:24:28 0 [Warning] Can't create test file /usr/local/data/mariadb/mariadb3.lower-test
May 13 10:24:28 mariadb3 mysqld[19221]: [99B blob data]
May 13 10:24:28 mariadb3 mysqld[19221]: 2019-05-13 10:24:28 0 [ERROR] Aborting
May 13 10:24:28 mariadb3 systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
May 13 10:24:28 mariadb3 systemd[1]: Failed to start MariaDB 10.3.14 database server.
– Subject: Unit mariadb.service has failed
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit mariadb.service has failed.

-- The result is failed.
----------------------------------------------------------------------------------------------------------------------------------------
I suspect that the problem is in the red line, but I don't know how to solve it.

Comment by Daniel Black [ 2019-05-13 ]

Your datadir and contents need to be owned by the `mysql` user. So `chown -R mysql: /usr/local/data/mariadb`. Then restart the service.

Comment by hongzhenwei [ 2019-05-13 ]

Hi Daniel Black,
I execute the command chown -R mysql: /usr/local/data/mariadb, but still can't start, can you test it with a virtual machine (centos 7.6)?
My operation steps are:
1. Download mariadb-10.3.14-rhel-7-x86_64-rpms.tar tarball,then Decompression;
2. Execute the ./setup_repository command to generate repo;
3. Execute the command: yum install MariaDB-server;
4. Execute the command: mysql_install_db --user=root --basedir=/usr --datadir=/usr/local/data/mariadb;
5. Configure the conf file: vi /etc/my.cnf.d/server.cnf Then, in [mysqld] Add datadir=/usr/local/data/mariadb to save and exit.
6. Start the service: systemctl start mariadb
Then there will be an error message.
This problem has plagued me for a few days and I hope to be solved.
Thank you!

Comment by Daniel Black [ 2019-05-13 ]

As said previously,

mysql_install_db --user=mysql

NOT:

mysql_install_db --user=root

Recommend configuration before mysql_install_db. You could even do it before installing packages. I prefer the repo tool as this simplifies the first two steps.

As previously pointed to `/usr/local/data/mariadb` needs to have the right selinux context with:

sudo chown -R mysql: /usr/local/data/mariadb
sudo chmod a+x /usr/local/data
sudo semanage fcontext -a -t mysqld_db_t "/usr/local/data/mariadb(/.*)?"
sudo restorecon -Rv /usr/local/data/mariadb/

Comment by hongzhenwei [ 2019-05-13 ]

Hi Daniel Black,
I implemented it according to your requirements, and even reinstalled a new system on the virtual machine , still can not solve the problem. And I have closed selinux.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
May 13 14:00:01 mariadb CROND[7944]: (root) CMD (/usr/lib64/sa/sa1 1 1)
May 13 14:00:56 mariadb polkitd[6256]: Registered Authentication Agent for unix-process:7948:97559 (system bus name :1.56 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
May 13 14:00:56 mariadb systemd[1]: Starting MariaDB 10.3.14 database server...
– Subject: Unit mariadb.service has begun start-up
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit mariadb.service has begun starting up.
May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [Note] /usr/sbin/mysqld (mysqld 10.3.14-MariaDB) starting as process 7971 ...
May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [Warning] Can't create test file /usr/local/data/mariadb/mariadb.lower-test
May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [ERROR] mysqld: File '/usr/local/data/mariadb/aria_log_control' not found (Errcode: 30 "Read-only file system")
May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [ERROR] mysqld: Got error 'Can't open file' when trying to use aria control file '/usr/local/data/mariadb/aria_log_control'
May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [ERROR] Plugin 'Aria' init function returned error.
May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [Note] InnoDB: Using Linux native AIO
May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [ERROR] Plugin 'InnoDB' init function returned error.
May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [Note] Plugin 'FEEDBACK' is disabled.
May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [ERROR] Unknown/unsupported storage engine: InnoDB
May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [ERROR] Aborting
May 13 14:00:56 mariadb systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
May 13 14:00:56 mariadb systemd[1]: Failed to start MariaDB 10.3.14 database server.
– Subject: Unit mariadb.service has failed
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit mariadb.service has failed.

-- The result is failed.
May 13 14:00:56 mariadb systemd[1]: Unit mariadb.service entered failed state.
May 13 14:00:56 mariadb systemd[1]: mariadb.service failed.
May 13 14:00:56 mariadb polkitd[6256]: Unregistered Authentication Agent for unix-process:7948:97559 (system bus name :1.56, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
May 13 14:01:01 mariadb systemd[1]: Started Session 6 of user root.
– Subject: Unit session-6.scope has finished start-up
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit session-6.scope has finished starting up.

-- The start-up result is done.
May 13 14:01:01 mariadb CROND[7977]: (root) CMD (run-parts /etc/cron.hourly)
May 13 14:01:01 mariadb run-parts(/etc/cron.hourly)[7980]: starting 0anacron
May 13 14:01:01 mariadb anacron[7987]: Anacron started on 2019-05-13
May 13 14:01:01 mariadb run-parts(/etc/cron.hourly)[7989]: finished 0anacron
May 13 14:01:01 mariadb anacron[7987]: Will run job `cron.daily' in 15 min.
May 13 14:01:01 mariadb anacron[7987]: Will run job `cron.weekly' in 35 min.
May 13 14:01:01 mariadb anacron[7987]: Will run job `cron.monthly' in 55 min.
May 13 14:01:01 mariadb anacron[7987]: Jobs will be executed sequentially

Below is the newly generated datadir:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[root@mariadb mariadb]# pwd
/usr/local/data/mariadb
[root@mariadb mariadb]# ll
total 110620
rw-rw---. 1 mysql mysql 16384 May 13 13:51 aria_log.00000001
rw-rw---. 1 mysql mysql 52 May 13 13:51 aria_log_control
rw-rw---. 1 mysql mysql 972 May 13 13:51 ib_buffer_pool
rw-rw---. 1 mysql mysql 12582912 May 13 13:51 ibdata1
rw-rw---. 1 mysql mysql 50331648 May 13 13:51 ib_logfile0
rw-rw---. 1 mysql mysql 50331648 May 13 13:51 ib_logfile1
drwx------. 2 mysql mysql 4096 May 13 13:51 mysql
drwx------. 2 mysql mysql 20 May 13 13:51 performance_schema
drwx------. 2 mysql mysql 20 May 13 13:51 test
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MySQL only needs to specify datadir in the my.cnf file, then close selinux and restart the server. I used MaxScale to achieve MySQL read-write separation. I found that MaxScale does not support MySQL8 very well, so I switched to mariadb. I think this should be a good choice. It took me a few days, and even the problem of starting the server was not solved.This makes me feel so depressed.I sincerely hope to solve this problem.
Thank you!

Comment by Daniel Black [ 2019-05-13 ]

bug reports aren't a help system:

use:
https://mariadb.zulipchat.com/
http://dba.stackexchange.com/
or the maria channel on freenode irc

May 13 14:00:56 mariadb mysqld[7971]: 2019-05-13 14:00:56 0 [ERROR] mysqld: File '/usr/local/data/mariadb/aria_log_control' not found (Errcode: 30 "Read-only file system")

This is a stronger indication of the issue, its mounted readonly. This is because of the systemd setting ProtectSystem being default https://mariadb.com/kb/en/library/systemd/. Disable this the same way as ProtectHome.

Recommend re-enabling selinux especially after disabling ProtectSystem. Changing the filecontext is still required.

Generated at Thu Feb 08 08:51:44 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.