[MDEV-15827] Running mariabackup with insufficient system privileges cause crash / coredump - Jira

Anders Karlsson created issue - 2018-04-09 15:31

Elena Stepanova made changes - 2018-04-09 15:46

Field	Original Value	New Value
Fix Version/s		10.2 [ 14601 ]
Fix Version/s		10.3 [ 22126 ]
Assignee		Vladislav Vaintroub [ wlad ]

Vladislav Vaintroub made changes - 2018-04-09 15:49

Description

As an ordinary user, run:
{code:bash}
$ mariabackup -backup
{code}

This is an error, but shouldn't cause a core dump like this:
{code:bash}
180409 17:26:16 Connecting to MySQL server host: localhost, user: not set, password: not set, port: not set, socket: not set
Using server version 10.2.14-MariaDB-log
mariabackup based on MariaDB server 10.2.14-MariaDB Linux (x86_64)
mariabackup: uses posix_fadvise().
mariabackup: cd to /var/lib/mysql/
mariabackup: open files limit requested 0, set to 1024
mariabackup: using the following InnoDB configuration:
mariabackup: innodb_data_home_dir = .
mariabackup: innodb_data_file_path = ibdata1:12M:autoextend
mariabackup: innodb_log_group_home_dir = ./
2018-04-09 17:26:16 139746500786304 [Note] InnoDB: Number of pools: 1
2018-04-09 17:26:16 139746500786304 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2018-04-09 17:26:16 139746500786304 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2018-04-09 17:26:16 0x7f194489b880 InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.14/storage/innobase/fil/fil0fil.cc line 761
InnoDB: Failing assertion: success
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/
InnoDB: about forcing recovery.
180409 17:26:16 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.2.14-MariaDB
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5421 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
addr2line: 'mariabackup': No such file
mariabackup(my_print_stacktrace+0x2e)[0x7f19454f552e]
mariabackup(handle_fatal_signal+0x355)[0x7f1944fe3835]
sigaction.c:0(__restore_rt)[0x7f19444ac100]
:0(__GI_raise)[0x7f1942a395f7]
:0(__GI_abort)[0x7f1942a3ace8]
addr2line: 'mariabackup': No such file
mariabackup(+0x434b1e)[0x7f1944d11b1e]
mariabackup(+0x8d40f5)[0x7f19451b10f5]
mariabackup(+0x8d432b)[0x7f19451b132b]
mariabackup(+0x8debe2)[0x7f19451bbbe2]
mariabackup(+0x989aa6)[0x7f1945266aa6]
mariabackup(+0x992245)[0x7f194526f245]
mariabackup(+0x457e22)[0x7f1944d34e22]
mariabackup(main+0x185)[0x7f1944d13b05]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1942a25b15]
addr2line: 'mariabackup': No such file
mariabackup(+0x45016d)[0x7f1944d2d16d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Aborted
{code}

Note that the dump says that "mysqld got signal 6", but actually mysqld per se didn't crash at all. In addition, there is a pointer at the end to a page in the MySQL manual that is unliekly to help (http://dev.mysql.com/doc/mysql/en/crashing.html) .

The workaround is to run mysqlbackup with sudo, obviously.

As an ordinary user, run:
{code:bash}
$ mariabackup -backup
{code}

This is an error, but shouldn't cause a core dump like this:
{code:bash}
180409 17:26:16 Connecting to MySQL server host: localhost, user: not set, password: not set, port: not set, socket: not set
Using server version 10.2.14-MariaDB-log
mariabackup based on MariaDB server 10.2.14-MariaDB Linux (x86_64)
mariabackup: uses posix_fadvise().
mariabackup: cd to /var/lib/mysql/
mariabackup: open files limit requested 0, set to 1024
mariabackup: using the following InnoDB configuration:
mariabackup: innodb_data_home_dir = .
mariabackup: innodb_data_file_path = ibdata1:12M:autoextend
mariabackup: innodb_log_group_home_dir = ./
2018-04-09 17:26:16 139746500786304 [Note] InnoDB: Number of pools: 1
2018-04-09 17:26:16 139746500786304 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2018-04-09 17:26:16 139746500786304 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2018-04-09 17:26:16 0x7f194489b880 InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.2.14/storage/innobase/fil/fil0fil.cc line 761
InnoDB: Failing assertion: success
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/
InnoDB: about forcing recovery.
180409 17:26:16 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.2.14-MariaDB
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5421 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
addr2line: 'mariabackup': No such file
mariabackup(my_print_stacktrace+0x2e)[0x7f19454f552e]
mariabackup(handle_fatal_signal+0x355)[0x7f1944fe3835]
sigaction.c:0(__restore_rt)[0x7f19444ac100]
:0(__GI_raise)[0x7f1942a395f7]
:0(__GI_abort)[0x7f1942a3ace8]
addr2line: 'mariabackup': No such file
mariabackup(+0x434b1e)[0x7f1944d11b1e]
mariabackup(+0x8d40f5)[0x7f19451b10f5]
mariabackup(+0x8d432b)[0x7f19451b132b]
mariabackup(+0x8debe2)[0x7f19451bbbe2]
mariabackup(+0x989aa6)[0x7f1945266aa6]
mariabackup(+0x992245)[0x7f194526f245]
mariabackup(+0x457e22)[0x7f1944d34e22]
mariabackup(main+0x185)[0x7f1944d13b05]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1942a25b15]
addr2line: 'mariabackup': No such file
mariabackup(+0x45016d)[0x7f1944d2d16d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Aborted
{code}

Note that the dump says that "mysqld got signal 6", but actually mysqld per se didn't crash at all. In addition, there is a pointer at the end to a page in the MySQL manual that is unliekly to help (http://dev.mysql.com/doc/mysql/en/crashing.html) .

The workaround is to run mariabackup with sudo, obviously.

Zdravelina Sokolovska (Inactive) made changes - 2018-04-11 12:26

Affects Version/s

10.3.5 [ 22905 ]

Zdravelina Sokolovska (Inactive) made changes - 2018-04-11 12:28

Attachment

mariabackup_for_non_root_users_how _to.txt [ 45470 ]

Zdravelina Sokolovska (Inactive) added a comment - 2018-04-11 12:33 - edited

workaround about mariabackup performing for non root user

[root@t4w6 ~]# sudo usermod -aG wheel foo4

[foo4@t4w6 ~]$ sudo chmod +s /usr/bin/mariabackup

[foo4@t4w6 ~]$ sudo chmod g+r /var/lib/mysql/ibdata1

Login as user 'foo4' and perform mariabackup

login as: foo4

foo4@t4w6's password:

Last login: Wed Apr 11 13:37:25 2018

[foo4@t4w6 ~]$  mariabackup -u root -p1  --backup

180411 13:41:49 Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: not set

Using server version 10.3.5-MariaDB

mariabackup based on MariaDB server 10.3.5-MariaDB Linux (x86_64)

mariabackup: uses posix_fadvise().

mariabackup: cd to /var/lib/mysql/

mariabackup: open files limit requested 0, set to 1024

mariabackup: using the following InnoDB configuration:

mariabackup:   innodb_data_home_dir = .

mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend

mariabackup:   innodb_log_group_home_dir = ./

2018-04-11 13:41:49 0 [Note] InnoDB: Number of pools: 1

180411 13:41:49 >> log scanned up to (51002181432)

mariabackup: Generating a list of tablespaces

2018-04-11 13:41:49 0 [Warning] InnoDB: Allocated tablespace ID 127 for md_15252/BRc273a13dfd, old maximum was 0

180411 13:41:49 [01] Copying ./ibdata1 to /home/foo4/xtrabackup_backupfiles/ibdata1

180411 13:41:50 >> log scanned up to (51002181432)

180411 13:42:08 [01]        ...done

180411 13:42:08 [01] Copying ./md_15252/BRc273a13dfd.ibd to /home/foo4/xtrabackup_backupfiles/md_15252/BRc273a13dfd.ibd

180411 13:42:08 [01]        ...done

~~~~~~ full log attached

180411 13:43:12 [01]        ...done

180411 13:43:12 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...

mariabackup: The latest check point (for incremental): '51002181423'

mariabackup: Stopping log copying thread.

2018-04-11 13:43:12 0 [Note] InnoDB: Read redo log up to LSN=51002181632

180411 13:43:12 >> log scanned up to (51002181432)

180411 13:43:12 Executing UNLOCK TABLES

180411 13:43:12 All tables unlocked

180411 13:43:12 [00] Copying ib_buffer_pool to /home/foo4/xtrabackup_backupfiles/ib_buffer_pool

180411 13:43:12 [00]        ...done

180411 13:43:12 Backup created in directory '/home/foo4/xtrabackup_backupfiles/'

180411 13:43:12 [00] Writing backup-my.cnf

180411 13:43:12 [00]        ...done

180411 13:43:12 [00] Writing xtrabackup_info

180411 13:43:12 [00]        ...done

mariabackup: Redo log (from LSN 51002181423 to 51002181432) was copied.

180411 13:43:12 completed OK!

[foo4@t4w6 ~]$

Zdravelina Sokolovska (Inactive) added a comment - 2018-04-11 12:33 - edited workaround about mariabackup performing for non root user [root@t4w6 ~] # sudo usermod -aG wheel foo4 [foo4@t4w6 ~] $ sudo chmod +s /usr/bin/mariabackup [foo4@t4w6 ~] $ sudo chmod g+r /var/lib/mysql/ibdata1 Login as user 'foo4' and perform mariabackup login as: foo4 foo4@t4w6's password: Last login: Wed Apr 11 13:37:25 2018 [foo4@t4w6 ~]$ mariabackup -u root -p1 --backup 180411 13:41:49 Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: not set Using server version 10.3.5-MariaDB mariabackup based on MariaDB server 10.3.5-MariaDB Linux (x86_64) mariabackup: uses posix_fadvise(). mariabackup: cd to /var/lib/mysql/ mariabackup: open files limit requested 0, set to 1024 mariabackup: using the following InnoDB configuration: mariabackup: innodb_data_home_dir = . mariabackup: innodb_data_file_path = ibdata1:12M:autoextend mariabackup: innodb_log_group_home_dir = ./ 2018-04-11 13:41:49 0 [Note] InnoDB: Number of pools: 1 180411 13:41:49 >> log scanned up to (51002181432) mariabackup: Generating a list of tablespaces 2018-04-11 13:41:49 0 [Warning] InnoDB: Allocated tablespace ID 127 for md_15252/BRc273a13dfd, old maximum was 0 180411 13:41:49 [01] Copying ./ibdata1 to /home/foo4/xtrabackup_backupfiles/ibdata1 180411 13:41:50 >> log scanned up to (51002181432) 180411 13:42:08 [01] ...done 180411 13:42:08 [01] Copying ./md_15252/BRc273a13dfd.ibd to /home/foo4/xtrabackup_backupfiles/md_15252/BRc273a13dfd.ibd 180411 13:42:08 [01] ...done ~~~~~~ full log attached 180411 13:43:12 [01] ...done 180411 13:43:12 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS... mariabackup: The latest check point (for incremental): '51002181423' mariabackup: Stopping log copying thread. 2018-04-11 13:43:12 0 [Note] InnoDB: Read redo log up to LSN=51002181632 180411 13:43:12 >> log scanned up to (51002181432) 180411 13:43:12 Executing UNLOCK TABLES 180411 13:43:12 All tables unlocked 180411 13:43:12 [00] Copying ib_buffer_pool to /home/foo4/xtrabackup_backupfiles/ib_buffer_pool 180411 13:43:12 [00] ...done 180411 13:43:12 Backup created in directory '/home/foo4/xtrabackup_backupfiles/' 180411 13:43:12 [00] Writing backup-my.cnf 180411 13:43:12 [00] ...done 180411 13:43:12 [00] Writing xtrabackup_info 180411 13:43:12 [00] ...done mariabackup: Redo log (from LSN 51002181423 to 51002181432) was copied. 180411 13:43:12 completed OK! [foo4@t4w6 ~]$

Anders Karlsson added a comment - 2018-04-11 14:07

Tried workaround with 10.2.14 and I got:
180411 16:06:19 Connecting to MySQL server host: localhost, user: root, password: not set, port: not set, socket: not set
Using server version 10.2.14-MariaDB-log
mariabackup based on MariaDB server 10.2.14-MariaDB Linux (x86_64)
mariabackup: uses posix_fadvise().
mariabackup: cd to /var/lib/mysql/
mariabackup: open files limit requested 0, set to 1024
mariabackup: using the following InnoDB configuration:
mariabackup: innodb_data_home_dir = .
mariabackup: innodb_data_file_path = ibdata1:12M:autoextend
mariabackup: innodb_log_group_home_dir = ./
2018-04-11 16:06:19 140510829865088 [Note] InnoDB: Number of pools: 1
2018-04-11 16:06:19 140510829865088 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be readable
2018-04-11 16:06:19 140510829865088 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be readable
mariabackup: could not find data files at the specified datadir
mariabackup: error: xb_load_tablespaces() failed with error Generic error.

Anders Karlsson added a comment - 2018-04-11 14:07 Tried workaround with 10.2.14 and I got: 180411 16:06:19 Connecting to MySQL server host: localhost, user: root, password: not set, port: not set, socket: not set Using server version 10.2.14-MariaDB-log mariabackup based on MariaDB server 10.2.14-MariaDB Linux (x86_64) mariabackup: uses posix_fadvise(). mariabackup: cd to /var/lib/mysql/ mariabackup: open files limit requested 0, set to 1024 mariabackup: using the following InnoDB configuration: mariabackup: innodb_data_home_dir = . mariabackup: innodb_data_file_path = ibdata1:12M:autoextend mariabackup: innodb_log_group_home_dir = ./ 2018-04-11 16:06:19 140510829865088 [Note] InnoDB: Number of pools: 1 2018-04-11 16:06:19 140510829865088 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be readable 2018-04-11 16:06:19 140510829865088 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be readable mariabackup: could not find data files at the specified datadir mariabackup: error: xb_load_tablespaces() failed with error Generic error.

Vladislav Vaintroub added a comment - 2018-04-11 14:09

it must be readable . mariabackup copies files. To copy a file, one needs to read a file.

Vladislav Vaintroub added a comment - 2018-04-11 14:09 it must be readable . mariabackup copies files. To copy a file, one needs to read a file.

Anders Karlsson added a comment - 2018-04-11 14:56

Yes, that is obvious, which is why I reported it, it doesn't even when the instructions are followed. I assume that you want

sudo usermod -aG mysql foo4
instead of
sudo usermod -aG wheel foo4

Anders Karlsson added a comment - 2018-04-11 14:56 Yes, that is obvious, which is why I reported it, it doesn't even when the instructions are followed. I assume that you want sudo usermod -aG mysql foo4 instead of sudo usermod -aG wheel foo4

Zdravelina Sokolovska (Inactive) added a comment - 2018-04-11 17:01

actually with ' usermod -aG wheel foo4 '
it's aimed to give to user 'foo4' sudo access – these details was done in the attached file :
edit the /etc/sudoers file and grant sudo access to users in the group wheel
then user may login and get mariabackup even w/o 'sudo'

Zdravelina Sokolovska (Inactive) added a comment - 2018-04-11 17:01 actually with ' usermod -aG wheel foo4 ' it's aimed to give to user 'foo4' sudo access – these details was done in the attached file : edit the /etc/sudoers file and grant sudo access to users in the group wheel then user may login and get mariabackup even w/o 'sudo'

Anders Karlsson added a comment - 2018-04-11 17:07

Ah, that was the sudoers thing, that was in the attached file but not in the text. Makes sense, but adding the mysql group to the foo4 user also seems to work. All in all, my main issue here wasn't how to run maricbackup but that just a simple error caused a core dump.

Anders Karlsson added a comment - 2018-04-11 17:07 Ah, that was the sudoers thing, that was in the attached file but not in the text. Makes sense, but adding the mysql group to the foo4 user also seems to work. All in all, my main issue here wasn't how to run maricbackup but that just a simple error caused a core dump.

Zdravelina Sokolovska (Inactive) added a comment - 2018-04-11 17:35

yes, thank you.
you should receive Error instead message about crash
the workaround is to
avoid the crash with [foo4@t4w6 ~]$ sudo chmod +s /usr/bin/mariabackup
avoid the the error with [foo4@t4w6 ~]$ sudo chmod g+r /var/lib/mysql/ibdata1
and if user is also given sudo access, it may just login and get mariabackup

Zdravelina Sokolovska (Inactive) added a comment - 2018-04-11 17:35 yes, thank you. you should receive Error instead message about crash the workaround is to avoid the crash with [foo4@t4w6 ~] $ sudo chmod +s /usr/bin/mariabackup avoid the the error with [foo4@t4w6 ~] $ sudo chmod g+r /var/lib/mysql/ibdata1 and if user is also given sudo access, it may just login and get mariabackup

Sergei Golubchik made changes - 2019-03-29 12:04

Fix Version/s

10.4 [ 22408 ]

Vladislav Vaintroub made changes - 2019-04-23 16:00

Assignee

Vladislav Vaintroub [ wlad ]

Vladislav Lesin [ vlad.lesin ]

Alice Sherepa made changes - 2019-08-19 14:40

Link

This issue is duplicated by ~~MDEV-20385~~ [ ~~MDEV-20385~~ ]

Julien Fritsch made changes - 2021-07-05 13:18

Assignee

Vladislav Lesin [ vlad.lesin ]

Vladislav Vaintroub [ wlad ]

Vladislav Vaintroub made changes - 2021-07-19 10:17

Assignee

Vladislav Vaintroub [ wlad ]

Julien Fritsch [ julien.fritsch ]

Julien Fritsch made changes - 2021-08-10 09:51

Assignee

Julien Fritsch [ julien.fritsch ]

Ralf Gebhardt made changes - 2021-08-12 09:59

Labels

crash

Ralf Gebhardt added a comment - 2021-08-12 10:01 - edited

serg, this issue is marked as priority minor, but results in a crash. Any chance that we get it fixed in a way that an error will be shown instead of a crash?

Ralf Gebhardt added a comment - 2021-08-12 10:01 - edited serg , this issue is marked as priority minor, but results in a crash. Any chance that we get it fixed in a way that an error will be shown instead of a crash?

Sergei Golubchik made changes - 2021-08-12 21:26

Link

This issue is blocked by ~~MDEV-13542~~ [ ~~MDEV-13542~~ ]

Sergei Golubchik added a comment - 2021-08-12 21:26

yes, when ~~MDEV-13542~~ is fixed.

Sergei Golubchik added a comment - 2021-08-12 21:26 yes, when MDEV-13542 is fixed.

Sergei Golubchik made changes - 2021-12-06 21:33

Workflow

MariaDB v3 [ 86451 ]

MariaDB v4 [ 140723 ]

Marko Mäkelä added a comment - 2022-06-07 14:59

I don’t think that any mariabackup --backup code was fixed in ~~MDEV-13542~~.
mariabackup --prepare yes, but not the backup part.

Marko Mäkelä added a comment - 2022-06-07 14:59 I don’t think that any mariabackup --backup code was fixed in MDEV-13542 . mariabackup --prepare yes, but not the backup part.

Marko Mäkelä made changes - 2022-06-07 14:59

Link

This issue is blocked by ~~MDEV-13542~~ [ ~~MDEV-13542~~ ]

Ralf Gebhardt made changes - 2022-08-04 08:40

Fix Version/s

10.2 [ 14601 ]

Julien Fritsch made changes - 2023-04-27 14:22

Fix Version/s

10.3 [ 22126 ]

MariaDB Server

Running mariabackup with insufficient system privileges cause crash / coredump

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

Git Integration