Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-21312

MariaDB Server fails to start up on Debian

Details

    Description

      mariadb.service - MariaDB 10.3.20 database server
      Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
      Active: failed (Result: exit-code) since Tue 2019-12-10 19:08:14 CST; 27s ago
      Docs: man:mysqld(8)
      https://mariadb.com/kb/en/library/systemd/
      Process: 19440 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, stat>
      Process: 19459 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, sta>
      Process: 19462 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera>
      Process: 19511 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=ex>
      Main PID: 19511 (code=exited, status=227/NO_NEW_PRIVILEGES)
      lines 1-10/10 (END)

      Attachments

        Activity

          I noticed the same on our CI system yesterday:

          10.3 0a20e5ab77f8a6532b41ea2518626397059ccf42

           Process: 5250 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, signal=ABRT)
          

          http://buildbot.askmonty.org/buildbot/builders/kvm-deb-buster-amd64/builds/1441/steps/mtr/logs/stdio
          http://buildbot.askmonty.org/buildbot/builders/kvm-deb-disco-amd64/builds/1020/steps/mtr/logs/stdio

          For what it is worth, I develop InnoDB on Debian GNU/Linux Sid (unstable), but I never install the packages; I run tests directly from the build directory. The message suggests that this could have something to do with the Galera cluster.

          marko Marko Mäkelä added a comment - I noticed the same on our CI system yesterday: 10.3 0a20e5ab77f8a6532b41ea2518626397059ccf42 Process: 5250 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, signal=ABRT) http://buildbot.askmonty.org/buildbot/builders/kvm-deb-buster-amd64/builds/1441/steps/mtr/logs/stdio http://buildbot.askmonty.org/buildbot/builders/kvm-deb-disco-amd64/builds/1020/steps/mtr/logs/stdio For what it is worth, I develop InnoDB on Debian GNU/Linux Sid (unstable), but I never install the packages; I run tests directly from the build directory. The message suggests that this could have something to do with the Galera cluster.

          I am almost certain that this problem is specific to Galera 3 or WITH_WSREP=ON before MariaDB 10.4. The issue is not present in 10.4, where the Galera write-set replication (WSREP) changes were heavily refactored.

          I see that 10.3 is the first platform where these Debian builders are enabled. The problem could affect 10.1 and 10.2 as well, but I did not set fixVersion for those, because I am basically guessing until we have narrowed down the exact cause of the failure.

          marko Marko Mäkelä added a comment - I am almost certain that this problem is specific to Galera 3 or WITH_WSREP=ON before MariaDB 10.4. The issue is not present in 10.4, where the Galera write-set replication (WSREP) changes were heavily refactored. I see that 10.3 is the first platform where these Debian builders are enabled. The problem could affect 10.1 and 10.2 as well, but I did not set fixVersion for those, because I am basically guessing until we have narrowed down the exact cause of the failure.

          From logs I see

          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: Backtrace: (Note: toku_do_assert=0x0x7f4f90039f50)
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/lib/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x2e)[0x7f4f9003927e]
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x863b3)[0x7f4f900393b3]
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x86a3e)[0x7f4f90039a3e]
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x872fc)[0x7f4f9003a2fc]
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/lib/mysql/plugin/ha_tokudb.so(_Z26toku_os_huge_pages_enabledv+0x52)[0x7f4f9003c372]
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x13e2c5)[0x7f4f900f12c5]
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x68332)[0x7f4f9001b332]
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x68)[0x558b79640fe8]
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/sbin/mysqld(+0x5d6189)[0x558b79475189]
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/sbin/mysqld(_Z11plugin_initPiPPci+0x972)[0x558b79476392]
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/sbin/mysqld(+0x512afb)[0x558b793b1afb]
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x3fe)[0x558b793b7a4e]
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f4fab8e209b]
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/sbin/mysqld(_start+0x2a)[0x558b793aa84a]
          Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: Engine status function not available
          

          jplindst Jan Lindström (Inactive) added a comment - From logs I see Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: Backtrace: (Note: toku_do_assert=0x0x7f4f90039f50) Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/lib/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x2e)[0x7f4f9003927e] Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x863b3)[0x7f4f900393b3] Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x86a3e)[0x7f4f90039a3e] Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x872fc)[0x7f4f9003a2fc] Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/lib/mysql/plugin/ha_tokudb.so(_Z26toku_os_huge_pages_enabledv+0x52)[0x7f4f9003c372] Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x13e2c5)[0x7f4f900f12c5] Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x68332)[0x7f4f9001b332] Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x68)[0x558b79640fe8] Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/sbin/mysqld(+0x5d6189)[0x558b79475189] Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/sbin/mysqld(_Z11plugin_initPiPPci+0x972)[0x558b79476392] Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/sbin/mysqld(+0x512afb)[0x558b793b1afb] Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x3fe)[0x558b793b7a4e] Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f4fab8e209b] Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: /usr/sbin/mysqld(_start+0x2a)[0x558b793aa84a] Dec 12 12:44:34 debian-buster-amd64 mysqld[5250]: Engine status function not available

          d'd, can you please include some relevant output of journalctl -xe? We would like to see what exact error is preventing the server from starting up.

          On our CI system, it sometimes fails due to TokuDB:

          10.3 246e2ae12b514ed3010ffcf6473abbfd9f648340

          Dec 10 02:07:28 debian-buster-amd64 systemd[1]: Starting MariaDB 10.3.21 database server...
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10  2:07:29 0 [Note] /usr/sbin/mysqld (mysqld 10.3.21-MariaDB-1:10.3.21+maria~buster-log) starting as process 5248 ...
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10  2:07:29 0 [Warning] Plugin 'OQGRAPH' is of maturity level gamma while the server is stable
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: Transparent huge pages are enabled, according to /sys/kernel/mm/transparent_hugepage/enabled
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: Transparent huge pages appear to be enabled according to mincore()
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10  2:07:29 0 [ERROR] TokuDB: Huge pages are enabled, disable them before continuing
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10  2:07:29 0 [ERROR] Plugin 'TokuDB' init function returned error.
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10  2:07:29 0 [ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed.
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 191210  2:07:29 [ERROR] mysqld got signal 11 ;
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: This could be because you hit a bug. It is also possible that this binary
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: or one of the libraries it was linked against is corrupt, improperly built,
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: or misconfigured. This error can also be caused by malfunctioning hardware.
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: We will try our best to scrape up some info that will hopefully help
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: diagnose the problem, but since we have already crashed,
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: something is definitely wrong and this may fail.
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: Server version: 10.3.21-MariaDB-1:10.3.21+maria~buster-log
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: key_buffer_size=134217728
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: read_buffer_size=2097152
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: max_used_connections=0
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: max_threads=102
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: thread_count=20
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: It is possible that mysqld could use up to
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760035 K  bytes of memory
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: Hope that's ok; if not, decrease some variables in the equation.
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: Thread pointer: 0x0
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: Attempting backtrace. You can use the following information to find out
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: where mysqld died. If you see no messages after this, something went
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: terribly wrong...
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10  2:07:29 0 [Note] InnoDB: Using Linux native AIO
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10  2:07:29 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10  2:07:29 0 [Note] InnoDB: Uses event mutexes
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10  2:07:29 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10  2:07:29 0 [Note] InnoDB: Number of pools: 1
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10  2:07:29 0 [Note] InnoDB: Using generic crc32 instructions
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10  2:07:29 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
          Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10  2:07:29 0 [Note] InnoDB: Completed initialization of buffer pool
          Dec 10 02:08:59 debian-buster-amd64 systemd[1]: mariadb.service: Start operation timed out. Terminating.
          Dec 10 02:10:29 debian-buster-amd64 systemd[1]: mariadb.service: State 'stop-sigterm' timed out. Skipping SIGKILL.
          Dec 10 02:11:59 debian-buster-amd64 systemd[1]: mariadb.service: State 'stop-final-sigterm' timed out. Skipping SIGKILL. Entering failed mode.
          Dec 10 02:11:59 debian-buster-amd64 systemd[1]: mariadb.service: Failed with result 'timeout'.
          Dec 10 02:11:59 debian-buster-amd64 systemd[1]: Failed to start MariaDB 10.3.21 database server.
          

          http://buildbot.askmonty.org/buildbot/builders/kvm-deb-buster-amd64/builds/1430/steps/mtr/logs/syslog

          We seem to have some incorrect error handling there. Why do we attempt to start further storage engines after reporting a fatal signal?

          marko Marko Mäkelä added a comment - d'd , can you please include some relevant output of journalctl -xe ? We would like to see what exact error is preventing the server from starting up. On our CI system, it sometimes fails due to TokuDB: 10.3 246e2ae12b514ed3010ffcf6473abbfd9f648340 Dec 10 02:07:28 debian-buster-amd64 systemd[1]: Starting MariaDB 10.3.21 database server... Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10 2:07:29 0 [Note] /usr/sbin/mysqld (mysqld 10.3.21-MariaDB-1:10.3.21+maria~buster-log) starting as process 5248 ... Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10 2:07:29 0 [Warning] Plugin 'OQGRAPH' is of maturity level gamma while the server is stable Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: Transparent huge pages are enabled, according to /sys/kernel/mm/transparent_hugepage/enabled Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: Transparent huge pages appear to be enabled according to mincore() Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10 2:07:29 0 [ERROR] TokuDB: Huge pages are enabled, disable them before continuing … Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10 2:07:29 0 [ERROR] Plugin 'TokuDB' init function returned error. Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10 2:07:29 0 [ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed. Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 191210 2:07:29 [ERROR] mysqld got signal 11 ; Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: This could be because you hit a bug. It is also possible that this binary Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: or one of the libraries it was linked against is corrupt, improperly built, Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: or misconfigured. This error can also be caused by malfunctioning hardware. Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: We will try our best to scrape up some info that will hopefully help Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: diagnose the problem, but since we have already crashed, Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: something is definitely wrong and this may fail. Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: Server version: 10.3.21-MariaDB-1:10.3.21+maria~buster-log Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: key_buffer_size=134217728 Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: read_buffer_size=2097152 Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: max_used_connections=0 Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: max_threads=102 Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: thread_count=20 Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: It is possible that mysqld could use up to Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760035 K bytes of memory Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: Hope that's ok; if not, decrease some variables in the equation. Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: Thread pointer: 0x0 Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: Attempting backtrace. You can use the following information to find out Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: where mysqld died. If you see no messages after this, something went Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: terribly wrong... Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10 2:07:29 0 [Note] InnoDB: Using Linux native AIO Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10 2:07:29 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10 2:07:29 0 [Note] InnoDB: Uses event mutexes Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10 2:07:29 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10 2:07:29 0 [Note] InnoDB: Number of pools: 1 Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10 2:07:29 0 [Note] InnoDB: Using generic crc32 instructions Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10 2:07:29 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M Dec 10 02:07:29 debian-buster-amd64 mysqld[5248]: 2019-12-10 2:07:29 0 [Note] InnoDB: Completed initialization of buffer pool Dec 10 02:08:59 debian-buster-amd64 systemd[1]: mariadb.service: Start operation timed out. Terminating. Dec 10 02:10:29 debian-buster-amd64 systemd[1]: mariadb.service: State 'stop-sigterm' timed out. Skipping SIGKILL. Dec 10 02:11:59 debian-buster-amd64 systemd[1]: mariadb.service: State 'stop-final-sigterm' timed out. Skipping SIGKILL. Entering failed mode. Dec 10 02:11:59 debian-buster-amd64 systemd[1]: mariadb.service: Failed with result 'timeout'. Dec 10 02:11:59 debian-buster-amd64 systemd[1]: Failed to start MariaDB 10.3.21 database server. http://buildbot.askmonty.org/buildbot/builders/kvm-deb-buster-amd64/builds/1430/steps/mtr/logs/syslog We seem to have some incorrect error handling there. Why do we attempt to start further storage engines after reporting a fatal signal?

          I found an older example. This occurred in the 10.3 branch before the mariadb-10.3.20 release was tagged:

          10.3 d759f764f6b115ca7ad7644ae3ee2e19c71c6138

          Dec  5 09:08:48 debian-buster-amd64 systemd[1]: Starting MariaDB 10.3.21 database server...
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: 2019-12-05  9:08:49 0 [Note] /usr/sbin/mysqld (mysqld 10.3.21-MariaDB-1:10.3.21+maria~buster-log) starting as process 5217 ...
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: 2019-12-05  9:08:49 0 [Warning] Plugin 'OQGRAPH' is of maturity level gamma while the server is stable
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: Transparent huge pages are enabled, according to /sys/kernel/mm/transparent_hugepage/enabled
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /home/buildbot/buildbot/build/mariadb-10.3.21/storage/tokudb/PerconaFT/portability/huge_page_detection.cc:110 check_huge_pages_in_practice: Assertion `!vec[i]' failed (errno=2)
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: : No such file or directory
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: Backtrace: (Note: toku_do_assert=0x0x7f041db2ef50)
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x2e)[0x7f041db2e27e]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x863b3)[0x7f041db2e3b3]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x86a3e)[0x7f041db2ea3e]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x872fc)[0x7f041db2f2fc]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(_Z26toku_os_huge_pages_enabledv+0x52)[0x7f041db31372]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x13e2c5)[0x7f041dbe62c5]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x68332)[0x7f041db10332]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x68)[0x5573a28d0a58]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(+0x5c91e9)[0x5573a27051e9]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_Z11plugin_initPiPPci+0x972)[0x5573a27063f2]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(+0x505b6b)[0x5573a2641b6b]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x3fe)[0x5573a2647abe]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f043939309b]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_start+0x2a)[0x5573a263a8ba]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: Engine status function not available
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: Memory usage:
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: Arena 0:
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: system bytes     =          0
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: in use bytes     =          0
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: Total (incl. mmap):
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: system bytes     =          0
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: in use bytes     =          0
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: max mmap regions =          0
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: max mmap bytes   =          0
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: 191205  9:08:49 [ERROR] mysqld got signal 6 ;
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: This could be because you hit a bug. It is also possible that this binary
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: or one of the libraries it was linked against is corrupt, improperly built,
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: or misconfigured. This error can also be caused by malfunctioning hardware.
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: We will try our best to scrape up some info that will hopefully help
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: diagnose the problem, but since we have already crashed,
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: something is definitely wrong and this may fail.
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: Server version: 10.3.21-MariaDB-1:10.3.21+maria~buster-log
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: key_buffer_size=134217728
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: read_buffer_size=2097152
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: max_used_connections=0
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: max_threads=102
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: thread_count=20
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: It is possible that mysqld could use up to
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760035 K  bytes of memory
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: Hope that's ok; if not, decrease some variables in the equation.
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: Thread pointer: 0x0
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: Attempting backtrace. You can use the following information to find out
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: where mysqld died. If you see no messages after this, something went
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: terribly wrong...
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: stack_bottom = 0x0 thread_stack 0x49000
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5573a2db4cce]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(handle_fatal_signal+0x54d)[0x5573a28ce4bd]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f0439f5e730]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f04393a67bb]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f0439391535]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x863b8)[0x7f041db2e3b8]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x86a3e)[0x7f041db2ea3e]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x872fc)[0x7f041db2f2fc]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(_Z26toku_os_huge_pages_enabledv+0x52)[0x7f041db31372]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x13e2c5)[0x7f041dbe62c5]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x68332)[0x7f041db10332]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x68)[0x5573a28d0a58]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(+0x5c91e9)[0x5573a27051e9]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_Z11plugin_initPiPPci+0x972)[0x5573a27063f2]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(+0x505b6b)[0x5573a2641b6b]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x3fe)[0x5573a2647abe]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f043939309b]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_start+0x2a)[0x5573a263a8ba]
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: information that should help you find out what is causing the crash.
          Dec  5 09:08:49 debian-buster-amd64 mysqld[5217]: Writing a core file...
          Dec  5 09:08:49 debian-buster-amd64 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
          Dec  5 09:08:49 debian-buster-amd64 systemd[1]: mariadb.service: Failed with result 'signal'.
          Dec  5 09:08:49 debian-buster-amd64 systemd[1]: Failed to start MariaDB 10.3.21 database server.
          

          This time it crashed ‘properly’ without trying to start further storage engines, and without timeout.

          But, we still do not know whether TokuDB caused the same failure that was reported by d'd on ARM.

          marko Marko Mäkelä added a comment - I found an older example . This occurred in the 10.3 branch before the mariadb-10.3.20 release was tagged: 10.3 d759f764f6b115ca7ad7644ae3ee2e19c71c6138 Dec 5 09:08:48 debian-buster-amd64 systemd[1]: Starting MariaDB 10.3.21 database server... Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: 2019-12-05 9:08:49 0 [Note] /usr/sbin/mysqld (mysqld 10.3.21-MariaDB-1:10.3.21+maria~buster-log) starting as process 5217 ... Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: 2019-12-05 9:08:49 0 [Warning] Plugin 'OQGRAPH' is of maturity level gamma while the server is stable Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: Transparent huge pages are enabled, according to /sys/kernel/mm/transparent_hugepage/enabled Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /home/buildbot/buildbot/build/mariadb-10.3.21/storage/tokudb/PerconaFT/portability/huge_page_detection.cc:110 check_huge_pages_in_practice: Assertion `!vec[i]' failed (errno=2) Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: : No such file or directory Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: Backtrace: (Note: toku_do_assert=0x0x7f041db2ef50) Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x2e)[0x7f041db2e27e] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x863b3)[0x7f041db2e3b3] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x86a3e)[0x7f041db2ea3e] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x872fc)[0x7f041db2f2fc] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(_Z26toku_os_huge_pages_enabledv+0x52)[0x7f041db31372] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x13e2c5)[0x7f041dbe62c5] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x68332)[0x7f041db10332] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x68)[0x5573a28d0a58] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(+0x5c91e9)[0x5573a27051e9] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_Z11plugin_initPiPPci+0x972)[0x5573a27063f2] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(+0x505b6b)[0x5573a2641b6b] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x3fe)[0x5573a2647abe] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f043939309b] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_start+0x2a)[0x5573a263a8ba] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: Engine status function not available Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: Memory usage: Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: Arena 0: Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: system bytes = 0 Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: in use bytes = 0 Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: Total (incl. mmap): Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: system bytes = 0 Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: in use bytes = 0 Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: max mmap regions = 0 Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: max mmap bytes = 0 Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: 191205 9:08:49 [ERROR] mysqld got signal 6 ; Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: This could be because you hit a bug. It is also possible that this binary Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: or one of the libraries it was linked against is corrupt, improperly built, Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: or misconfigured. This error can also be caused by malfunctioning hardware. Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: We will try our best to scrape up some info that will hopefully help Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: diagnose the problem, but since we have already crashed, Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: something is definitely wrong and this may fail. Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: Server version: 10.3.21-MariaDB-1:10.3.21+maria~buster-log Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: key_buffer_size=134217728 Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: read_buffer_size=2097152 Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: max_used_connections=0 Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: max_threads=102 Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: thread_count=20 Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: It is possible that mysqld could use up to Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760035 K bytes of memory Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: Hope that's ok; if not, decrease some variables in the equation. Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: Thread pointer: 0x0 Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: Attempting backtrace. You can use the following information to find out Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: where mysqld died. If you see no messages after this, something went Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: terribly wrong... Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: stack_bottom = 0x0 thread_stack 0x49000 Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5573a2db4cce] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(handle_fatal_signal+0x54d)[0x5573a28ce4bd] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f0439f5e730] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f04393a67bb] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f0439391535] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x863b8)[0x7f041db2e3b8] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x86a3e)[0x7f041db2ea3e] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x872fc)[0x7f041db2f2fc] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(_Z26toku_os_huge_pages_enabledv+0x52)[0x7f041db31372] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x13e2c5)[0x7f041dbe62c5] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/lib/mysql/plugin/ha_tokudb.so(+0x68332)[0x7f041db10332] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x68)[0x5573a28d0a58] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(+0x5c91e9)[0x5573a27051e9] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_Z11plugin_initPiPPci+0x972)[0x5573a27063f2] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(+0x505b6b)[0x5573a2641b6b] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x3fe)[0x5573a2647abe] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f043939309b] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: /usr/sbin/mysqld(_start+0x2a)[0x5573a263a8ba] Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: information that should help you find out what is causing the crash. Dec 5 09:08:49 debian-buster-amd64 mysqld[5217]: Writing a core file... … Dec 5 09:08:49 debian-buster-amd64 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT Dec 5 09:08:49 debian-buster-amd64 systemd[1]: mariadb.service: Failed with result 'signal'. Dec 5 09:08:49 debian-buster-amd64 systemd[1]: Failed to start MariaDB 10.3.21 database server. This time it crashed ‘properly’ without trying to start further storage engines, and without timeout. But, we still do not know whether TokuDB caused the same failure that was reported by d'd on ARM.

          d'd, thank you for the screenshots. Unfortunately, I do not see any MariaDB server error log output among them. If the server crashes before writing any log, then that crash should be different from the TokuDB-related crash that we occasionally see on our CI system. That one will hopefully be addressed by disabling a check.

          d'd, what happens if you try to start up the server manually, by invoking /usr/sbin/mysqld? I tested the following steps with the default settings of the mariadb-server-10.3 package (version 10.3.20-1) on my Debian GNU/Linux AMD64 system:

          sudo service mariadb stop
          sudo su -s /bin/bash - mysql
          gdb /usr/sbin/mysqld
          

          and then in gdb, execute the following:

          run
          backtrace
          quit
          

          Note: if the server startup does not fail, you can initiate shutdown from another connection, or send SIGQUIT to the process, which is usually mapped to ctrl-\.

          Another option would be to run

          sudo su -s /bin/bash - mysql -c 'strace /usr/sbin/mysqld'
          

          In any case, please try to attach the output as text, not as screenshots.

          marko Marko Mäkelä added a comment - d'd , thank you for the screenshots. Unfortunately, I do not see any MariaDB server error log output among them. If the server crashes before writing any log, then that crash should be different from the TokuDB-related crash that we occasionally see on our CI system. That one will hopefully be addressed by disabling a check . d'd , what happens if you try to start up the server manually, by invoking /usr/sbin/mysqld ? I tested the following steps with the default settings of the mariadb-server-10.3 package (version 10.3.20-1) on my Debian GNU/Linux AMD64 system: sudo service mariadb stop sudo su -s /bin/bash - mysql gdb /usr/sbin/mysqld and then in gdb, execute the following: run backtrace quit Note: if the server startup does not fail, you can initiate shutdown from another connection, or send SIGQUIT to the process, which is usually mapped to ctrl-\. Another option would be to run sudo su -s /bin/bash - mysql -c 'strace /usr/sbin/mysqld' In any case, please try to attach the output as text, not as screenshots.
          serg Sergei Golubchik added a comment - - edited

          d'd, note the line in your logs

          Main PID: 13381 (code=exited, status=227/NO_NEW_PRIVILEGES)
          

          there are some hints (e.g. here) that this relates to the old kernel, SELinux, and NoNewPrivileges line in the mariadb.service file. In that case you should be able to start mysqld directly, but not via systemd unless you upgrade your kernel or somehow fix the service file.

          serg Sergei Golubchik added a comment - - edited d'd , note the line in your logs Main PID: 13381 (code=exited, status=227/NO_NEW_PRIVILEGES) there are some hints (e.g. here ) that this relates to the old kernel, SELinux, and NoNewPrivileges line in the mariadb.service file. In that case you should be able to start mysqld directly, but not via systemd unless you upgrade your kernel or somehow fix the service file.

          People

            Unassigned Unassigned
            d'd aa chen
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.