Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-22979

"mysqld --bootstrap" / mysql_install_db hangs when Spider is installed

Details

    Description

      If Spider is installed, then mysqld --bootstrap will hang when you run mysql_install_db. The hang looks similar to MDEV-20179.

      For example, you can see that Spider is installed:

      $ sudo cat /etc/mysql/mariadb.conf.d/spider.cnf 
      [mariadb]
      		 
      plugin-load-add=ha_spider.so

      I stop the service and delete the datadir:

      $ sudo systemctl stop mariadb
      		 
      $ sudo rm -fr /var/lib/mysql/*

      And then I run mysql_install_db:

      $ sudo mysql_install_db --user=mysql --basedir=/usr/ --datadir=/var/lib/mysql/

      This hangs with the following output:

      Installing MariaDB/MySQL system tables in '/var/lib/mysql/' ...
      		 
      2020-06-22 23:43:17 0 [Warning] Plugin 'SPIDER' is of maturity level gamma while the server is stable

      The stacktrace of the hang is:

      Thread 4 (Thread 0x7f2fcc0ac700 (LWP 18846)):
      		 
      #0  0x00007f2fd5c6a26c in __GI___sigtimedwait (set=<optimized out>, info=0x7f2fcc0abd30, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:42
      		 
      #1  0x00005627f9ac217b in signal_hand ()
      		 
      #2  0x00005627f9fc7a0a in ?? ()
      		 
      #3  0x00007f2fd694d6db in start_thread (arg=0x7f2fcc0ac700) at pthread_create.c:463
      		 
      #4  0x00007f2fd5d4b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
      		 
       
      		 
      Thread 3 (Thread 0x7f2fc7fff700 (LWP 18819)):
      		 
      #0  0x00007f2fd69539f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5627fadfbd48 <COND_server_started+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      		 
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x5627fadfbd60 <LOCK_server_started>, cond=0x5627fadfbd20 <COND_server_started>) at pthread_cond_wait.c:502
      		 
      #2  __pthread_cond_wait (cond=0x5627fadfbd20 <COND_server_started>, mutex=0x5627fadfbd60 <LOCK_server_started>) at pthread_cond_wait.c:655
      		 
      #3  0x00007f2fcc39732b in spider_table_bg_sts_action(void*) () from /usr/lib/mysql/plugin/ha_spider.so
      		 
      #4  0x00007f2fd694d6db in start_thread (arg=0x7f2fc7fff700) at pthread_create.c:463
      		 
      #5  0x00007f2fd5d4b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
      		 
       
      		 
      Thread 2 (Thread 0x7f2fd80cc700 (LWP 18818)):
      		 
      #0  0x00007f2fd6953f85 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f2fd80cbdb0, expected=0, futex_word=0x5627fb6da0ac) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      		 
      #1  __pthread_cond_wait_common (abstime=0x7f2fd80cbdb0, mutex=0x5627fb6da0c0, cond=0x5627fb6da080) at pthread_cond_wait.c:539
      		 
      #2  __pthread_cond_timedwait (cond=0x5627fb6da080, mutex=0x5627fb6da0c0, abstime=0x7f2fd80cbdb0) at pthread_cond_wait.c:667
      		 
      #3  0x00005627fa3980b2 in ?? ()
      		 
      #4  0x00007f2fd694d6db in start_thread (arg=0x7f2fd80cc700) at pthread_create.c:463
      		 
      #5  0x00007f2fd5d4b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
      		 
       
      		 
      Thread 1 (Thread 0x7f2fd8247800 (LWP 18815)):
      		 
      #0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
      		 
      #1  0x00007f2fd6950154 in __GI___pthread_mutex_lock (mutex=0x5627fd060680) at ../nptl/pthread_mutex_lock.c:133
      		 
      #2  0x00007f2fcc39bbd8 in spider_free_sts_threads(st_spider_thread*) () from /usr/lib/mysql/plugin/ha_spider.so
      		 
      #3  0x00007f2fcc39c096 in spider_db_done(void*) () from /usr/lib/mysql/plugin/ha_spider.so
      		 
      #4  0x00005627f9d8b800 in ha_finalize_handlerton(st_plugin_int*) ()
      		 
      #5  0x00005627f9b98575 in ?? ()
      		 
      #6  0x00005627f9b9c47e in ?? ()
      		 
      #7  0x00005627f9b9d265 in plugin_shutdown() ()
      		 
      #8  0x00005627f9ac1028 in ?? ()
      		 
      #9  0x00005627f9ac472a in unireg_abort ()
      		 
      #10 0x00005627f9acd1ba in mysqld_main(int, char**) ()
      		 
      #11 0x00007f2fd5c4bb97 in __libc_start_main (main=0x5627f9a8e4b0 <main>, argc=12, argv=0x7ffed856ce28, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffed856ce18) at ../csu/libc-start.c:310
      		 
      #12 0x00005627f9abfe1a in _start ()

      Attachments

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27233 Server hangs when using --init-file which loads Spider and creates a Spider table

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-27321 test cross- and upgrades on debian with spider being in mariadb-server-10.5

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28218 Spider: thread hang/deadlock as result of INSTALL PLUGIN and DROP TABLE

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-30722 sporadic failure spider/bugfix.insert_select

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-31101 spider/bugfix.mdev_29904 fails with "Server shutdown in progress" in CI

          • Major - Major loss of function.
          • Closed
          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-32485 Upgrading spider fails after the init bug fix due to insertion of duplicate rows into mysql_func

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-32507 Spider tests with init-file using $MYSQL_TEST_DIR fails in certain CIs due to different directory structure

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-32515 The test spider/bugfix.mdev_30370 fails with "98: Address already in use"

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-32532 Assertion failure in ddl_log_increment_phase_no_lock upon partition operations with spider plugin loaded

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Task - A task that needs to be done. MDEV-32559 Move alter table statements in spider init queries to be executed in the signal_ddl_recovery_done callback

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-32683 Spider engine does not load with non-default alter-algorithm

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-32753 Spider engine does not load in ORACLE mode

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23271 mysql_install_db freeze using config with plugin_load_add=ha_spider.so

          • Major - Major loss of function.
          • Closed
          includes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27095 Simplify spider init queries

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is blocked by

          Task - A task that needs to be done. MDEV-27595 Backport SQL service, introduced by MDEV-19275

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-31400 Introduce plugin startup dependency

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-30192 Attempting to load ha_spider.so as an option to mariadb-install-db will cause the same to hang

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-19842 Crash while creating statistics for Spider table

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27103 mariadb-upgrade fails with 'System table spider_tables is different version' => Can't create database 'performance_schema'

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-31401 Allow CREATE FUNCTION to run before udf_init()

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-32559 Move alter table statements in spider init queries to be executed in the signal_ddl_recovery_done callback

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-19850 per-plugin install/uninstall scriptlets

          • Major - Major loss of function.
          • Stalled

          Bug - A problem which impairs or prevents the functions of the product. MDEV-20179 Server hangs on shutdown during installation of Spider

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-30370 mariadbd hangs when running with --wsrep-recover and --plugin-load-add=ha_spider.so

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            View 47 older comments
            holyfoot Alexey Botchkov added a comment -

            ok to push.

            holyfoot Alexey Botchkov added a comment - ok to push.
            ycp Yuchen Pei added a comment -

            Thanks for the review holyfoot. Will push this once the blocking
            issue MDEV-31400 is done.

            ycp Yuchen Pei added a comment - Thanks for the review holyfoot . Will push this once the blocking issue MDEV-31400 is done.
            ycp Yuchen Pei added a comment - - edited

            The commit for the blocking issue MDEV-31400 has been pushed to 10.4.
            Now because of the SQL service, we need to wait for either the change
            to bubble to 10.9 or (better yet) holyfoot to fix MDEV-27595 before
            pushing the commit for this ticket.

            ycp Yuchen Pei added a comment - - edited The commit for the blocking issue MDEV-31400 has been pushed to 10.4. Now because of the SQL service, we need to wait for either the change to bubble to 10.9 or (better yet) holyfoot to fix MDEV-27595 before pushing the commit for this ticket.
            ycp Yuchen Pei added a comment -

            The push is blocked by the unstable test spider/bugfix.mdev_27240:

            https://buildbot.mariadb.org/#/builders/221/builds/26142
            https://buildbot.mariadb.org/#/builders/576/builds/3104

            so I'm marking MDEV-32046 as blocking this issue.

            ycp Yuchen Pei added a comment - The push is blocked by the unstable test spider/bugfix.mdev_27240: https://buildbot.mariadb.org/#/builders/221/builds/26142 https://buildbot.mariadb.org/#/builders/576/builds/3104 so I'm marking MDEV-32046 as blocking this issue.
            ycp Yuchen Pei added a comment - - edited

            Pushed the following to 10.10

            ef14d6d6a42 @ upstream/bb-10.10-mdev-22979 upstream/10.10 MDEV-32046 Adding ER_NET_READ_ERROR to spider/bugfix.mdev_27240
            		 
            a60cf9c7ae4 MDEV-22979 MDEV-27233 MDEV-28218 Fixing spider init bugs
            		 
            c6ba81d6bf6 MDEV-27095 clean up spd_init_query.h
            		 
            fc2548c862b MDEV-27095 installing one spider plugin should not trigger others

            Re-opened MDEV-29870 for 10.4-10.6

            ycp Yuchen Pei added a comment - - edited Pushed the following to 10.10 ef14d6d6a42 @ upstream/bb-10.10-mdev-22979 upstream/10.10 MDEV-32046 Adding ER_NET_READ_ERROR to spider/bugfix.mdev_27240 a60cf9c7ae4 MDEV-22979 MDEV-27233 MDEV-28218 Fixing spider init bugs c6ba81d6bf6 MDEV-27095 clean up spd_init_query.h fc2548c862b MDEV-27095 installing one spider plugin should not trigger others Re-opened MDEV-29870 for 10.4-10.6

            People

              ycp Yuchen Pei
              GeoffMontee Geoff Montee (Inactive)
              Votes:
              3 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.