Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-27095

Simplify spider init queries

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Duplicate
    • 10.3(EOL), 10.4(EOL), 10.5, 10.6, 10.7(EOL), 10.11
    • N/A
    • Storage Engine - Spider

    Description

      The loading of the spider plugin causes a background thread to be created and the set of sql in storage/spider/spd_init_query.h to be run.

      Running mysql_upgrade at the same time the initialization of spider is occurring can cause problems as MDEV-27068 shows.

      One of the conflicts was

      • ALTER TABLE proc MODIFY
      • and spiders' {{ drop procedure}}

      Other similar conflicts can occur on plugin and udf function installation

      MDEV-19275 sql service for plugins means 10.7+ can simply this. Is it valid during initialization? Not 100% sure, but if it is, it would avoid the mariadb-upgrade conflicts.

      Other simplifications are:

      spider_fix_one_table looks like a simple modification of the ALTER TABLE syntax to use the column IF NOT EXISTS syntax of https://mariadb.com/kb/en/alter-table/ would negate the need for this function.

      mysql.spider_fix_system_tables, a procedure with zero arguments that sets user variables could be inline. Or better, removed and use the executable comment syntax in https://mariadb.com/kb/en/comment-syntax/#executable-comment-syntax to run statements based on the server version.

      Can the following be implemented another way install plugin [spider_rewrite soname 'ha_spider' (INSTALL PLUGIN has IF NOT EXISTS from 10.4)

      Can the udf functions created another way?

      Is there are reason (all of) this couldn't be part of mysql-upgrade? wrapped under conditional upgrade if spider is installed?

      Attachments

        Issue Links

          is blocked by

          Technical task - A technical task. MDEV-27107 prevent two mariadb-upgrade running in parallel

          • Major - Major loss of function.
          • Closed
          is part of

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22979 "mysqld --bootstrap" / mysql_install_db hangs when Spider is installed

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-19842 Crash while creating statistics for Spider table

          • Major - Major loss of function.
          • Closed

          Technical task - A technical task. MDEV-27998 Delete spider_table_crd_thread_count and spider_table_sts_thread_count

          • Major - Major loss of function.
          • Stalled

          Task - A task that needs to be done. MDEV-28553 Spider: clean up plugin initialization SQLs in spd_init_query.h

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-32804 Remove the (unimplemented and unused) spider_rewrite_plugin

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-14622 Running mysql_upgrade in background upon package installation or upgrade causes deadlocks and other side-effects

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-19850 per-plugin install/uninstall scriptlets

          • Major - Major loss of function.
          • Stalled

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27233 Server hangs when using --init-file which loads Spider and creates a Spider table

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            View 8 older comments
            ycp Yuchen Pei added a comment - - edited

            Thanks for the comment serg. As with the discussions at #spider, after refining on the first idea above, I think it is worth trying to re-interpret plugin-load-add=ha_spider as a "internal" file containing install soname ha_spider;. Here is a PoC: https://github.com/MariaDB/server/commit/603a4c218b7

            This way we can get rid of the bg thread completely. It also means we can decouple the present issue from MDEV-22979 and MDEV-27233 as by the time of executing the init queries in the internal file it will have all it needs. A clean-up of the queries is still useful though.

            ycp Yuchen Pei added a comment - - edited Thanks for the comment serg . As with the discussions at #spider, after refining on the first idea above, I think it is worth trying to re-interpret plugin-load-add=ha_spider as a "internal" file containing install soname ha_spider; . Here is a PoC: https://github.com/MariaDB/server/commit/603a4c218b7 This way we can get rid of the bg thread completely. It also means we can decouple the present issue from MDEV-22979 and MDEV-27233 as by the time of executing the init queries in the internal file it will have all it needs. A clean-up of the queries is still useful though.
            ycp Yuchen Pei added a comment -

            Hi holyfoot, PTAL thanks:

            The commit to fix the init bugs MDEV-22979 MDEV-27233 will be based on these ones

            ycp Yuchen Pei added a comment - Hi holyfoot , PTAL thanks: https://github.com/MariaDB/server/commit/27f33965ef0 https://github.com/MariaDB/server/commit/cb0e0c915f6 The commit to fix the init bugs MDEV-22979 MDEV-27233 will be based on these ones
            ycp Yuchen Pei added a comment - - edited

            I wrote a testcase with plugin-load-add=ha_spider in a .cnf file, and each time the server starts, spider gets loaded fine:

            # ~/source/mariadb-server/mdev-22979/src/storage/spider/mysql-test/spider/bugfix/t/plugin_load_add_repeatedly.test
            		 
             
            		 
            --echo #
            		 
            --echo # plugin-load-add=ha_spider
            		 
            --echo #
            		 
             
            		 
            --let SEARCH_PATTERN=\\[Note\\] Loading of SPIDER[a-zA-Z0-9_]* from ha_spider(|.so|.dll) is deferred to a temporary init file
            		 
            --let SEARCH_FILE=`select regexp_replace(@@tmpdir, '^.*/','')`
            		 
            --let SEARCH_FILE=$MYSQLTEST_VARDIR/log/$SEARCH_FILE.err
            		 
            --source include/search_pattern_in_file.inc
            		 
            select * from mysql.plugin;
            		 
            create table t (c int) Engine=SPIDER;
            		 
            --source include/restart_mysqld.inc
            		 
            --source include/search_pattern_in_file.inc
            		 
            drop table t;
            		 
             
            		 
            # ~/source/mariadb-server/mdev-22979/src/storage/spider/mysql-test/spider/bugfix/t/plugin_load_add_repeatedly.cnf
            		 
             
            		 
            [mysqld.1.1]
            		 
            plugin-load-add=ha_spider.so
            		 
             
            		 
            # the test result:
            		 
             
            		 
            #
            		 
            # plugin-load-add=ha_spider
            		 
            #
            		 
            FOUND 3 /\[Note\] Loading of SPIDER[a-zA-Z0-9_]* from ha_spider(|.so|.dll) is deferred to a temporary init file/ in mysqld.1.1.err
            		 
            select * from mysql.plugin;
            		 
            name	dl
            		 
            create table t (c int) Engine=SPIDER;
            		 
            # restart
            		 
            FOUND 6 /\[Note\] Loading of SPIDER[a-zA-Z0-9_]* from ha_spider(|.so|.dll) is deferred to a temporary init file/ in mysqld.1.1.err
            		 
            drop table t;

            ycp Yuchen Pei added a comment - - edited I wrote a testcase with plugin-load-add=ha_spider in a .cnf file, and each time the server starts, spider gets loaded fine: # ~/source/mariadb-server/mdev-22979/src/storage/spider/mysql-test/spider/bugfix/t/plugin_load_add_repeatedly.test   --echo # --echo # plugin-load-add=ha_spider --echo #   --let SEARCH_PATTERN=\\[Note\\] Loading of SPIDER[a-zA-Z0-9_]* from ha_spider(|.so|.dll) is deferred to a temporary init file --let SEARCH_FILE=`select regexp_replace(@@tmpdir, '^.*/','')` --let SEARCH_FILE=$MYSQLTEST_VARDIR/log/$SEARCH_FILE.err --source include/search_pattern_in_file.inc select * from mysql.plugin; create table t (c int) Engine=SPIDER; --source include/restart_mysqld.inc --source include/search_pattern_in_file.inc drop table t;   # ~/source/mariadb-server/mdev-22979/src/storage/spider/mysql-test/spider/bugfix/t/plugin_load_add_repeatedly.cnf   [mysqld.1.1] plugin-load-add=ha_spider.so   # the test result:   # # plugin-load-add=ha_spider # FOUND 3 /\[Note\] Loading of SPIDER[a-zA-Z0-9_]* from ha_spider(|.so|.dll) is deferred to a temporary init file/ in mysqld.1.1.err select * from mysql.plugin; name dl create table t (c int) Engine=SPIDER; # restart FOUND 6 /\[Note\] Loading of SPIDER[a-zA-Z0-9_]* from ha_spider(|.so|.dll) is deferred to a temporary init file/ in mysqld.1.1.err drop table t;
            holyfoot Alexey Botchkov added a comment -

            ok to push.
            It seems to me i already reviewed these patches. Hope i reviewed proper ones this time

            holyfoot Alexey Botchkov added a comment - ok to push. It seems to me i already reviewed these patches. Hope i reviewed proper ones this time
            ycp Yuchen Pei added a comment - - edited

            Thanks holyfoot. Yes we can consider this as part of MDEV-22979 as
            some of the patches there are named after this ticket. Marking as
            blocked by that ticket.

            Updates on 2023-10-06: close as duplicate to MDEV-22979.

            ycp Yuchen Pei added a comment - - edited Thanks holyfoot . Yes we can consider this as part of MDEV-22979 as some of the patches there are named after this ticket. Marking as blocked by that ticket. Updates on 2023-10-06: close as duplicate to MDEV-22979 .

            People

              ycp Yuchen Pei
              danblack Daniel Black
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.