[MDEV-19917] Spider doesn't remove itself from mysql.plugin upon uninstallation Created: 2019-07-01  Updated: 2020-06-10  Resolved: 2020-06-10

Status: Closed
Project: MariaDB Server
Component/s: Platform Debian, Storage Engine - Spider
Affects Version/s: 10.2, 10.3, 10.4
Fix Version/s: 10.5.4

Type: Bug Priority: Minor
Reporter: Elena Stepanova Assignee: Anel Husakovic
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-19918 Server hangs or crashes while trying ... Closed

 Description   

When mariadb-plugin-spider package is installed, it runs install_spider.sql script which installs two plugins into mysql.plugin. Apparently, there is no reverse operation, so when the package is uninstalled and the library is removed, orphan entries remain in mysql.plugin

+------------------+--------------+
 
| name             | dl           |
 
+------------------+--------------+
 
| spider           | ha_spider.so |
 
| spider_alloc_mem | ha_spider.so |
 
+------------------+--------------+



 Comments   
Comment by Otto Kekäläinen [ 2020-02-24 ]

Indeed, there is a mariadb-plugin-spider.postinst but no prerm/postrm. I will fix this both in upstream MariaDB and in Debian.

Comment by Otto Kekäläinen [ 2020-03-14 ]

The reason there is no reverse operation in the Debian packaging is that the source itself does not have any reverse SQL script, just install:

mariadb-10.5$ find . | grep spider.sql
 
./storage/spider/scripts/install_spider.sql

Comment by Otto Kekäläinen [ 2020-04-14 ]

Or should we be loading this via a configuration option, just like e.g. Connect plugin does?

cat mariadb.conf.d/connect.cnf
 
 
 
[mariadb]
 
plugin-load-add=ha_connect.so

Comment by Otto Kekäläinen [ 2020-04-14 ]

The mroonga install.sql and uninstall.sql seem to execute fancy functions and procedures, but to me the spider_install.sql is just about adding the plugin name to plugins table, so we might as well install it just like connect.cnf with a config file. This however would require CMake coding. I did not even find out that CMake code exactly generates the connect.cnf. Could anel help here?

This could be expanded to:
1. Ensure all mariadb-plugin-* packages ship a <plugin>.cnf that activates with with a appropriate `plugin-load-add` line (and thus automatically de-activates when the plugin is uninstalled, which was the root cause issue in this bug report). Plugins like connect, oqgraph and rocksdb already do this.
AND
2. Ensure we have some post install tests to verify that the plugin indeed did install and that there is some plugin specific function that works (and would only work if plugin installation was successful). Current MTR is "flawed" in the sense that it runs certain plugin tests only if that plugin was present (.so for it loaded) and thus is just skips tests if plugin was not present, when we want install tests that visibly fail if plugin installation did not work.

Comment by Otto Kekäläinen [ 2020-04-26 ]

anel Can you figure our what generates connect.cnf or oqgraph.cnf? What would it take to generate spider.cnf with similar contents?

Comment by Daniel Black [ 2020-04-28 ]

https://github.com/MariaDB/server/blob/10.5/cmake/plugin.cmake#L252

Take a look at `cmake --trace -DPLUGIN_SPIDER=DYNAMIC .` to see why it isn't included. Seems to fail this condition:

cmake/plugin.cmake(235):  IF(CPACK_COMPONENTS_ALL AND NOT CPACK_COMPONENTS_ALL MATCHES ${ARG_COMPONENT} AND INSTALL_SYSCONF2DIR )

Comment by Anel Husakovic [ 2020-04-28 ]

otto will take a look currently working on other thing. Could a workaround be the same as for mroonga?
danblack thank you for hint will investigate ASAP .

Comment by Otto Kekäläinen [ 2020-05-17 ]

anel Did you debug this and find a way how we get Spider to generate a nice spider.cnf and can skip the install/uninstall steps?

Comment by Anel Husakovic [ 2020-05-19 ]

otto replied on zulip.

Comment by Anel Husakovic [ 2020-05-19 ]

Here is the debugging of spider and connect for cmake/plugin.cmake file (redirection to a file doesn't work for --trace only to stdout and it is quite hard to inspect this (if there is no other hack to observe?))

cmake ../../10.2/ -DCMAKE_BUILD_TYPE=Debug --trace --trace-expand -DPLUGIN_SPIDER=DYNAMIC -DWITH_EMBEDDED_SERVER=OFF -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,PERFSCHEMA,SPHINX}=NO --trace-source="cmake/plugin.cmake" #>t.txt not working

Highlighted are difference in cmake variables ARG_COMPONENT (this variable get set in cmake/install_macros.cmake) and ARG_MODULE_OUTPUT_NAME (don't know where this variable gets set?).
Also don't why danblack mentioned the line 235 (222 in my example) as potential root, we can see below it will not be executed in case of connect which uses connect.cnf. So still is open question for connect.cnf how is used?
todo: debugging of cmake/install_macros.cmake and CMakeLists.txt for connect and spider.

Connect SE:
 
 
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(273):  IF(EXISTS /home/anel/mariadb/10.2/storage/connect/CMakeLists.txt )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(274):  ADD_SUBDIRECTORY(/home/anel/mariadb/10.2/storage/connect )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(32):  CMAKE_PARSE_ARGUMENTS(ARG STORAGE_ENGINE;STATIC_ONLY;MODULE_ONLY;MANDATORY;DEFAULT;DISABLED;RECOMPILE_FOR_EMBEDDED;CLIENT MODULE_OUTPUT_NAME;STATIC_OUTPUT_NAME;COMPONENT;CONFIG LINK_LIBRARIES;DEPENDENCIES connect;ha_connect.cc;connect.cc;user_connect.cc;mycat.cc;fmdlex.c;osutil.c;rcmsg.c;rcmsg.h;array.cpp;blkfil.cpp;colblk.cpp;csort.cpp;filamap.cpp;filamdbf.cpp;filamfix.cpp;filamgz.cpp;filamtxt.cpp;filter.cpp;json.cpp;jsonudf.cpp;maputil.cpp;myconn.cpp;myutil.cpp;plgdbutl.cpp;plugutil.cpp;reldef.cpp;tabcol.cpp;tabdos.cpp;tabext.cpp;tabfix.cpp;tabfmt.cpp;tabjson.cpp;table.cpp;tabmul.cpp;tabmysql.cpp;taboccur.cpp;tabpivot.cpp;tabsys.cpp;tabtbl.cpp;tabutil.cpp;tabvir.cpp;tabxcl.cpp;valblk.cpp;value.cpp;xindex.cpp;xobject.cpp;array.h;blkfil.h;block.h;catalog.h;checklvl.h;colblk.h;connect.h;csort.h;engmsg.h;filamap.h;filamdbf.h;filamfix.h;filamgz.h;filamtxt.h;filter.h;global.h;ha_connect.h;inihandl.h;json.h;jsonudf.h;maputil.h;msgid.h;mycat.h;myconn.h;myutil.h;os.h;osutil.h;plgcnx.h;plgdbsem.h;preparse.h;reldef.h;resource.h;tabcol.h;tabdos.h;tabext.h;tabfix.h;tabfmt.h;tabjson.h;tabmul.h;tabmysql.h;taboccur.h;tabpivot.h;tabsys.h;tabtbl.h;tabutil.h;tabvir.h;tabxcl.h;user_connect.h;valblk.h;value.h;xindex.h;xobject.h;xtable.h;inihandl.cpp;filamvct.cpp;tabvct.cpp;filamvct.h;tabvct.h;libdoc.cpp;libdoc.h;tabxml.cpp;tabxml.h;plgxml.cpp;plgxml.h;tabodbc.cpp;odbconn.cpp;filamzip.cpp;tabzip.cpp;unzip.c;ioapi.c;zip.c;filamzip.h;tabzip.h;ioapi.h;unzip.h;zip.h;tabrest.cpp;tabrest.h;STORAGE_ENGINE;COMPONENT;connect-engine;RECOMPILE_FOR_EMBEDDED;LINK_LIBRARIES;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libxml2.so;/usr/lib/x86_64-linux-gnu/libodbc.so )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(38):  IF(NOT WITHOUT_SERVER OR ARG_CLIENT )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(41):  INCLUDE_DIRECTORIES(/home/anel/mariadb/10.2/include /home/anel/mariadb/10.2/sql  /usr/include /usr/include )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(47):  LIST(GET ARG_UNPARSED_ARGUMENTS 0 plugin )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(48):  SET(SOURCES connect;ha_connect.cc;connect.cc;user_connect.cc;mycat.cc;fmdlex.c;osutil.c;rcmsg.c;rcmsg.h;array.cpp;blkfil.cpp;colblk.cpp;csort.cpp;filamap.cpp;filamdbf.cpp;filamfix.cpp;filamgz.cpp;filamtxt.cpp;filter.cpp;json.cpp;jsonudf.cpp;maputil.cpp;myconn.cpp;myutil.cpp;plgdbutl.cpp;plugutil.cpp;reldef.cpp;tabcol.cpp;tabdos.cpp;tabext.cpp;tabfix.cpp;tabfmt.cpp;tabjson.cpp;table.cpp;tabmul.cpp;tabmysql.cpp;taboccur.cpp;tabpivot.cpp;tabsys.cpp;tabtbl.cpp;tabutil.cpp;tabvir.cpp;tabxcl.cpp;valblk.cpp;value.cpp;xindex.cpp;xobject.cpp;array.h;blkfil.h;block.h;catalog.h;checklvl.h;colblk.h;connect.h;csort.h;engmsg.h;filamap.h;filamdbf.h;filamfix.h;filamgz.h;filamtxt.h;filter.h;global.h;ha_connect.h;inihandl.h;json.h;jsonudf.h;maputil.h;msgid.h;mycat.h;myconn.h;myutil.h;os.h;osutil.h;plgcnx.h;plgdbsem.h;preparse.h;reldef.h;resource.h;tabcol.h;tabdos.h;tabext.h;tabfix.h;tabfmt.h;tabjson.h;tabmul.h;tabmysql.h;taboccur.h;tabpivot.h;tabsys.h;tabtbl.h;tabutil.h;tabvir.h;tabxcl.h;user_connect.h;valblk.h;value.h;xindex.h;xobject.h;xtable.h;inihandl.cpp;filamvct.cpp;tabvct.cpp;filamvct.h;tabvct.h;libdoc.cpp;libdoc.h;tabxml.cpp;tabxml.h;plgxml.cpp;plgxml.h;tabodbc.cpp;odbconn.cpp;filamzip.cpp;tabzip.cpp;unzip.c;ioapi.c;zip.c;filamzip.h;tabzip.h;ioapi.h;unzip.h;zip.h;tabrest.cpp;tabrest.h )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(49):  LIST(REMOVE_AT SOURCES 0 )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(50):  STRING(TOUPPER connect plugin )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(51):  STRING(TOLOWER CONNECT target )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(53):  IF(ARG_MANDATORY )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(56):  ELSE()
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(57):  SET(compat . )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(60):  IF(WITHOUT_CONNECT_STORAGE_ENGINE OR WITHOUT_CONNECT OR WITHOUT_PLUGIN_CONNECT OR WITH_NONE )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(67):  IF(WITH_CONNECT_STORAGE_ENGINE OR WITH_CONNECT OR WITH_PLUGIN_CONNECT OR WITH_ALL OR WITH_MAX OR WITH_MAX_NO_NDB OR ARG_DEFAULT )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(78):  IF(ARG_DISABLED )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(80):  ELSEIF(compat STREQUAL . )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(81):  SET(howtobuild DYNAMIC )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(99):  SET(PLUGIN_CONNECT DYNAMIC CACHE STRING How to build plugin CONNECT. Options are: NO STATIC DYNAMIC YES AUTO. )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(103):  IF(NOT PLUGIN_CONNECT MATCHES ^(NO|YES|AUTO|STATIC|DYNAMIC)$ )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(107):  IF(ARG_STORAGE_ENGINE )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(108):  SET(with_var WITH_CONNECT_STORAGE_ENGINE )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(112):  UNSET(WITH_CONNECT_STORAGE_ENGINE CACHE )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(114):  IF(NOT ARG_DEPENDENCIES )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(115):  SET(ARG_DEPENDENCIES )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(118):  IF(NOT ARG_MODULE_OUTPUT_NAME )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(119):  IF(ARG_STORAGE_ENGINE )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(120):  SET(ARG_MODULE_OUTPUT_NAME ha_connect )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(127):  IF(PLUGIN_CONNECT MATCHES (STATIC|AUTO|YES) AND NOT ARG_MODULE_ONLY AND NOT ARG_CLIENT )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(183):  ELSEIF(PLUGIN_CONNECT MATCHES (DYNAMIC|AUTO|YES) AND NOT ARG_STATIC_ONLY AND NOT WITHOUT_DYNAMIC_PLUGINS )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(186):  ADD_VERSION_INFO(connect MODULE SOURCES )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(187):  ADD_LIBRARY(connect MODULE ha_connect.cc;connect.cc;user_connect.cc;mycat.cc;fmdlex.c;osutil.c;rcmsg.c;rcmsg.h;array.cpp;blkfil.cpp;colblk.cpp;csort.cpp;filamap.cpp;filamdbf.cpp;filamfix.cpp;filamgz.cpp;filamtxt.cpp;filter.cpp;json.cpp;jsonudf.cpp;maputil.cpp;myconn.cpp;myutil.cpp;plgdbutl.cpp;plugutil.cpp;reldef.cpp;tabcol.cpp;tabdos.cpp;tabext.cpp;tabfix.cpp;tabfmt.cpp;tabjson.cpp;table.cpp;tabmul.cpp;tabmysql.cpp;taboccur.cpp;tabpivot.cpp;tabsys.cpp;tabtbl.cpp;tabutil.cpp;tabvir.cpp;tabxcl.cpp;valblk.cpp;value.cpp;xindex.cpp;xobject.cpp;array.h;blkfil.h;block.h;catalog.h;checklvl.h;colblk.h;connect.h;csort.h;engmsg.h;filamap.h;filamdbf.h;filamfix.h;filamgz.h;filamtxt.h;filter.h;global.h;ha_connect.h;inihandl.h;json.h;jsonudf.h;maputil.h;msgid.h;mycat.h;myconn.h;myutil.h;os.h;osutil.h;plgcnx.h;plgdbsem.h;preparse.h;reldef.h;resource.h;tabcol.h;tabdos.h;tabext.h;tabfix.h;tabfmt.h;tabjson.h;tabmul.h;tabmysql.h;taboccur.h;tabpivot.h;tabsys.h;tabtbl.h;tabutil.h;tabvir.h;tabxcl.h;user_connect.h;valblk.h;value.h;xindex.h;xobject.h;xtable.h;inihandl.cpp;filamvct.cpp;tabvct.cpp;filamvct.h;tabvct.h;libdoc.cpp;libdoc.h;tabxml.cpp;tabxml.h;plgxml.cpp;plgxml.h;tabodbc.cpp;odbconn.cpp;filamzip.cpp;tabzip.cpp;unzip.c;ioapi.c;zip.c;filamzip.h;tabzip.h;ioapi.h;unzip.h;zip.h;tabrest.cpp;tabrest.h )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(188):  DTRACE_INSTRUMENT(connect )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(190):  SET_TARGET_PROPERTIES(connect PROPERTIES PREFIX  )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(191):  IF(NOT ARG_CLIENT )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(192):  SET_TARGET_PROPERTIES(connect PROPERTIES COMPILE_DEFINITIONS MYSQL_DYNAMIC_PLUGIN )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(196):  TARGET_LINK_LIBRARIES(connect mysqlservices /usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libxml2.so;/usr/lib/x86_64-linux-gnu/libodbc.so )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(205):  IF(ARG_RECOMPILE_FOR_EMBEDDED OR ARG_STORAGE_ENGINE )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(206):  IF(MSVC )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(209):  ELSEIF(NOT CMAKE_SYSTEM_NAME STREQUAL Linux )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(216):  ADD_DEPENDENCIES(connect GenError  )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(218):  SET_TARGET_PROPERTIES(connect PROPERTIES OUTPUT_NAME ha_connect )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(221):  IF(ARG_COMPONENT )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(222):  IF(CPACK_COMPONENTS_ALL AND NOT CPACK_COMPONENTS_ALL MATCHES connect-engine AND INSTALL_SYSCONF2DIR )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(251):  MYSQL_INSTALL_TARGETS(connect DESTINATION lib/plugin COMPONENT connect-engine )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(255):  GET_FILENAME_COMPONENT(subpath /home/anel/mariadb/10.2/storage/connect NAME )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(256):  IF(EXISTS /home/anel/mariadb/10.2/storage/connect/mysql-test )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(257):  INSTALL_MYSQL_TEST(/home/anel/mariadb/10.2/storage/connect/mysql-test/ plugin/connect )
 
 
 
 
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(280):  IF(V MATCHES ^PLUGIN_ AND CONNECT_WITH_JDBC MATCHES YES )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(280):  IF(V MATCHES ^PLUGIN_ AND CONNECT_WITH_LIBXML2 MATCHES YES )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(280):  IF(V MATCHES ^PLUGIN_ AND CONNECT_WITH_MONGO MATCHES YES )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(280):  IF(V MATCHES ^PLUGIN_ AND CONNECT_WITH_ODBC MATCHES YES )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(280):  IF(V MATCHES ^PLUGIN_ AND CONNECT_WITH_REST MATCHES YES )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(280):  IF(V MATCHES ^PLUGIN_ AND CONNECT_WITH_VCT MATCHES YES )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(280):  IF(V MATCHES ^PLUGIN_ AND CONNECT_WITH_XMAP MATCHES YES )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(280):  IF(V MATCHES ^PLUGIN_ AND CONNECT_WITH_ZIP MATCHES YES )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(280):  IF(V MATCHES ^PLUGIN_ AND PLUGIN_CONNECT MATCHES YES )
 
 
 
 
 
SPIDER SE:
 
 
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(273):  IF(EXISTS /home/anel/mariadb/10.2/storage/spider/CMakeLists.txt )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(274):  ADD_SUBDIRECTORY(/home/anel/mariadb/10.2/storage/spider )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(32):  CMAKE_PARSE_ARGUMENTS(ARG STORAGE_ENGINE;STATIC_ONLY;MODULE_ONLY;MANDATORY;DEFAULT;DISABLED;RECOMPILE_FOR_EMBEDDED;CLIENT MODULE_OUTPUT_NAME;STATIC_OUTPUT_NAME;COMPONENT;CONFIG LINK_LIBRARIES;DEPENDENCIES spider;spd_param.cc;spd_sys_table.cc;spd_trx.cc;spd_db_conn.cc;spd_conn.cc;spd_table.cc;spd_direct_sql.cc;spd_udf.cc;spd_ping_table.cc;spd_copy_tables.cc;spd_i_s.cc;spd_malloc.cc;ha_spider.cc;spd_udf.def;spd_db_mysql.cc;spd_db_handlersocket.cc;spd_db_oracle.cc;hs_client/config.cpp;hs_client/escape.cpp;hs_client/fatal.cpp;hs_client/hstcpcli.cpp;hs_client/socket.cpp;hs_client/string_util.cpp;STORAGE_ENGINE;MODULE_ONLY;MODULE_OUTPUT_NAME;ha_spider )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(38):  IF(NOT WITHOUT_SERVER OR ARG_CLIENT )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(41):  INCLUDE_DIRECTORIES(/home/anel/mariadb/10.2/include /home/anel/mariadb/10.2/sql  /usr/include /usr/include )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(47):  LIST(GET ARG_UNPARSED_ARGUMENTS 0 plugin )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(48):  SET(SOURCES spider;spd_param.cc;spd_sys_table.cc;spd_trx.cc;spd_db_conn.cc;spd_conn.cc;spd_table.cc;spd_direct_sql.cc;spd_udf.cc;spd_ping_table.cc;spd_copy_tables.cc;spd_i_s.cc;spd_malloc.cc;ha_spider.cc;spd_udf.def;spd_db_mysql.cc;spd_db_handlersocket.cc;spd_db_oracle.cc;hs_client/config.cpp;hs_client/escape.cpp;hs_client/fatal.cpp;hs_client/hstcpcli.cpp;hs_client/socket.cpp;hs_client/string_util.cpp )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(49):  LIST(REMOVE_AT SOURCES 0 )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(50):  STRING(TOUPPER spider plugin )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(51):  STRING(TOLOWER SPIDER target )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(53):  IF(ARG_MANDATORY )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(56):  ELSE()
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(57):  SET(compat . )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(60):  IF(WITHOUT_SPIDER_STORAGE_ENGINE OR WITHOUT_SPIDER OR WITHOUT_PLUGIN_SPIDER OR WITH_NONE )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(67):  IF(WITH_SPIDER_STORAGE_ENGINE OR WITH_SPIDER OR WITH_PLUGIN_SPIDER OR WITH_ALL OR WITH_MAX OR WITH_MAX_NO_NDB OR ARG_DEFAULT )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(78):  IF(ARG_DISABLED )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(80):  ELSEIF(compat STREQUAL . )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(81):  SET(howtobuild DYNAMIC )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(99):  SET(PLUGIN_SPIDER DYNAMIC CACHE STRING How to build plugin SPIDER. Options are: NO STATIC DYNAMIC YES AUTO. )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(103):  IF(NOT PLUGIN_SPIDER MATCHES ^(NO|YES|AUTO|STATIC|DYNAMIC)$ )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(107):  IF(ARG_STORAGE_ENGINE )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(108):  SET(with_var WITH_SPIDER_STORAGE_ENGINE )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(112):  UNSET(WITH_SPIDER_STORAGE_ENGINE CACHE )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(114):  IF(NOT ARG_DEPENDENCIES )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(115):  SET(ARG_DEPENDENCIES )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(118):  IF(NOT ARG_MODULE_OUTPUT_NAME )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(127):  IF(PLUGIN_SPIDER MATCHES (STATIC|AUTO|YES) AND NOT ARG_MODULE_ONLY AND NOT ARG_CLIENT )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(183):  ELSEIF(PLUGIN_SPIDER MATCHES (DYNAMIC|AUTO|YES) AND NOT ARG_STATIC_ONLY AND NOT WITHOUT_DYNAMIC_PLUGINS )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(186):  ADD_VERSION_INFO(spider MODULE SOURCES )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(187):  ADD_LIBRARY(spider MODULE spd_param.cc;spd_sys_table.cc;spd_trx.cc;spd_db_conn.cc;spd_conn.cc;spd_table.cc;spd_direct_sql.cc;spd_udf.cc;spd_ping_table.cc;spd_copy_tables.cc;spd_i_s.cc;spd_malloc.cc;ha_spider.cc;spd_udf.def;spd_db_mysql.cc;spd_db_handlersocket.cc;spd_db_oracle.cc;hs_client/config.cpp;hs_client/escape.cpp;hs_client/fatal.cpp;hs_client/hstcpcli.cpp;hs_client/socket.cpp;hs_client/string_util.cpp )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(188):  DTRACE_INSTRUMENT(spider )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(190):  SET_TARGET_PROPERTIES(spider PROPERTIES PREFIX  )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(191):  IF(NOT ARG_CLIENT )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(192):  SET_TARGET_PROPERTIES(spider PROPERTIES COMPILE_DEFINITIONS MYSQL_DYNAMIC_PLUGIN )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(196):  TARGET_LINK_LIBRARIES(spider mysqlservices  )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(205):  IF(ARG_RECOMPILE_FOR_EMBEDDED OR ARG_STORAGE_ENGINE )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(206):  IF(MSVC )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(209):  ELSEIF(NOT CMAKE_SYSTEM_NAME STREQUAL Linux )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(216):  ADD_DEPENDENCIES(spider GenError  )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(218):  SET_TARGET_PROPERTIES(spider PROPERTIES OUTPUT_NAME ha_spider )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(221):  IF(ARG_COMPONENT )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(248):  ELSE()
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(249):  SET(ARG_COMPONENT Server )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(251):  MYSQL_INSTALL_TARGETS(spider DESTINATION lib/plugin COMPONENT Server )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(255):  GET_FILENAME_COMPONENT(subpath /home/anel/mariadb/10.2/storage/spider NAME )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(256):  IF(EXISTS /home/anel/mariadb/10.2/storage/spider/mysql-test )
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(257):  INSTALL_MYSQL_TEST(/home/anel/mariadb/10.2/storage/spider/mysql-test/ plugin/spider )
 
 
 
/home/anel/mariadb/10.2/cmake/plugin.cmake(280):  IF(V MATCHES ^PLUGIN_ AND PLUGIN_SPIDER MATCHES YES )

Comment by Otto Kekäläinen [ 2020-05-19 ]

anel This is the location where the connect.cnf and others are generated:

$ grep -C 2 .cnf cmake/plugin.cmake 
        IF(NOT ARG_CLIENT AND UNIX)
 
          IF (NOT ARG_CONFIG)
 
            SET(ARG_CONFIG "${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/${target}.cnf")
 
            FILE(WRITE ${ARG_CONFIG} "[mariadb]\nplugin-load-add=${ARG_MODULE_OUTPUT_NAME}.so\n")
 
          ENDIF()

Can you debug why it does not run for Spider?

Comment by Anel Husakovic [ 2020-05-19 ]

otto I debugged above and tried again to understand.
This part of code doesn't run for connect either for the above configuration, you have result above (plugin.cmake (222)).
However per zulip conversation it may be problem that COMPONENT is not set for SPIDER:
Found out about setting MODULE_OUTPUT_NAME for spider in MYSQL_ADD_PLUGIN storage/spider/CMakeLists.txt 46: MYSQL_ADD_PLUGIN(spider ${SPIDER_SOURCES} STORAGE_ENGINE MODULE_ONLY MODULE_OUTPUT_NAME "ha_spider"), while in case of connect MYSQL_ADD_PLUGIN is setting COMPONENT to connect-engine.
The patch that is setting the component for spider:

diff --git a/storage/spider/CMakeLists.txt b/storage/spider/CMakeLists.txt
 
index dec1cb4c6ba..a23ee9e1869 100644
 
--- a/storage/spider/CMakeLists.txt
 
+++ b/storage/spider/CMakeLists.txt
 
@@ -41,9 +41,10 @@ ELSE()
 
 
   INSTALL(FILES
 
     ${CMAKE_SOURCE_DIR}/storage/spider/scripts/install_spider.sql
 
-    DESTINATION ${INSTALL_MYSQLSHAREDIR} COMPONENT Server
 
+    DESTINATION ${INSTALL_MYSQLSHAREDIR} COMPONENT spider-engine
 
   )
 
-  MYSQL_ADD_PLUGIN(spider ${SPIDER_SOURCES} STORAGE_ENGINE MODULE_ONLY MODULE_OUTPUT_NAME "ha_spider")
 
+  MYSQL_ADD_PLUGIN(spider ${SPIDER_SOURCES} 
+    STORAGE_ENGINE COMPONENT spider-engine MODULE_ONLY MODULE_OUTPUT_NAME "ha_spider")
 
 ENDIF()

Can you please test it and let me know how you have done tests with applied patch?

Comment by Otto Kekäläinen [ 2020-05-19 ]

Thanks for the patch anel! Sure, I can test it, but I thought you wanted to get some Debian packaging experience under your belt and this would have been a good opportunity for you to test and finish it as well.

What do you recond about Mroonga, could it's `install.sql` be replaced with a simple ha_mroonga.so load like the connect and others or does it really need this complex install?

SET @inst=IF(EXISTS(SELECT * FROM mysql.plugin WHERE NAME='mroonga'),'DO 1', "INSTALL PLUGIN mroonga SONAME 'ha_mroonga'");
 
PREPARE s FROM @inst;
 
EXECUTE s;
 
 
 
DROP FUNCTION IF EXISTS last_insert_grn_id;
 
CREATE FUNCTION last_insert_grn_id RETURNS INTEGER
 
  SONAME 'ha_mroonga@MRN_PLUGIN_SUFFIX@';
 
 
 
DROP FUNCTION IF EXISTS mroonga_snippet;
 
CREATE FUNCTION mroonga_snippet RETURNS STRING
 
  SONAME 'ha_mroonga@MRN_PLUGIN_SUFFIX@';
 
 
 
DROP FUNCTION IF EXISTS mroonga_command;
 
CREATE FUNCTION mroonga_command RETURNS STRING
 
  SONAME 'ha_mroonga@MRN_PLUGIN_SUFFIX@';
 
 
 
DROP FUNCTION IF EXISTS mroonga_escape;
 
CREATE FUNCTION mroonga_escape RETURNS STRING
 
  SONAME 'ha_mroonga@MRN_PLUGIN_SUFFIX@';
 
 
 
DROP FUNCTION IF EXISTS mroonga_snippet_html;
 
CREATE FUNCTION mroonga_snippet_html RETURNS STRING
 
  SONAME 'ha_mroonga@MRN_PLUGIN_SUFFIX@';
 
 
 
DROP FUNCTION IF EXISTS mroonga_normalize;
 
CREATE FUNCTION mroonga_normalize RETURNS STRING
 
  SONAME 'ha_mroonga@MRN_PLUGIN_SUFFIX@';
 
 
 
DROP FUNCTION IF EXISTS mroonga_highlight_html;
 
CREATE FUNCTION mroonga_highlight_html RETURNS STRING
 
  SONAME 'ha_mroonga@MRN_PLUGIN_SUFFIX@';
 
 
 
DROP FUNCTION IF EXISTS mroonga_query_expand;
 
CREATE FUNCTION mroonga_query_expand RETURNS STRING
 
  SONAME 'ha_mroonga@MRN_PLUGIN_SUFFIX@';

Comment by Anel Husakovic [ 2020-05-19 ]

I would like to test sure, just please guide me how to test the changes from source.
Let's see does spider work correctly and we can take a look on mroonga also (which is already working).

Comment by Otto Kekäläinen [ 2020-05-19 ]

anel Seems it worked

------------------------------------------------
 
mariadb-plugin-spider
 
drwxr-xr-x root/root ./  
drwxr-xr-x root/root ./etc/  
drwxr-xr-x root/root ./etc/mysql/  
drwxr-xr-x root/root ./etc/mysql/mariadb.conf.d/  
-rw-r--r-- root/root ./etc/mysql/mariadb.conf.d/spider.cnf  
drwxr-xr-x root/root ./usr/  
drwxr-xr-x root/root ./usr/lib/  
drwxr-xr-x root/root ./usr/lib/mysql/  
drwxr-xr-x root/root ./usr/lib/mysql/plugin/  
-rw-r--r-- root/root ./usr/lib/mysql/plugin/ha_spider.so  
drwxr-xr-x root/root ./usr/share/  
drwxr-xr-x root/root ./usr/share/doc/  
drwxr-xr-x root/root ./usr/share/doc/mariadb-plugin-spider/  
-rw-r--r-- root/root ./usr/share/doc/mariadb-plugin-spider/changelog.gz  
-rw-r--r-- root/root ./usr/share/doc/mariadb-plugin-spider/copyright  
------------------------------------------------

When you test something yourself, just build, modify and re-build, it should not be complex.

Comment by Otto Kekäläinen [ 2020-05-19 ]

Fixing commit now included in PR at https://github.com/MariaDB/server/pull/1504/commits

Comment by Anel Husakovic [ 2020-05-21 ]

Thanks Otto, after pushing #1504 I will close this MDEV.
In meantime, can you show me your package-foo to get mariadb-plugin-spider as above.
I suppose you are doing this with autobake and docker?

Comment by Otto Kekäläinen [ 2020-05-21 ]

anel Simply run `debian/autobake-deb.sh` to run the build just like buildbot does. Study the source of that script to see what it does. At the end there is also the part that generates a list of all files in the completed *.deb files. You can also study the contents of debian/salsa-ci.yml to see the exact commands and steps (it's basically just shell script) that the Salsa-CI system runs to build and test packages. All of that can be just copy pasted into a local environment, Docker, Vagrant or whatever you want to use.

Comment by Anel Husakovic [ 2020-06-10 ]

Pushed on 10.5 with ba4148698fa648aaf15aa3.

Generated at Thu Feb 08 08:55:21 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.