Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4486

Buster/Sid installation tests hang upon operations on a Columnstore table

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 5.4.3
    • Fix Version/s: 5.5.2
    • Component/s: PrimProc
    • Labels:
      None
    • Sprint:
      2020-8, 2021-2

      Description

      Since approximately this commit (not necessarily this one, but somewhere close)

      commit 1edd224372d9399a43a21c5c387aff84512f55e5
      Author: Roman Nozdrin
      Date:   Sat Nov 14 14:13:33 2020 +0000
       
          This patch puts MCS debian packaging files and part of debian/control
          into the engine directory
      

      most of installation test runs on Buster and Sid amd64 fail with server hang.
      The hang happens not during package installation, but upon basic checks which it the test performs afterwards, specifically DML on a Columnstore table.
      It is also reproducible (not deterministically, but easily enough) on a Buster VM image by performing the same steps that buildbot runs (assuming that the tested packages are already downloaded to the local disk, and the apt source list is configured to point at them, as well at the location of Galera library needed for smooth installation):

      sudo sh -c 'DEBIAN_FRONTEND=noninteractive MYSQLD_STARTUP_TIMEOUT=180 apt-get install --allow-unauthenticated -y libmariadb-dev libmariadb-dev-compat libmariadb-dev-dbgsym libmariadb3 libmariadb3-compat libmariadb3-dbgsym libmariadbclient18 libmariadbd-dev libmariadbd19 libmariadbd19-dbgsym libmysqlclient18 mariadb-backup mariadb-backup-dbgsym mariadb-client mariadb-client-10.5 mariadb-client-10.5-dbgsym mariadb-client-core-10.5 mariadb-client-core-10.5-dbgsym mariadb-common mariadb-plugin-connect mariadb-plugin-connect-dbgsym mariadb-plugin-cracklib-password-check mariadb-plugin-cracklib-password-check-dbgsym mariadb-plugin-gssapi-client mariadb-plugin-gssapi-client-dbgsym mariadb-plugin-gssapi-server mariadb-plugin-gssapi-server-dbgsym mariadb-plugin-mroonga mariadb-plugin-mroonga-dbgsym mariadb-plugin-oqgraph mariadb-plugin-oqgraph-dbgsym mariadb-plugin-rocksdb mariadb-plugin-rocksdb-dbgsym mariadb-plugin-s3 mariadb-plugin-s3-dbgsym mariadb-server mariadb-server-10.5 mariadb-server-10.5-dbgsym mariadb-server-core-10.5 mariadb-server-core-10.5-dbgsym mariadb-test mariadb-test-data mariadb-test-dbgsym mysql-common mariadb-plugin-columnstore mariadb-plugin-columnstore-dbgsym'
      sudo systemctl status mariadb --no-pager
      sudo sh -c 'DEBIAN_FRONTEND=noninteractive MYSQLD_STARTUP_TIMEOUT=180 apt-get install --allow-unauthenticated -y mariadb-plugin-spider mariadb-plugin-spider-dbgsym'
      sudo mysql -e 'set password=password("rootpass")'
      mysql --verbose -uroot -prootpass -e 'select @@version; create database test; use test; create table t(a int primary key) engine=innodb; insert into t values (1); select * from t; drop table t; drop database test; create user galera identified by "gal3ra123"; grant all on *.* to galera;'
      mysql --verbose -uroot -prootpass -e 'create database cs; use cs; create table cs.t_columnstore (a int, b char(8)) engine=ColumnStore; insert into cs.t_columnstore select seq, concat("val",seq) from seq_1_to_10; select * from cs.t_columnstore'
      sudo systemctl restart mariadb
      mysql --verbose -uroot -prootpass -e 'select * from cs.t_columnstore; update cs.t_columnstore set b = "updated"'
      sudo systemctl restart mariadb-columnstore
      mysql --verbose -uroot -prootpass -e 'update cs.t_columnstore set a = a + 10; select * from cs.t_columnstore'
      

      The hang happens at different statements from the script above, starting from create table cs.t_columnstore ....

      Here is a stack trace from the only thread which seems to be up to something on a running (hanging) server on one such occurrence, the hanging statement here is insert into cs.t_columnstore select seq, concat("val",seq) from seq_1_to_10:

      10.5 1bf9acceef2520

       Thread 17 (Thread 0x7f9a383ee700 (LWP 2875)):
      #0  __libc_read (nbytes=1, buf=0x7f99d17ee31f, fd=99) at ../sysdeps/unix/sysv/linux/read.c:26
      #1  __libc_read (fd=99, buf=buf@entry=0x7f99d17ee31f, nbytes=nbytes@entry=1) at ../sysdeps/unix/sysv/linux/read.c:24
      #2  0x00007f9a60b38b92 in read (__nbytes=1, __buf=0x7f99d17ee31f, __fd=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/unistd.h:44
      #3  messageqcpp::InetStreamSocket::readToMagic (this=0x7f99d17ee2d0, msecs=-1, isTimeOut=0x0, stats=0x0) at ./storage/columnstore/columnstore/utils/messageqcpp/inetstreamsocket.cpp:419
      #4  0x00007f9a60b39a99 in messageqcpp::InetStreamSocket::read (this=this@entry=0x7f99d17ee2d0, timeout=0x0, isTimeOut=0x0, stats=0x0) at ./storage/columnstore/columnstore/utils/messageqcpp/inetstreamsocket.cpp:490
      #5  0x00007f9a60b3dfe0 in messageqcpp::CompressedInetStreamSocket::read (this=0x7f99d17ee2d0, timeout=<optimized out>, isTimeOut=<optimized out>, stats=<optimized out>) at ./storage/columnstore/columnstore/utils/messageqcpp/compressed_iss.cpp:92
      #6  0x00007f9a60b285c9 in messageqcpp::IOSocket::read (stats=0x0, isTimeOut=0x0, timeout=0x0, this=0x7f99d0067098) at ./storage/columnstore/columnstore/utils/messageqcpp/iosocket.h:245
      #7  messageqcpp::MessageQueueClient::read (this=0x7f99d0067060, timeout=timeout@entry=0x0, isTimeOut=isTimeOut@entry=0x0, stats=stats@entry=0x0) at ./storage/columnstore/columnstore/utils/messageqcpp/messagequeue.cpp:284
      #8  0x00007f9a613d85a5 in execplan::ClientRotator::read (this=0x7f99d186e7f0) at ./storage/columnstore/columnstore/dbcon/execplan/clientrotator.cpp:226
      #9  0x00007f9a613986d9 in execplan::CalpontSystemCatalog::getSysData_FE (this=0x7f99d1842bd0, csep=..., sysDataList=..., sysTableName=...) at ./storage/columnstore/columnstore/dbcon/execplan/calpontsystemcatalog.cpp:999
      #10 0x00007f9a6139961c in execplan::CalpontSystemCatalog::getSysData (this=0x7f99d1842bd0, csep=..., sysDataList=..., sysTableName=...) at ./storage/columnstore/columnstore/dbcon/execplan/calpontsystemcatalog.cpp:859
      #11 0x00007f9a613a45d8 in execplan::CalpontSystemCatalog::columnRIDs (this=0x7f99d1842bd0, tableName=..., useCache=useCache@entry=false, lower_case_table_names=<optimized out>) at ./storage/columnstore/columnstore/dbcon/execplan/calpontsystemcatalog.cpp:3196
      #12 0x00007f9a61e78ee2 in ha_mcs_impl_start_bulk_insert (rows=<optimized out>, table=0x7f99d18050c8, is_cache_insert=is_cache_insert@entry=false) at /usr/include/boost/smart_ptr/shared_ptr.hpp:732
      #13 0x00007f9a61e705e8 in ha_mcs::start_bulk_insert (this=<optimized out>, rows=<optimized out>, flags=<optimized out>) at ./storage/columnstore/columnstore/dbcon/mysql/ha_mcs.cpp:332
      #14 0x00005646301b7f26 in handler::ha_start_bulk_insert (flags=0, rows=0, this=<optimized out>) at ./sql/sql_insert.cc:3884
      #15 select_insert::prepare (this=0x7f99d0015d70, values=..., u=<optimized out>) at ./sql/sql_insert.cc:3939
      #16 0x000056463023c2d2 in JOIN::prepare (this=0x7f99d0015e28, tables_init=<optimized out>, conds_init=<optimized out>, og_num=<optimized out>, order_init=<optimized out>, skip_order_by=<optimized out>, group_init=0x0, having_init=0x0, proc_param_init=0x0, select_lex_arg=0x7f99d0013388, unit_arg=0x7f99d0004bf8) at ./sql/sql_select.cc:1388
      #17 0x000056463024b9a3 in mysql_select (thd=0x7f99d0000c18, tables=0x7f99d0013cd0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2202244745984, result=0x7f99d0015d70, unit=0x7f99d0004bf8, select_lex=0x7f99d0013388) at ./sql/sql_select.cc:4634
      #18 0x000056463024bc1e in handle_select (thd=thd@entry=0x7f99d0000c18, lex=lex@entry=0x7f99d0004b30, result=result@entry=0x7f99d0015d70, setup_tables_done_option=setup_tables_done_option@entry=1073741824) at ./sql/sql_select.cc:417
      #19 0x00005646301f64a7 in mysql_execute_command (thd=0x7f99d0000c18) at ./sql/sql_parse.cc:4733
      #20 0x00005646301e4743 in mysql_parse (thd=0x7f99d0000c18, rawbuf=<optimized out>, length=75, parser_state=0x7f9a383ed460, is_com_multi=<optimized out>, is_next_command=<optimized out>) at ./sql/sql_parse.cc:8042
      #21 0x00005646301ef7f7 in dispatch_command (command=COM_QUERY, thd=0x7f99d0000c18, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at ./sql/sql_class.h:1255
      #22 0x00005646301f06ad in do_command (thd=0x7f99d0000c18) at ./sql/sql_parse.cc:1353
      #23 0x00005646302dffe1 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x564632fc88c8, put_in_cache=put_in_cache@entry=true) at ./sql/sql_connect.cc:1410
      #24 0x00005646302e037d in handle_one_connection (arg=arg@entry=0x564632fc88c8) at ./sql/sql_connect.cc:1312
      #25 0x0000564630607e7b in pfs_spawn_thread (arg=0x564632fe9ce8) at ./storage/perfschema/pfs.cc:2201
      #26 0x00007f9a645c6fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
      #27 0x00007f9a63aca4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
      

      Most recent example in buildbot as of now:
      http://buildbot.askmonty.org/buildbot/builders/kvm-deb-buster-amd64/builds/3638/steps/install/logs/stdio

        Attachments

          Activity

            People

            Assignee:
            alexey.antipovsky Alexey Antipovsky
            Reporter:
            elenst Elena Stepanova
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration