[MDEV-20736] MariaDB 10.3.18 Crash "[ERROR] mysqld got signal 11 ;" Created: 2019-10-03  Updated: 2019-10-22

Status: Open
Project: MariaDB Server
Component/s: None
Affects Version/s: 10.3.18
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Martin Mørch Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: crash
Environment:

CentOS Linux release 7.7.1908 (Core)
Linux 3.10.0-1062.1.2.el7.x86_64 #1 SMP Mon Sep 30 14:19:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux


 Description   

We recently ran into a crash with a user performing a query causing MariaDB to crash.

Crash log:

191003 10:27:53 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.3.18-MariaDB-log
 
key_buffer_size=16777216
 
read_buffer_size=131072
 
max_used_connections=3
 
max_threads=153
 
thread_count=9
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 352734 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f4c880009a8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x202c090a0d22656d thread_stack 0x40000

The query in question:

DROP TABLE IF EXISTS DBA_memDatabaseList;
 
CREATE TEMPORARY TABLE IF NOT EXISTS DBA_memDatabaseList ENGINE=MEMORY AS (
 
        select schema_name as database_name
 
        from information_schema.schemata
 
);
 
 
 
SELECT database_name "DatabaseName"
 
        , ROUND(SUM(data_length + index_length + data_free) / 1024 / 1024, 1) "DB_Size_MB"
 
        , ROUND(SUM(data_length + index_length) / 1024 / 1024, 1) "Space_Used_ MB"
 
        , ROUND(SUM(data_free) / 1024 / 1024, 1) "Free_space_MB"
 
        , IF(t.table_schema IS NULL, NULL, COUNT(*)) "Table count"
 
FROM DBA_memDatabaseList d
 
LEFT JOIN information_schema.TABLES t ON t.table_schema = d.database_name
 
GROUP BY d.database_name
 
ORDER BY 1;
 
 
 
DROP TABLE DBA_memDatabaseList

When the query is run statement by statement, in 4 separate queries, is succeeds with no issue, like this:

Time                Id Command  Argument
 
191003 10:27:28      8 Query    DROP TABLE IF EXISTS DBA_memDatabaseList
 
                     8 Query    CREATE TEMPORARY TABLE IF NOT EXISTS DBA_memDatabaseList ENGINE=MEMORY AS (
 
        select schema_name as database_name
 
        from information_schema.schemata
 
)
 
                     8 Query    SELECT database_name "DatabaseName"
 
        , ROUND(SUM(data_length + index_length + data_free) / 1024 / 1024, 1) "DB_Size_MB"
 
        , ROUND(SUM(data_length + index_length) / 1024 / 1024, 1) "Space_Used_ MB"
 
        , ROUND(SUM(data_free) / 1024 / 1024, 1) "Free_space_MB"
 
        , IF(t.table_schema IS NULL, NULL, COUNT(*)) "Table count"
 
FROM DBA_memDatabaseList d
 
LEFT JOIN information_schema.TABLES t ON t.table_schema = d.database_name
 
GROUP BY d.database_name
 
ORDER BY 1
 
                     8 Query    DROP TABLE DBA_memDatabaseList

However, when run as a single query with multiple statements, it causes the entire server to crash. This is the query log when a crash occurs:

Time                Id Command  Argument
 
191003 10:27:53     10 Query    DROP TABLE IF EXISTS DBA_memDatabaseList;
 
CREATE TEMPORARY TABLE IF NOT EXISTS DBA_memDatabaseList ENGINE=MEMORY AS (
 
        select schema_name as database_name
 
        from information_schema.schemata
 
);
 
 
 
SELECT database_name "DatabaseName"
 
        , ROUND(SUM(data_length + index_length + data_free) / 1024 / 1024, 1) "DB_Size_MB"
 
        , ROUND(SUM(data_length + index_length) / 1024 / 1024, 1) "Space_Used_ MB"
 
        , ROUND(SUM(data_free) / 1024 / 1024, 1) "Free_space_MB"
 
        , IF(t.table_schema IS NULL, NULL, COUNT(*)) "Table count"
 
FROM DBA_memDatabaseList d
 
LEFT JOIN information_schema.TABLES t ON t.table_schema = d.database_name
 
GROUP BY d.database_name
 
ORDER BY 1;
 
 
 
DROP TABLE DBA_memDatabaseList

Steps to re-produce:

  1. Install MariaDB-server-10.3.18-1.el7.centos.x86_64
  2. Run the above query as multiple statements in a single query

 Comments   
Comment by Alice Sherepa [ 2019-10-04 ]

Could you please describe how exactly you run the queries, does it crash every time for you? Please add your
.cnf file(s).
I tried to reproduce via php - mysqli_multi_query - it worked as expected for me. 10.3.18, Ubuntu 16.04:

191004 11:11:21	    22 Connect	root@localhost as anonymous on test
 
		    22 Query	DROP TABLE IF EXISTS DBA_memDatabaseList;
 
CREATE TEMPORARY TABLE IF NOT EXISTS DBA_memDatabaseList ENGINE=MEMORY AS (
 
        select schema_name as database_name
 
        from information_schema.schemata
 
);
 
 
SELECT database_name "DatabaseName"
 
        , ROUND(SUM(data_length + index_length + data_free) / 1024 / 1024, 1) "DB_Size_MB"
 
        , ROUND(SUM(data_length + index_length) / 1024 / 1024, 1) "Space_Used_ MB"
 
        , ROUND(SUM(data_free) / 1024 / 1024, 1) "Free_space_MB"
 
        , IF(t.table_schema IS NULL, NULL, COUNT(*)) "Table count"
 
FROM DBA_memDatabaseList d
 
LEFT JOIN information_schema.TABLES t ON t.table_schema = d.database_name
 
GROUP BY d.database_name
 
ORDER BY 1;
 
 
DROP TABLE DBA_memDatabaseList
 
191004 11:11:22	    23 Connect	root@localhost as anonymous on test
 
		    23 Query	DROP TABLE IF EXISTS DBA_memDatabaseList;
 
CREATE TEMPORARY TABLE IF NOT EXISTS DBA_memDatabaseList ENGINE=MEMORY AS (
 
        select schema_name as database_name
 
        from information_schema.schemata
 
);
 
 
SELECT database_name "DatabaseName"
 
        , ROUND(SUM(data_length + index_length + data_free) / 1024 / 1024, 1) "DB_Size_MB"
 
        , ROUND(SUM(data_length + index_length) / 1024 / 1024, 1) "Space_Used_ MB"
 
        , ROUND(SUM(data_free) / 1024 / 1024, 1) "Free_space_MB"
 
        , IF(t.table_schema IS NULL, NULL, COUNT(*)) "Table count"
 
FROM DBA_memDatabaseList d
 
LEFT JOIN information_schema.TABLES t ON t.table_schema = d.database_name
 
GROUP BY d.database_name
 
ORDER BY 1;
 
 
DROP TABLE DBA_memDatabaseList

Comment by Martin Mørch [ 2019-10-04 ]

This is an example of a session that caused a crash. Doing the exact same set of queries can re-produce a crash every single time:

                  9803 Connect  dbadmin@192.168.XXX.XXX as anonymous on DBA
 
                  9803 Query    SHOW VARIABLES
 
                  9803 Query    SELECT TIMEDIFF(NOW(), UTC_TIMESTAMP())
 
191004  9:23:31   9803 Query    SHOW COLLATION
 
                  9803 Query    SET NAMES latin1
 
                  9803 Query    SET character_set_results=NULL
 
                  9803 Init DB  DBA
 
                  9803 Query    DROP TABLE IF EXISTS DBA_memDatabaseList;
 
CREATE TEMPORARY TABLE IF NOT EXISTS DBA_memDatabaseList ENGINE=MEMORY AS (
 
        select schema_name as database_name
 
        from information_schema.schemata
 
);
 
 
 
SELECT database_name "DatabaseName"
 
        , ROUND(SUM(data_length + index_length + data_free) / 1024 / 1024, 1) "DB_Size_MB"
 
        , ROUND(SUM(data_length + index_length) / 1024 / 1024, 1) "Space_Used_ MB"
 
        , ROUND(SUM(data_free) / 1024 / 1024, 1) "Free_space_MB"
 
        , IF(t.table_schema IS NULL, NULL, COUNT(*)) "Table count"
 
FROM DBA_memDatabaseList d
 
LEFT JOIN information_schema.TABLES t ON t.table_schema = d.database_name
 
GROUP BY d.database_name
 
ORDER BY 1;
 
 
 
DROP TABLE DBA_memDatabaseList

Server config: https://pastebin.com/RbbuDdMM

Comment by Elena Stepanova [ 2019-10-06 ]

I can't reproduce it either, although I don't have CentOS 7.7 handy, so I'm trying 7.5, and smaller buffer pool size.

$ rpm -qa | grep -i maria
 
MariaDB-common-10.3.18-1.el7.centos.x86_64
 
MariaDB-server-10.3.18-1.el7.centos.x86_64
 
MariaDB-compat-10.3.18-1.el7.centos.x86_64
 
MariaDB-client-10.3.18-1.el7.centos.x86_64


191006 14:54:25     17 Connect  dbadmin@localhost as anonymous on 
                    17 Query    select @@version_comment limit 1
 
191006 14:54:26     17 Query    SHOW VARIABLES
 
                    17 Query    SELECT TIMEDIFF(NOW(), UTC_TIMESTAMP())
 
                    17 Query    SHOW COLLATION
 
                    17 Query    SET NAMES latin1
 
                    17 Query    SET character_set_results=NULL
 
                    17 Query    SELECT DATABASE()
 
                    17 Init DB  DBA
 
                    17 Query    DROP TABLE IF EXISTS DBA_memDatabaseList;
 
CREATE TEMPORARY TABLE IF NOT EXISTS DBA_memDatabaseList ENGINE=MEMORY AS (
 
        select schema_name as database_name
 
        from information_schema.schemata
 
);
 
 
SELECT database_name "DatabaseName"
 
        , ROUND(SUM(data_length + index_length + data_free) / 1024 / 1024, 1) "DB_Size_MB"
 
        , ROUND(SUM(data_length + index_length) / 1024 / 1024, 1) "Space_Used_ MB"
 
        , ROUND(SUM(data_free) / 1024 / 1024, 1) "Free_space_MB"
 
        , IF(t.table_schema IS NULL, NULL, COUNT(*)) "Table count"
 
FROM DBA_memDatabaseList d
 
LEFT JOIN information_schema.TABLES t ON t.table_schema = d.database_name
 
GROUP BY d.database_name
 
ORDER BY 1;
 
 
DROP TABLE DBA_memDatabaseList
 
191006 14:54:36     17 Query    select 'Still alive?'

I would think that it is somehow something specific to the database, for example there are corrupt tables which make information schema misbehave, or the queries values are such that they trigger a failure, but it doesn't explain why it wouldn't fail when the statements are executed one by one.

martinmorch, would you be able to enable the coredump and see if you can acquire a stack trace? You might need to additionally install MariaDB-server-debuginfo-10.3.18-1.el7.centos.x86_64.rpm to make the stack trace meaningful.

Comment by Martin Mørch [ 2019-10-08 ]

I was messing with getting mysqld-debug up and running and had to stop halfway. Remove *-debuginfo packages and did a yum reinstall on MariaDB-server. After reinstall, I cannot re-produce the issue. Had a small PHP script using PDO that was able to make mysqld crash every single time. After the reinstall, it simply throws an error:

Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error'

Before, this same script would simply hang and never produce an error.
Running the script with prepared statement emulation off, reveals:

Uncaught exception 'PDOException' with message 'SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'CREATE TEMPORARY TABLE IF NOT EXISTS DBA_memDatabaseList ENGINE=MEMORY AS (selec' at line 1'

Weird part is, this statement works just fine using the MariaDB mysql CLI tool.

Anyway, I think you can park this as "user error" or similar. Something obviously happened when reinstalling RPM's, unfortunately I do not know what

Comment by Martin Mørch [ 2019-10-22 ]

Our DBA mentioned today that he was missing MariaDB-connect-engine on the server that previously had issues.

Did 'yum install MariaDB-connect-engine' (specifically MariaDB-connect-engine-10.3.18-1.el7.centos.x86_64), verified MariaDB had restarted correctly and out of curiosity tried running the above statement again and VOILA! MariaDB crashed!

So, something related to MariaDB-connect-engine-10.3.18-1.el7.centos.x86_64 seems to cause this issue.

Can anyone else verify?

Generated at Thu Feb 08 09:01:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.