[MDEV-10398] segfault in mariadb 5.5.50 with galera 25.3.16 Created: 2016-07-19  Updated: 2020-10-20  Resolved: 2020-10-20

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 5.5.50-galera
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Evan Jardine-Skinner Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Environment:

A 3 node cluster using Ubuntu 14.04 with mariadb 5.5.50 and galera 25.3.16.


Attachments: File etcmysql.tgz     File healthcheckcat_5.users.sql     File mariadb-bin.000077.gz     File mariadb-bin.index     File mysql.log.gz     File syslog.gz    

 Description   

segfault during normal operation of mariadb. Right at the bottom of syslog.

I had hit this before so I had turned on the general query log. I've attached the general query log, syslog (with most of the irrelevant stuff, like CRON etc filtered out) and the last binary log from the node that crashed. Let me know if you need more files from the machine. It will liekly be reinstalled in a few days.

 Comments   
Comment by Elena Stepanova [ 2016-07-19 ]

Ev, hi,

if you can afford an experiment (at the risk of getting the server crashed again), could you please check whether the following sequence executed from a new MySQL client session causes the crash?

USE healthcheckcat_5;
 
SET NAMES utf8;
 
SET NAMES utf8mb4 COLLATE utf8mb4_general_ci;
 
SET sql_mode = 'REAL_AS_FLOAT,PIPES_AS_CONCAT,ANSI_QUOTES,IGNORE_SPACE,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO
 
_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER';
 
 
 
SELECT base.uid AS uid, base.name AS name, base.pass AS pass, base.mail AS mail, base.theme AS theme, base.signature AS signature, base.signature_format AS signature_format, base.created AS created, base.access AS access, base.login AS login, base.status AS status, base.timezone AS timezone, base.language AS language, base.picture AS picture, base.init AS init, base.data AS data
 
FROM 
users base
 
WHERE  (base.uid IN  ('0'));

In any case, please attach your cnf file(s) and paste the output of 

USE healthcheckcat_5;
 
SHOW CREATE TABLE users\G
 
SHOW INDEX IN users;
 
SHOW TABLE STATUS LIKE 'users';

If you can provide the datadump of the users table (or .frm/.MYI/.MYD files if it's MyISAM), it would be even better. You can upload them to ftp.askmonty.org/private.

Thanks.

Comment by Evan Jardine-Skinner [ 2016-07-19 ]

I will try those commands asap. Here is a dump of the mysql database in the meantime. And also jsut he suer table which is what you actually wanted

Comment by Evan Jardine-Skinner [ 2016-07-19 ]

Output of:

USE healthcheckcat_5;
 
 
 
SHOW CREATE TABLE users\G
 
 
 
SHOW INDEX IN users;
 
 
 
SHOW TABLE STATUS LIKE 'users';

is

Welcome to the MariaDB monitor.  Commands end with ; or \g.
 
Your MariaDB connection id is 2262
 
Server version: 5.5.50-MariaDB-1~trusty-wsrep mariadb.org binary distribution, wsrep_25.14.r9949137
 
 
 
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
 
 
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
 
 
MariaDB [(none)]> USE healthcheckcat_5;
 
Reading table information for completion of table and column names
 
You can turn off this feature to get a quicker startup with -A
 
 
 
Database changed
 
MariaDB [healthcheckcat_5]> 
MariaDB [healthcheckcat_5]> SHOW CREATE TABLE users\G
 
*************************** 1. row ***************************
 
       Table: users
 
Create Table: CREATE TABLE `users` (
 
  `uid` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'Primary Key: Unique user ID.',
 
  `name` varchar(60) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 
  `pass` varchar(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 
  `mail` varchar(254) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 
  `theme` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 
  `signature` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 
  `signature_format` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 
  `created` int(11) NOT NULL DEFAULT '0' COMMENT 'Timestamp for when user was created.',
 
  `access` int(11) NOT NULL DEFAULT '0' COMMENT 'Timestamp for previous time user accessed the site.',
 
  `login` int(11) NOT NULL DEFAULT '0' COMMENT 'Timestamp for user’s last login.',
 
  `status` tinyint(4) NOT NULL DEFAULT '0' COMMENT 'Whether the user is active(1) or blocked(0).',
 
  `timezone` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 
  `language` varchar(12) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 
  `picture` int(11) NOT NULL DEFAULT '0' COMMENT 'Foreign key: file_managed.fid of user’s picture.',
 
  `init` varchar(254) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 
  `data` longblob COMMENT 'A serialized array of name value pairs that are related to the user. Any form values posted during user edit are stored and are loaded into the $user object during user_load(). Use of this field is discouraged and it will likely disappear in a future...',
 
  PRIMARY KEY (`uid`),
 
  UNIQUE KEY `name` (`name`),
 
  KEY `access` (`access`),
 
  KEY `created` (`created`),
 
  KEY `mail` (`mail`(191)),
 
  KEY `picture` (`picture`)
 
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Stores user data.'
 
1 row in set (0.00 sec)
 
 
 
MariaDB [healthcheckcat_5]> 
MariaDB [healthcheckcat_5]> SHOW INDEX IN users;
 
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
 
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
 
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
 
| users |          0 | PRIMARY  |            1 | uid         | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
 
| users |          0 | name     |            1 | name        | A         |           2 |     NULL | NULL   | YES  | BTREE      |         |               |
 
| users |          1 | access   |            1 | access      | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
 
| users |          1 | created  |            1 | created     | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
 
| users |          1 | mail     |            1 | mail        | A         |           2 |      191 | NULL   | YES  | BTREE      |         |               |
 
| users |          1 | picture  |            1 | picture     | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
 
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
 
6 rows in set (0.00 sec)
 
 
 
MariaDB [healthcheckcat_5]> 
MariaDB [healthcheckcat_5]> SHOW TABLE STATUS LIKE 'users';
 
+-------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+-------------------+
 
| Name  | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time         | Update_time | Check_time | Collation       | Checksum | Create_options | Comment           |
 
+-------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+-------------------+
 
| users | InnoDB |      10 | Compact    |    2 |           8192 |       16384 |               0 |        81920 |  54525952 |           NULL | 2016-07-19 12:32:55 | NULL        | NULL       | utf8_general_ci |     NULL |                | Stores user data. |
 
+-------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+-------------------+
 
1 row in set (0.00 sec)

Comment by Elena Stepanova [ 2016-07-19 ]

Ev,

Thanks, but the dump is actually from a different table. The query is from healthcheckcat_5.users, not from mysql.user.
According to the above, the healthcheckcat_5.users table only has 2 rows, so the dump should be tiny.
The request for cnf file(s) still holds.

Comment by Evan Jardine-Skinner [ 2016-07-19 ]

Oops. Many apologies I was in a hurry then before getting to a meeting. Am uploading a tgz of the /etc/mysql directory and also the dump of the correct table

Comment by Evan Jardine-Skinner [ 2016-07-19 ]

Running the commands in your first comment does not cause mysql to crash. I see the following:

admin@advancedportalnode-test-20160711-10-137-29-166:~$ mysql
 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
 
Your MariaDB connection id is 49540
 
Server version: 5.5.50-MariaDB-1~trusty-wsrep mariadb.org binary distribution, wsrep_25.14.r9949137
 
 
 
Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
 
 
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
 
 
MariaDB [(none)]> USE healthcheckcat_5;
 
Reading table information for completion of table and column names
 
You can turn off this feature to get a quicker startup with -A
 
 
 
Database changed
 
MariaDB [healthcheckcat_5]> SET NAMES utf8;
 
Query OK, 0 rows affected (0.00 sec)
 
 
 
MariaDB [healthcheckcat_5]> SET NAMES utf8mb4 COLLATE utf8mb4_general_ci;
 
Query OK, 0 rows affected (0.00 sec)
 
 
 
MariaDB [healthcheckcat_5]> SET sql_mode = 'REAL_AS_FLOAT,PIPES_AS_CONCAT,ANSI_QUOTES,IGNORE_SPACE,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER';
 
Query OK, 0 rows affected (0.00 sec)
 
 
 
MariaDB [healthcheckcat_5]> SELECT base.uid AS uid, base.name AS name, base.pass AS pass, base.mail AS mail, base.theme AS theme, base.signature AS signature, base.signature_format AS signature_format, base.created AS created, base.access AS access, base.login AS login, base.status AS status, base.timezone AS timezone, base.language AS language, base.picture AS picture, base.init AS init, base.data AS data
 
    -> 
    -> 
    -> FROM 
    -> 
    -> users base
 
    -> WHERE  (base.uid IN  ('0'));
 
+-----+------+------+------+-------+-----------+------------------+---------+--------+-------+--------+----------+----------+---------+------+------+
 
| uid | name | pass | mail | theme | signature | signature_format | created | access | login | status | timezone | language | picture | init | data |
 
+-----+------+------+------+-------+-----------+------------------+---------+--------+-------+--------+----------+----------+---------+------+------+
 
|   0 |      |      |      |       |           | NULL             |       0 |      0 |     0 |      0 | NULL     |          |       0 |      | NULL |
 
+-----+------+------+------+-------+-----------+------------------+---------+--------+-------+--------+----------+----------+---------+------+------+
 
1 row in set (0.01 sec)

And afterwards I still have Primary status for the node

Comment by Elena Stepanova [ 2016-07-19 ]

That's rather bad news... If it's not readily reproducible on your side, it's going to be difficult to reproduce on ours.

Do you happen to know where connections which run this query actually come from? Given that the connection that crashed executed ~70 queries in 2 seconds, it's probably not a client, but some connector, do you know which it is?

Please also paste the output of 

dpkg -l | grep -iE 'mysql|maria|ssl|crypto'

Comment by Evan Jardine-Skinner [ 2016-07-21 ]

Here's the dpkg output

$ dpkg -l | grep -iE 'mysql|maria|ssl|crypto'
 
ii  libcurl3:amd64                             7.35.0-1ubuntu2.6                amd64        easy-to-use client-side URL transfer library (OpenSSL flavour)
 
ii  libdbd-mysql:amd64                         0.9.0-2ubuntu2                   amd64        MySQL database server driver for libdbi
 
ii  libdbd-mysql-perl                          4.025-1                          amd64        Perl5 database interface to the MySQL database
 
ii  libgcr-3-common                            3.10.1-1                         all          Library for Crypto UI related tasks - common files
 
ii  libgcr-base-3-1:amd64                      3.10.1-1                         amd64        Library for Crypto related tasks
 
ii  libgcrypt11:amd64                          1.5.3-2ubuntu4.3                 amd64        LGPL Crypto library - runtime library
 
ii  libgnutls-openssl27:amd64                  2.12.23-12ubuntu2.5              amd64        GNU TLS library - OpenSSL wrapper
 
ii  libhcrypto4-heimdal:amd64                  1.6~git20131207+dfsg-1ubuntu1.1  amd64        Heimdal Kerberos - crypto library
 
ii  libk5crypto3:amd64                         1.12+dfsg-2ubuntu5.2             amd64        MIT Kerberos runtime libraries - Crypto Library
 
ii  libmariadbclient18                         5.5.50+maria-1~trusty            amd64        MariaDB database client library
 
ii  libmysqlclient18                           5.5.50+maria-1~trusty            amd64        Virtual package to satisfy external depends
 
ii  libssl1.0.0:amd64                          1.0.1f-1ubuntu2.18               amd64        Secure Sockets Layer toolkit - shared libraries
 
ii  mariadb-client-5.5                         5.5.50+maria-1~trusty            amd64        MariaDB database client binaries
 
ii  mariadb-client-core-5.5                    5.5.50+maria-1~trusty            amd64        MariaDB database core client binaries
 
ii  mariadb-common                             5.5.50+maria-1~trusty            all          MariaDB database common files (e.g. /etc/mysql/conf.d/mariadb.cnf)
 
ii  mariadb-galera-server-5.5                  5.5.50+maria-1~trusty            amd64        MariaDB database server with Galera cluster binaries
 
ii  mysql-common                               5.5.50+maria-1~trusty            all          MariaDB database common files (e.g. /etc/mysql/my.cnf)
 
ii  openssl                                    1.0.1f-1ubuntu2.18               amd64        Secure Sockets Layer toolkit - cryptographic utility
 
ii  php5-mysql                                 5.5.9+dfsg-1ubuntu4.17           amd64        MySQL module for php5
 
ii  python-openssl                             0.13-2ubuntu6                    amd64        Python 2 wrapper around the OpenSSL library
 
ii  ssl-cert                                   1.0.33                           all          simple debconf wrapper for OpenSSL

This output has reminded me that these servers are actually Ubuntu 14.04 and not Debian 7 as I originally mentioned. Sorry.

Comment by Evan Jardine-Skinner [ 2016-07-21 ]

The code that is running the queries comes from three places:

1) Aegir (site management for Drupal): creating and deleting drupal web sites. Creating the site involves restoring a database (I think via the mysql cmd line client) then changing some values in the database (via mysql cmd line) and running a bunch of php to do other configuration stuff to the DB. It then immediately kicks off the cron tasks referenced in 2)

2) Drupal 7 websites (no users are navigating around the GUI as this is a test system): running cron tasks (in PHP), which do stuff like build up caches of data, insert data into the drupal db etc.

3) Some system scripts that do various checks and use DB tables to create lock rows so that conflicting tasks do not happen at once. These all us the mysql cmd line client.

Comment by Elena Stepanova [ 2016-08-28 ]

Stack trace from the syslog (to make it searchable in JIRA by whatever little it has):

Jul 18 20:07:50 160718 20:07:50 [ERROR] mysqld got signal 11 ;
 
...
 
Jul 18 20:07:50 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f49099dd88e]
 
Jul 18 20:07:50 /usr/sbin/mysqld(handle_fatal_signal+0x457)[0x7f49095c24a7]
 
Jul 18 20:07:50 /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f490800f340]
 
Jul 18 20:07:50 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0(CRYPTO_gcm128_setiv+0x88)[0x7f49082af5b8]
 
Jul 18 20:07:50 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0(+0xef8d7)[0x7f490830c8d7]
 
Jul 18 20:07:50 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0(EVP_CIPHER_CTX_ctrl+0x17)[0x7f4908308e17]
 
Jul 18 20:07:50 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0(+0xefc0c)[0x7f490830cc0c]
 
Jul 18 20:07:50 /lib/x86_64-linux-gnu/libssl.so.1.0.0(+0x2b189)[0x7f4908624189]
 
Jul 18 20:07:50 /lib/x86_64-linux-gnu/libssl.so.1.0.0(+0x211c6)[0x7f490861a1c6]
 
Jul 18 20:07:50 /lib/x86_64-linux-gnu/libssl.so.1.0.0(+0x215c3)[0x7f490861a5c3]
 
Jul 18 20:07:50 /usr/sbin/mysqld(vio_ssl_write+0x22)[0x7f4909a13782]
 
Jul 18 20:07:50 /usr/sbin/mysqld(net_real_write+0x101)[0x7f49093d57b1]
 
Jul 18 20:07:50 /usr/sbin/mysqld(_ZN11Query_cache21send_result_to_clientEP3THDPcj+0xd70)[0x7f4909423ef0]
 
Jul 18 20:07:50 /usr/sbin/mysqld(+0x396e6d)[0x7f4909469e6d]
 
Jul 18 20:07:50 /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1f78)[0x7f490946c648]
 
Jul 18 20:07:50 /usr/sbin/mysqld(_Z10do_commandP3THD+0x22f)[0x7f490946d17f]
 
Jul 18 20:07:50 /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x33e)[0x7f490952cf7e]
 
Jul 18 20:07:50 /usr/sbin/mysqld(handle_one_connection+0x4a)[0x7f490952d06a]
 
Jul 18 20:07:50 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7f4908007182]
 
Jul 18 20:07:50 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f490772a47d]
 
Jul 18 20:07:50 
Jul 18 20:07:50 Trying to get some variables.
 
Jul 18 20:07:50 Some pointers may be invalid and cause the dump to abort.
 
Jul 18 20:07:50 Query (0x7f48104ee018): is an invalid pointer
 
Jul 18 20:07:50 Connection ID (thread ID): 124278
 
Jul 18 20:07:50 Status: NOT_KILLED

Comment by Elena Stepanova [ 2020-10-20 ]

We have never been able to reproduce it, the stack trace is not distinctive enough to match it with other open or fixed bugs, and 5.5-galera is long gone now; so I'm closing it as incomplete. The issue can be re-opened if it gets fresh information related to active versions.

Generated at Thu Feb 08 07:41:55 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.