[MDEV-14248] Segfault and access to data Created: 2017-11-02  Updated: 2018-03-19  Resolved: 2018-03-19

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.2.9, 10.2.10
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Gaëtan SL Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: need_feedback
Environment:

CentOS 7 x64

Attachments: File crash.log    

 Description   

Hi,

Here is the dmesg log we get each time of doing mysqldump or simple "SELECT" :

[65080.400295] mysqld[24605]: segfault at a0 ip 00005631c416d13d sp 00007fdb441fd160 error 4 in mysqld[5631c37a1000+1124000]

The data inside the DB has been "freshly imported" a few days ago from a SQL dump. Now the server is on prod but impossible to make a backup or access to some records :

{{MariaDB [otrs2]> select content from article_attachment where id = 32669;

ERROR 2013 (HY000): Lost connection to MySQL server during query

MariaDB [otrs2]> update article_attachment set content = 'xx' where id = 32669;
ERROR 2013 (HY000): Lost connection to MySQL server during query

MariaDB [otrs2]> delete from article_attachment where id = 32669;
ERROR 2013 (HY000): Lost connection to MySQL server during query}}

Tested a lot of parameters. Here is the current ones :

[mysqld]
innodb_file_per_table=1
innodb_log_buffer_size = 32M
innodb_buffer_pool_size = 12G
innodb_log_file_size = 768M
innodb_thread_concurrency = 16
innodb_flush_log_at_trx_commit = 2
innodb_fast_shutdown = 0
max_allowed_packet = 1G
net_read_timeout = 7200
net_write_timeout = 7200
#innodb_force_recovery = 1

Thanks

 Comments   
Comment by Daniel Black [ 2017-11-02 ]

Can you please include the mariadb errorlog.

Also 10.2.10 was released yesterday and if convienent could you see if you get a segfault after this update

Comment by Gaëtan SL [ 2017-11-02 ]

Hi,

Thank you for your answer.

I updated yesteday. issue also happens on 10.1

I just uploaded the crash.log

Thank you !

Comment by Daniel Black [ 2017-11-03 ]

This looks like a massively corrupted innodb files from the datadir. Can you describe the history of your datadir. What mysql/mariadb version what initially installed? What version upgrades/downgrades have you done? If you have attempted to correct these errors though manual manipulation of the datadir files what has been done? Are you experiencing a storage failure? What filesystem is the datadir on?

Comment by Gaëtan SL [ 2017-11-03 ]

Hi,

Thank you for your answer. Strange. This is a "fresh" SQL import from last week. I made some tests before but before the import I did a DROP table.

I was using mariadb 10.1 until the begining of the week. Tried to update to 10.2 to solve the issue (with no luck).

No manual manipulation done and no storage failure. The filesystem is XFS. A this time I only found one record in this DB which makes the SGBD crash.

This is a very critical issue because I cannot do the SQL dump and import again ... And the application is in production is working properly unless we access to this record..

What could we do ?

Thanks !

Comment by Gaëtan SL [ 2017-11-07 ]

Hi,

In test environment I tried to exctract what is possible. I was able to extract a lot of data, but now here is the messages I have :

MariaDB [otrs2]> select article_id from article_attachment where id = 32669;
ERROR 1030 (HY000): Got error 1877 "Unknown error 1877" from storage engine InnoDB
MariaDB [otrs2]> select article_id from article_attachment;
ERROR 1712 (HY000): Index article_attachment is corrupted

This is very critical at that time

Comment by Elena Stepanova [ 2017-11-20 ]

gslongo,
If that's a database freshly imported a few days ago, could you please provide the whole error log, beginning from the server startup before it was imported, and up to and including the failures that you are getting?
Thanks.

Comment by Elena Stepanova [ 2017-12-19 ]

Crash report from the log (to make it searchable in JIRA):

171102 10:54:53 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.9-MariaDB
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=11
 
max_threads=153
 
thread_count=17
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467216 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x559b36918008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fa5282ded70 thread_stack 0x49000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x559b20f6e3ee]
 
/usr/sbin/mysqld(handle_fatal_signal+0x30d)[0x559b209eb6fd]
 
/lib64/libpthread.so.0(+0xf5e0)[0x7fa544ce65e0]
 
/usr/sbin/mysqld(+0x9cc13d)[0x559b20d4f13d]
 
/usr/sbin/mysqld(+0x9cc69d)[0x559b20d4f69d]
 
/usr/sbin/mysqld(+0x42273e)[0x559b207a573e]
 
/usr/sbin/mysqld(+0x4229e3)[0x559b207a59e3]
 
/usr/sbin/mysqld(+0x92d282)[0x559b20cb0282]
 
/usr/sbin/mysqld(+0x84fd6a)[0x559b20bd2d6a]
 
/usr/sbin/mysqld(_ZN7handler18index_read_idx_mapEPhjPKhm16ha_rkey_function+0x85)[0x559b209ec055]
 
mysys/stacktrace.c:268(my_print_stacktrace)[0x559b209f07ec]
 
btr/btr0cur.cc:7446(btr_copy_blob_prefix(unsigned char*, unsigned long, unsigned long, unsigned long, unsigned long))[0x559b2089fc34]
 
row/row0sel.cc:3232(row_sel_store_mysql_rec(unsigned char*, row_prebuilt_t*, unsigned char const*, dtuple_t const*, bool, dict_index_t const*, unsigned long const*))[0x559b2089fd99]
 
handler/ha_innodb.cc:9922(ha_innobase::index_read(unsigned char*, unsigned char const*, unsigned int, ha_rkey_function))[0x559b208a74bb]
 
sql/handler.cc:5489(handler::index_read_idx_map(unsigned char*, unsigned int, unsigned char const*, unsigned long, ha_rkey_function))[0x559b208add70]
 
/usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x3f)[0x559b208b004f]
 
sql/sql_select.cc:19140(join_read_const(st_join_table*))[0x559b208b1754]
 
sql/sql_select.cc:373(/mariadb-10.2.9/sql/sql_select.cc:3664)[0x559b208b2334]
 
/usr/sbin/mysqld(+0x4156a4)[0x559b207986a4]
 
sql/sql_parse.cc:6435(execute_sqlcom_select(THD*, TABLE_LIST*))[0x559b20862137]
 
sql/sql_parse.cc:7876(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x559b20864bbe]
 
sql/sql_parse.cc:1812(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x559b20867b8d]
 
sql/sql_parse.cc:1362(do_command(THD*))[0x559b20868799]
 
sql/sql_connect.cc:1354(do_handle_one_connection(CONNECT*))[0x559b2092bc1a]
 
sql/sql_connect.cc:1262(handle_one_connection)[0x559b2092bd3d]
 
/lib64/libpthread.so.0(+0x7e25)[0x7fa544cdee25]
 
/lib64/libc.so.6(clone+0x6d)[0x7fa5432b634d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x559b36925d60): SELECT content_type, content, content_id, content_alternative, disposition, filename
 
            FROM article_attachment
 
            WHERE id = '32669'
 
 
 
Connection ID (thread ID): 25
 
Status: NOT_KILLED

Generated at Thu Feb 08 08:12:05 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.