[MDEV-10410] MariaDB crashes with TokuDB and ANALYZE TABLE in background. Created: 2016-07-21  Updated: 2016-11-24  Resolved: 2016-11-24

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - TokuDB
Affects Version/s: 10.1.16
Fix Version/s: 10.1.18

Type: Bug Priority: Major
Reporter: jocelyn fournier Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: upstream

Attachments: File dump.sql     File my.cnf    

 Description   

Hi,

I've enabled the ANALYZE TABLE background task for TokuDB :

SET GLOBAL tokudb_analyze_in_background=ON;

Then I ran a mysqlcheck -a -A.

And to check the ANALYZE TABLE task status, I use :

SELECT * FROM INFORMATION_SCHEMA.TOKUDB_BACKGROUND_JOB_STATUS;

This command crashes MariaDB :

MariaDB [(none)]> SELECT * FROM INFORMATION_SCHEMA.TOKUDB_BACKGROUND_JOB_STATUS;
 
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
 
======= Backtrace: =========
 
/lib/x86_64-linux-gnu/libc.so.6(+0x7329f)[0x7f91bb70329f]
 
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f91bb79abbc]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x109a90)[0x7f91bb799a90]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x10ab07)[0x7f91bb79ab07]
 
/usr/sbin/mysqld(my_addr_resolve+0x48)[0x7f91be41f3a8]
 
/usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x7f91be40ba92]
 
/usr/sbin/mysqld(handle_fatal_signal+0x2d5)[0x7f91bdf32e75]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f91bc273330]
 
/lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x1d03)[0x7f91bb6db943]
 
/lib/x86_64-linux-gnu/libc.so.6(__vsprintf_chk+0x84)[0x7f91bb799024]
 
/lib/x86_64-linux-gnu/libc.so.6(__sprintf_chk+0x7d)[0x7f91bb798f7d]
 
/usr/lib/mysql/plugin/ha_tokudb.so(_ZN6tokudb7analyze10standard_t18get_analyze_statusEPc+0x124)[0x7f91b4c9fc74]
 
/usr/lib/mysql/plugin/ha_tokudb.so(_ZNK6tokudb10background13job_manager_t12iterate_jobsEPFvmPKcS3_S3_S3_S3_bllPvES4_+0xfa)[0x7f91b4cc88ea]
 
/usr/lib/mysql/plugin/ha_tokudb.so(_ZN6tokudb18information_schema28report_background_job_statusEP5TABLEP3THD+0x2a)[0x7f91b4ccc4ca]
 
/usr/lib/mysql/plugin/ha_tokudb.so(_ZN6tokudb18information_schema32background_job_status_fill_tableEP3THDP10TABLE_LISTP4Item+0x13b)[0x7f91b4ccc60b]
 
/usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x28e)[0x7f91bde226be]
 
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0x6bd)[0x7f91bde08c3d]
 
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x54)[0x7f91bde0b0c4]
 
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x112)[0x7f91bde07782]
 
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x245)[0x7f91bde08255]
 
/usr/sbin/mysqld(+0x428291)[0x7f91bddaa291]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x61b9)[0x7f91bddb60c9]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x7f91bddb973d]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x7f91bddbca80]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7f91bddbd239]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7f91bde8189a]
 
/usr/sbin/mysqld(handle_one_connection+0x40)[0x7f91bde81a70]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f91bc26b184]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f91bb78a37d]

Thanks and regards,
Jocelyn Fournier

 Comments   
Comment by Elena Stepanova [ 2016-07-21 ]

Apparently it depends on the schema. How big is yours? Would you be able to upload it to our ftp?

Please also attach your cnf file(s).

Comment by jocelyn fournier [ 2016-07-21 ]

The tar.gz of the tokudb backup is about 270GB

Comment by Elena Stepanova [ 2016-07-21 ]

Yeah, it's unrealistic.
Could you maybe create a schema dump (DDL only, without a data) and attach it? I'm not very hopeful, but maybe we'll be able to reproduce it with smaller random data.

Comment by jocelyn fournier [ 2016-07-21 ]

I've just attached the my.cnf

my.cnf

Comment by jocelyn fournier [ 2016-07-21 ]

I have to restore an offline slave with the latest backup, I'll try to use it to simplify the testcase with fewer tables, if possible.

Comment by jocelyn fournier [ 2016-07-21 ]

I've also attached the schema dump.
Some tables are using partitions with tokudb, perhaps there are some issues with them & analyze, as there's a particular treatment for them. (if you didn't succeed in reproducing the issue with the schema, I'll try to work on a simplified testcase)

dump.sql

Comment by jocelyn fournier [ 2016-07-21 ]

Hi Elena,

If I import a table big enough to have the time to run the SELECT * FROM INFORMATION_SCHEMA.TOKUDB_BACKGROUND_JOB_STATUS;, import the schema dump, run an analyze on all the table in background, and finally run the SELECT * FROM INFORMATION_SCHEMA.TOKUDB_BACKGROUND_JOB_STATUS; command, it always crashes.
I wonder if it's not just the number of tables which will be displayed which is important here.

HTH,
Jocelyn

Comment by Rich Prohaska [ 2016-07-22 ]

Problem is caused by an sprintf buffer overflow. Code needs to use snprintf. See https://tokutek.atlassian.net/browse/DB-1005

Comment by Elena Stepanova [ 2016-07-22 ]

prohaska7, thanks!
So I guess we just wait till the upstream bug is fixed, and merge as usual.

Comment by jocelyn fournier [ 2016-11-24 ]

This one seems to be fixed now

Comment by Elena Stepanova [ 2016-11-24 ]

joce,

Thanks for the update.
Closing as fixed in 10.1.18 because that was the last time TokuDB merge happened in 10.1 (5.6.32-78.1).

Generated at Thu Feb 08 07:42:01 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.