[MDEV-4279] Periodicaly MariaDB crashes Created: 2013-03-16  Updated: 2013-05-12  Due: 2013-05-30  Resolved: 2013-05-12

Status: Closed
Project: MariaDB Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Mikhail Paschenko (Inactive) Assignee: Sergei Golubchik
Resolution: Not a Bug Votes: 0
Labels: freebsd
Environment:

9.0-RELEASE-p4 amd64


 Description   

After upgrade from 5.3 to 5.5 MariaDB starts crash wish signal 10 about every hour, max uptime 20h. It was on 5.5.29 and nothing changed with 5.5.30.

Backtrace always the same in gdb:

(gdb) bt
 
#0  0x00000008022fb81e in _malloc_prefork () from /lib/libc.so.7
 
#1  0x0000000802300bd3 in free () from /lib/libc.so.7
 
#2  0x00000000009b2206 in free_root ()
 
#3  0x000000000056c9cf in dispatch_command ()
 
#4  0x000000000056dfe3 in do_command ()
 
#5  0x0000000000613072 in do_handle_one_connection ()
 
#6  0x00000000006131aa in handle_one_connection ()
 
#7  0x0000000802066274 in pthread_getprio () from /lib/libthr.so.3
 
#8  0x00007fffeb044000 in ?? ()

General log finishes with different queries before crash, Init DB or simple select.

We have 2 servers with Maria for HA with master-master, and this problem on both. Also have another server with another data and with the same problem.

For a first time we have used to use Aria SE, then converted to InnoDB/ExtaDB, but no effect on the problem.

What additional helpful info can I provide for you?

 Comments   
Comment by Elena Stepanova [ 2013-03-17 ]

Hi Sergei,

Since you recently looked into FreeBSD builds, maybe you have an idea what could cause the problem or advice on how to investigate it further?

I can wild-guess everything from a bad distribution/build to a lack of memory, but have no base for any of the guesses.

Comment by Sergei Golubchik [ 2013-03-20 ]

I have no idea either.

Mikhail, where did you get the MariaDB binaries? Did you compile it yourself? From our source tarball? Download a binary tarball from mariadb.org? Built from ports? Download a binary package from ports? Something else?

Comment by Mikhail Paschenko (Inactive) [ 2013-03-21 ]

I have built it from ports(databases/mariadb55-server) with default config options: SSL enabled, FASTMTX disabled.

Comment by Mikhail Paschenko (Inactive) [ 2013-03-22 ]

A few more details.
Very often we saw next backtrace:

#0  0x00000008022fb81e in _malloc_prefork () from /lib/libc.so.7
 
#1  0x00000008022fde31 in _malloc_prefork () from /lib/libc.so.7
 
#2  0x00000008022ffb9a in malloc () from /lib/libc.so.7
 
#3  0x00000000008bb73c in mem_heap_create_block (heap=0x0, n=1024, type=0, file_name=0xb3f848 "/usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/storage/xtradb/btr/btr0cur.c", line=2142)
 
    at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/storage/xtradb/mem/mem0mem.c:333
 
#4  0x00000000008bbc43 in mem_heap_create_func (n=Variable "n" is not available.
 
) at mem0mem.ic:439
 
#5  0x000000000086bedb in btr_cur_optimistic_update (flags=2, cursor=0x8d2807c78, update=0x8cddf4b30, cmpl_info=1, thr=0x8cddf4f08, mtr=0x7fffed33aa10) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/storage/xtradb/btr/btr0cur.c:2142
 
#6  0x000000000084552a in row_upd_clust_rec (node=0x8cddf4a28, index=0x8d10b6178, thr=0x8cddf4f08, mtr=0x7fffed33aa10) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/storage/xtradb/row/row0upd.c:1994
 
#7  0x00000000008470ca in row_upd_step (thr=0x8cddf4f08) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/storage/xtradb/row/row0upd.c:2276
 
#8  0x000000000083daf1 in row_update_for_mysql (mysql_rec=Variable "mysql_rec" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/storage/xtradb/row/row0mysql.c:1506
 
#9  0x0000000000821c8f in ha_innobase::update_row (this=0x8cdd83018, old_row=0x8cdcc3448 "?>v", new_row=0x8cdcc3418 "?>v") at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/storage/xtradb/handler/ha_innodb.cc:6480
 
#10 0x000000000069884e in handler::ha_update_row (this=0x8cdd83018, old_data=0x8cdcc3448 "?>v", new_data=0x8cdcc3418 "?>v") at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/handler.cc:5190
 
#11 0x00000000005e37e5 in multi_update::do_updates (this=0x8e7fdb0d0) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_update.cc:2210
 
#12 0x00000000005e3b70 in multi_update::send_eof (this=0x8e7fdb0d0) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_update.cc:2300
 
#13 0x000000000059d216 in do_select (join=0x8e7fdb190, fields=0x7fffed33b780, table=0x0, procedure=0x0) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_select.cc:15989
 
#14 0x00000000005acd7f in JOIN::exec (this=0x8e7fdb190) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_select.cc:2842
 
#15 0x00000000005af350 in mysql_select (thd=0x8b12f4000, rref_pointer_array=0x8b12f6aa8, tables=0x8ce7c0490, wild_num=0, fields=Variable "fields" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_select.cc:3062
 
#16 0x00000000005e1177 in mysql_multi_update (thd=0x8b12f4000, table_list=0x8ce7c0490, fields=0x8b12f6960, values=0x8b12f6dc8, conds=0x0, options=1342177408, handle_duplicates=DUP_ERROR, ignore=Variable "ignore" is not available.
 
)
 
    at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_update.cc:1414
 
#17 0x0000000000566bda in mysql_execute_command (thd=0x8b12f4000) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_parse.cc:2876
 
#18 0x000000000056c6a2 in mysql_parse (thd=0x8b12f4000, 
    rawbuf=0x8ce7c0018 "UPDATE `writer_discipline_ratings`\n\t\t\tINNER JOIN\n\t\t\t(\n\t\t\t\tSELECT\n\t\t\t\t"..., 
    length=Variable "length" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_parse.cc:5756
 
#19 0x000000000056dc5e in dispatch_command (command=COM_QUERY, thd=0x8b12f4000, packet=Variable "packet" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_parse.cc:1068
 
#20 0x000000000056dfe3 in do_command (thd=0x8b12f4000) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_parse.cc:794
 
#21 0x0000000000613072 in do_handle_one_connection (thd_arg=0x8f9829800) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_connect.cc:1266
 
#22 0x00000000006131aa in handle_one_connection (arg=Variable "arg" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_connect.cc:1181
 
#23 0x0000000802066274 in pthread_getprio () from /lib/libthr.so.3
 
#24 0x00007fffed2f4000 in ?? ()
 
Cannot access memory at address 0x7fffed33d000

Then we set innodb_use_sys_malloc=OFF and got new backtraces:

#0  0x00000008022fb81e in _malloc_prefork () from /lib/libc.so.7
 
#1  0x0000000802300bd3 in free () from /lib/libc.so.7
 
#2  0x00000000008612ab in ut_free (ptr=0x8b07ed320) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/storage/xtradb/ut/ut0mem.c:235
 
#3  0x00000000008bb53f in mem_heap_free_func (heap=0x8b07ed320, file_name=Variable "file_name" is not available.
 
) at mem0mem.ic:500
 
#4  0x000000000084865a in row_vers_impl_x_locked_off_kernel (rec=0x8552aae7f "\200", index=0x8db76db98, offsets=0x7fffedcc5010) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/storage/xtradb/row/row0vers.c:289
 
#5  0x00000000008aec1e in lock_rec_convert_impl_to_expl (block=0x8292fe980, rec=0x8552aae7f "\200", index=0x8db76db98, offsets=0x7fffedcc5010) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/storage/xtradb/lock/lock0lock.c:1630
 
#6  0x00000000008b1962 in lock_sec_rec_read_check_and_lock (flags=Variable "flags" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/storage/xtradb/lock/lock0lock.c:5628
 
#7  0x0000000000840e05 in sel_set_rec_lock (block=0x8292fe980, rec=0x8552aae7f "\200", index=0x8db76db98, offsets=0x7fffedcc5010, mode=3, type=1024, thr=0x8ed0e5190) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/storage/xtradb/row/row0sel.c:1009
 
#8  0x00000000008438c7 in row_search_for_mysql (buf=0x8f3814c18 "??\f", mode=2, prebuilt=0x8ed0e4898, match_mode=1, direction=1) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/storage/xtradb/row/row0sel.c:4436
 
#9  0x00000000008211ef in ha_innobase::general_fetch (this=0x919415018, buf=0x8f3814c18 "??\f", direction=Variable "direction" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/storage/xtradb/handler/ha_innodb.cc:7167
 
#10 0x0000000000585337 in join_read_next_same (info=Variable "info" is not available.
 
) at sql_class.h:4266
 
#11 0x000000000059cb09 in sub_select (join=0x8fee0e190, join_tab=0x8feded7e8, end_of_records=Variable "end_of_records" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_select.cc:16296
 
#12 0x000000000059c85a in evaluate_join_record (join=0x8fee0e190, join_tab=0x8feded4c8, error=Variable "error" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_select.cc:16496
 
#13 0x000000000059cb3b in sub_select (join=0x8fee0e190, join_tab=0x8feded4c8, end_of_records=Variable "end_of_records" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_select.cc:16316
 
#14 0x000000000059d08d in do_select (join=0x8fee0e190, fields=0x7fffedcc5780, table=0x0, procedure=0x0) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_select.cc:15947
 
#15 0x00000000005acd7f in JOIN::exec (this=0x8fee0e190) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_select.cc:2842
 
#16 0x00000000005af350 in mysql_select (thd=0x8b076d800, rref_pointer_array=0x8b07702a8, tables=0x8b6de3c90, wild_num=0, fields=Variable "fields" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_select.cc:3062
 
#17 0x00000000005e1177 in mysql_multi_update (thd=0x8b076d800, table_list=0x8b6de3c90, fields=0x8b0770160, values=0x8b07705c8, conds=0x0, options=1342177408, handle_duplicates=DUP_ERROR, ignore=Variable "ignore" is not available.
 
)
 
    at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_update.cc:1414
 
#18 0x0000000000566bda in mysql_execute_command (thd=0x8b076d800) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_parse.cc:2876
 
#19 0x000000000056c6a2 in mysql_parse (thd=0x8b076d800, 
    rawbuf=0x8b6de3818 "UPDATE `writer_discipline_ratings`\n\t\t\tINNER JOIN\n\t\t\t(\n\t\t\t\tSELECT\n\t\t\t\t"..., 
    length=Variable "length" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_parse.cc:5756
 
#20 0x000000000056dc5e in dispatch_command (command=COM_QUERY, thd=0x8b076d800, packet=Variable "packet" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_parse.cc:1068
 
#21 0x000000000056dfe3 in do_command (thd=0x8b076d800) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_parse.cc:794
 
#22 0x0000000000613072 in do_handle_one_connection (thd_arg=0x8f9138800) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_connect.cc:1266
 
#23 0x00000000006131aa in handle_one_connection (arg=Variable "arg" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_connect.cc:1181
 
#24 0x0000000802066274 in pthread_getprio () from /lib/libthr.so.3
 
#25 0x00007fffedc7e000 in ?? ()
 

#0  0x00000008022f8ec0 in _malloc_postfork () from /lib/libc.so.7
 
#1  0x00000008022fb2fd in _malloc_postfork () from /lib/libc.so.7
 
#2  0x00000008022fdb00 in _malloc_prefork () from /lib/libc.so.7
 
#3  0x00000008022ffb9a in malloc () from /lib/libc.so.7
 
#4  0x00000000009b9470 in my_malloc (size=4808, my_flags=4112) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/mysys/my_malloc.c:41
 
#5  0x00000000009b22db in alloc_root (mem_root=0x7fffedb9cf10, length=4784) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/mysys/my_alloc.c:217
 
#6  0x00000000009b24f5 in multi_alloc_root (root=Variable "root" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/mysys/my_alloc.c:281
 
#7  0x0000000000595fac in create_tmp_table (thd=0x90f03e800, param=0x9089bbc88, fields=@0x8f01420e0, group=0x0, distinct=false, save_sum_fields=true, select_options=2416188160, rows_limit=18446744073709551615, table_alias=0x8fc06f528 "sr", do_not_open=true)
 
    at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_select.cc:14374
 
#8  0x00000000005dcd2b in select_union::create_result_table (this=0x9089bbc70, thd_arg=0x90f03e800, column_types=0x8f01420e0, is_union_distinct=Variable "is_union_distinct" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_union.cc:150
 
#9  0x0000000000548268 in mysql_derived_prepare (thd=0x90f03e800, lex=Variable "lex" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_derived.cc:648
 
#10 0x0000000000548b36 in mysql_handle_single_derived (lex=0x90f0408c8, derived=0x8fc06f590, phases=2) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_derived.cc:179
 
#11 0x00000000005eb08c in TABLE_LIST::handle_derived (this=0x8fc06f590, lex=0x90f0408c8, phases=2) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/table.cc:6531
 
#12 0x00000000005599e5 in st_select_lex::handle_derived (this=Variable "this" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_lex.cc:3522
 
#13 0x0000000000593119 in JOIN::prepare (this=0x8fc071dd0, rref_pointer_array=0x90f0412a8, tables_init=Variable "tables_init" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_select.cc:627
 
#14 0x00000000005af9f2 in mysql_select (thd=0x90f03e800, rref_pointer_array=0x90f0412a8, tables=0x8fc06f590, wild_num=0, fields=@0x90f041160, conds=0x8fc071aa0, og_num=1, order=0x8fc071ca8, group=0x0, having=0x0, proc_param=0x0, select_options=536937153, 
    result=0x8fc071db0, unit=0x90f040978, select_lex=0x90f041050) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_select.cc:3040
 
#15 0x00000000005afe6e in handle_select (thd=0x90f03e800, lex=0x90f0408c8, result=0x8fc071db0, setup_tables_done_option=0) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_select.cc:318
 
#16 0x0000000000560d03 in execute_sqlcom_select (thd=0x90f03e800, all_tables=0x8fc06f590) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_parse.cc:4641
 
#17 0x0000000000568bac in mysql_execute_command (thd=0x90f03e800) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_parse.cc:2195
 
#18 0x000000000056c6a2 in mysql_parse (thd=0x90f03e800, 
    rawbuf=0x8e5dc5818 "SELECT w.fname, w.lname, w.category, sr.writer_id, sr.reassign_id, FROM_UNIXTIME(sr.reassigned_at) reassigned_at, sr.unrequested_sum, sr.requested_sum, sr.paid_sum, sr.paid, sr.cost\r\n\tFROM (\r\n\t\tSELECT"..., length=Variable "length" is not available.
 
)
 
    at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_parse.cc:5756
 
#19 0x000000000056dc5e in dispatch_command (command=COM_QUERY, thd=0x90f03e800, packet=Variable "packet" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_parse.cc:1068
 
#20 0x000000000056dfe3 in do_command (thd=0x90f03e800) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_parse.cc:794
 
#21 0x0000000000613072 in do_handle_one_connection (thd_arg=0x90c8e2800) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_connect.cc:1266
 
#22 0x00000000006131aa in handle_one_connection (arg=Variable "arg" is not available.
 
) at /usr/ports/databases/mariadb55-server/work/mariadb-5.5.30/sql/sql_connect.cc:1181
 
#23 0x0000000802066274 in pthread_getprio () from /lib/libthr.so.3
 
#24 0x0000000000000000 in ?? ()

It looks like mysqld crashes when resident set size(RSS) reaches virtual size(VSZ).

We can provide core dumps, if it helps.

Comment by Sergei Golubchik [ 2013-05-02 ]

Do you still have this issue? Does it happen only with MariaDB or with other memory-hungry programs too?
Because this looks like a heap corruption, and it also looks like MariaDB is not at fault here. You could try, for example, to run a small program that only allocates and bfill-s memory in blocks, and see whether it'll crash when RSS reaches VSZ.

Comment by Mikhail Paschenko (Inactive) [ 2013-05-12 ]

We have found an error in our UDF function and problem completly solved after UDF fix.
Close this task, please.
Thanks for help.

Generated at Thu Feb 08 06:55:11 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.