[MDEV-9446] Fatal crash of mariadb when using JSON functionality Created: 2016-01-22  Updated: 2016-01-30  Resolved: 2016-01-25

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Connect
Affects Version/s: 10.1.10
Fix Version/s: 10.0.24, 10.1.12

Type: Bug Priority: Major
Reporter: Norbert van Nobelen Assignee: Olivier Bertrand
Resolution: Fixed Votes: 0
Labels: json
Environment:

os: ubuntu 14.04
Mariadb installation: mariadb-10.1.10-linux-x86_64.tar.gz
unixodbc: /var/cache/apt/archives/unixodbc_2.3.1-3_amd64.deb



 Description   

Scenario:

Install the functions (just 1 shown, complete set installed as indicated in documentation) :
create function jsonvalue returns string soname 'ha_connect.so';
Run the function:

select jsonvalue(3.14);

Result: mysqld dies with sig 11

Error:

Version: '10.1.10-MariaDB-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MariaDB Server
160121 16:21:36 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.1.10-MariaDB-log
key_buffer_size=134217728
read_buffer_size=134217728
max_used_connections=1
max_threads=252
thread_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 66196487 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x7f2b30383008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f2b4a6fce88 thread_stack 0x100000
./mysqld(my_print_stacktrace+0x2e)[0xbfcb7e]
./mysqld(handle_fatal_signal+0x464)[0x762094]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10d10)[0x7f2b4ddb8d10]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f2b28637020): is an invalid pointer
Connection ID (thread ID): 4
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Config

[client]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock
 
[mysqld_safe]
socket          = /var/run/mysqld/mysqld.sock
nice            = 0
 
[mysqld]
innodb         = on
event_scheduler    = on
default-storage-engine = InnoDB
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp location
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address            = 127.0.0.1
#
key_buffer_size      = 128M
sort_buffer_size    = 128M
read_buffer_size    = 128M
read_rnd_buffer_size = 128M
max_allowed_packet      = 16M
thread_stack            = 1024K
thread_cache_size       = 0
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover         = BACKUP
max_connections        = 250
table_cache            = 1024
table_open_cache    = 512
thread_concurrency     = 8
#
# * Query Cache Configuration
#
query_cache_type     = 1
query_cache_limit       = 128K
query_cache_size        = 4M
 
# Logging:
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
 
slow_query_log        = On
slow_query_log_file      = /var/log/mysql/mysql-slow.log
long_query_time = 60
log_queries_not_using_indexes
 
#Allow command line access to the database:
plugin-load-add         = unix_socket=auth_socket.so
 
#Master slave settings:
#Master:
server-id              = 1
log_bin              = /var/log/mysql/mysql-bin.log
expire_logs_days        = 10
max_binlog_size         = 100M
binlog_do_db           = adacado
binlog_do_db            = adacado_dimension_replica
binlog_format           = mixed
 
# Send feedback to MariaDB.org:
feedback = Off
 
#InnoDB settings:
[InnoDB]
innodb_buffer_pool_size = 1017287424
innodb_buffer_pool_instances = 4
innodb_log_file_size = 64M
innodb_log_files_in_group = 4
innodb_log_buffer_size = 32M
innodb_flush_log_at_trx_commit = 1
innodb_file_per_table = On
innodb_file_format = Barracuda
# InnoDB Performance options:
innodb_thread_concurrency = 8
innodb_concurrency_tickets = 5000
innodb_adaptive_hash_index = On
innodb_adaptive_hash_index_parts = 16
 
[mysqldump]
quick
quote-names
max_allowed_packet      = 16M
 
[mysql]
 
[isamchk]
key_buffer              = 16M



 Comments   
Comment by Elena Stepanova [ 2016-01-22 ]

Thanks for the report.

Stack trace from 10.0 commit 74b1af19e9270a932f9e5f907eff53852984a7d3

#2  0x000000000085a8c8 in handle_fatal_signal (sig=11) at /src/10.0/sql/signal_handler.cc:262
#3  <signal handler called>
#4  0x0000000000000000 in ?? ()
#5  0x00007f3b174ee844 in GetTraceValue () at /src/10.0/storage/connect/ha_connect.cc:345
#6  0x00007f3b175089c7 in PlugInit (Language=0x0, worksize=4306) at /src/10.0/storage/connect/plugutil.c:138
#7  0x00007f3b1753b0d3 in JsonInit (initid=0x7f3b1a4d3360, args=0x7f3b1a4d3320, message=0x7f3b43658140 "@\202eC;\177", mbn=0 '\000', reslen=14, memlen=4306) at /src/10.0/storage/connect/jsonudf.cpp:1077
#8  0x00007f3b1753cede in jsonvalue_init (initid=0x7f3b1a4d3360, args=0x7f3b1a4d3320, message=0x7f3b43658140 "@\202eC;\177") at /src/10.0/storage/connect/jsonudf.cpp:1703
#9  0x00000000008c87a1 in udf_handler::fix_fields (this=0x7f3b1a4d3310, thd=0x7f3b291b52f0, func=0x7f3b1a4d3248, arg_count=1, arguments=0x7f3b1a4d32e0) at /src/10.0/sql/item_func.cc:3610
#10 0x00000000008d3808 in Item_udf_func::fix_fields (this=0x7f3b1a4d3248, thd=0x7f3b291b52f0, ref=0x7f3b1a4d33b8) at /src/10.0/sql/item_func.h:1389
#11 0x00000000006169b0 in setup_fields (thd=0x7f3b291b52f0, ref_pointer_array=0x7f3b1a4d39d0, fields=..., mark_used_columns=MARK_COLUMNS_READ, sum_func_list=0x7f3b1a4d37d0, allow_sum_func=true) at /src/10.0/sql/sql_base.cc:7918
#12 0x00000000006a0b1d in JOIN::prepare (this=0x7f3b1a4d3488, rref_pointer_array=0x7f3b291b95c8, tables_init=0x0, wild_num=0, conds_init=0x0, og_num=0, order_init=0x0, skip_order_by=false, group_init=0x0, having_init=0x0, proc_param_init=0x0, select_lex_arg=0x7f3b291b9350, unit_arg=0x7f3b291b8c60) at /src/10.0/sql/sql_select.cc:785
#13 0x00000000006a998b in mysql_select (thd=0x7f3b291b52f0, rref_pointer_array=0x7f3b291b95c8, tables=0x0, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0x7f3b1a4d3468, unit=0x7f3b291b8c60, select_lex=0x7f3b291b9350) at /src/10.0/sql/sql_select.cc:3286
#14 0x000000000069fdb9 in handle_select (thd=0x7f3b291b52f0, lex=0x7f3b291b8b98, result=0x7f3b1a4d3468, setup_tables_done_option=0) at /src/10.0/sql/sql_select.cc:373
#15 0x000000000067466c in execute_sqlcom_select (thd=0x7f3b291b52f0, all_tables=0x0) at /src/10.0/sql/sql_parse.cc:5304
#16 0x000000000066ca55 in mysql_execute_command (thd=0x7f3b291b52f0) at /src/10.0/sql/sql_parse.cc:2562
#17 0x00000000006771f3 in mysql_parse (thd=0x7f3b291b52f0, rawbuf=0x7f3b1a4d3088 "select jsonvalue(3.14)", length=22, parser_state=0x7f3b436596a0) at /src/10.0/sql/sql_parse.cc:6565
#18 0x0000000000669cbf in dispatch_command (command=COM_QUERY, thd=0x7f3b291b52f0, packet=0x7f3b1a4b3e71 "select jsonvalue(3.14)", packet_length=22) at /src/10.0/sql/sql_parse.cc:1308
#19 0x0000000000668f8c in do_command (thd=0x7f3b291b52f0) at /src/10.0/sql/sql_parse.cc:998
#20 0x00000000007872c5 in do_handle_one_connection (thd_arg=0x7f3b291b52f0) at /src/10.0/sql/sql_connect.cc:1377
#21 0x0000000000787037 in handle_one_connection (arg=0x7f3b291b52f0) at /src/10.0/sql/sql_connect.cc:1292
#22 0x00007f3b432e90a4 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f3b4167d04d in clone () from /lib64/libc.so.6

Comment by Olivier Bertrand [ 2016-01-22 ]

Apparently the crash occurs in the function GetTraceValue() that is defined as:

extern "C" int GetTraceValue(void) {return THDVAR(current_thd, xtrace);}

It is used to get the value of the connect_xtrace variable. This function is used in ALL connect queries and until now has never failed.
The only explanation I see is if a json udf is created and used without the CONNECT storage engine being installed. Unfortunately I cannot test this because I did not find a way to de-install a plugin engine and starting mysqld with --skip-connect does not work.
Can you confirm this is the case so I can try to find a fix.
In that case the turnaround is obvious: install CONNECT.

Comment by Olivier Bertrand [ 2016-01-22 ]

Downloading a new MariaDB from scratch, I could make it crash by creating and using a Json UDF while CONNECT was not installed.
And verify that installing CONNECT fixes the problem.

Comment by Olivier Bertrand [ 2016-01-25 ]

To make Json UDFs usable without CONNECT being installed, the way Json group size is handled was modified.
The session variable connect_json_grp_size is now deprecated and will not be kept in future version.
The new value called JsonGrpSize used by the Json aggregate functions is now set and can be queried using two new functions jsonset_grp_size and jsonget_grp_size.
For older applications, the use of the session variable connect_json_grp_size can be temporarily continued by setting JsonGrpSize to 0 but only when CONNECT is installed.

Generated at Thu Feb 08 07:34:44 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.