Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9446

Fatal crash of mariadb when using JSON functionality

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.1.10
    • 10.0.24, 10.1.12
    • os: ubuntu 14.04
      Mariadb installation: mariadb-10.1.10-linux-x86_64.tar.gz
      unixodbc: /var/cache/apt/archives/unixodbc_2.3.1-3_amd64.deb

    Description

      Scenario:

      Install the functions (just 1 shown, complete set installed as indicated in documentation) :
      create function jsonvalue returns string soname 'ha_connect.so';
      Run the function:

      select jsonvalue(3.14);

      Result: mysqld dies with sig 11

      Error:

      Version: '10.1.10-MariaDB-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MariaDB Server
      160121 16:21:36 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
       
      Server version: 10.1.10-MariaDB-log
      key_buffer_size=134217728
      read_buffer_size=134217728
      max_used_connections=1
      max_threads=252
      thread_count=2
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 66196487 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0x7f2b30383008
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f2b4a6fce88 thread_stack 0x100000
      ./mysqld(my_print_stacktrace+0x2e)[0xbfcb7e]
      ./mysqld(handle_fatal_signal+0x464)[0x762094]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x10d10)[0x7f2b4ddb8d10]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7f2b28637020): is an invalid pointer
      Connection ID (thread ID): 4
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
       
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.

      Config

      [client]
      port            = 3306
      socket          = /var/run/mysqld/mysqld.sock
       
      [mysqld_safe]
      socket          = /var/run/mysqld/mysqld.sock
      nice            = 0
       
      [mysqld]
      innodb         = on
      event_scheduler    = on
      default-storage-engine = InnoDB
      user            = mysql
      pid-file        = /var/run/mysqld/mysqld.pid
      socket          = /var/run/mysqld/mysqld.sock
      port            = 3306
      basedir         = /usr
      datadir         = /var/lib/mysql
      tmpdir          = /tmp location
      lc-messages-dir = /usr/share/mysql
      skip-external-locking
      #
      # Instead of skip-networking the default is now to listen only on
      # localhost which is more compatible and is not less secure.
      #bind-address            = 127.0.0.1
      #
      key_buffer_size      = 128M
      sort_buffer_size    = 128M
      read_buffer_size    = 128M
      read_rnd_buffer_size = 128M
      max_allowed_packet      = 16M
      thread_stack            = 1024K
      thread_cache_size       = 0
      # This replaces the startup script and checks MyISAM tables if needed
      # the first time they are touched
      myisam-recover         = BACKUP
      max_connections        = 250
      table_cache            = 1024
      table_open_cache    = 512
      thread_concurrency     = 8
      #
      # * Query Cache Configuration
      #
      query_cache_type     = 1
      query_cache_limit       = 128K
      query_cache_size        = 4M
       
      # Logging:
      #
      # Error log - should be very few entries.
      #
      log_error = /var/log/mysql/error.log
       
      slow_query_log        = On
      slow_query_log_file      = /var/log/mysql/mysql-slow.log
      long_query_time = 60
      log_queries_not_using_indexes
       
      #Allow command line access to the database:
      plugin-load-add         = unix_socket=auth_socket.so
       
      #Master slave settings:
      #Master:
      server-id              = 1
      log_bin              = /var/log/mysql/mysql-bin.log
      expire_logs_days        = 10
      max_binlog_size         = 100M
      binlog_do_db           = adacado
      binlog_do_db            = adacado_dimension_replica
      binlog_format           = mixed
       
      # Send feedback to MariaDB.org:
      feedback = Off
       
      #InnoDB settings:
      [InnoDB]
      innodb_buffer_pool_size = 1017287424
      innodb_buffer_pool_instances = 4
      innodb_log_file_size = 64M
      innodb_log_files_in_group = 4
      innodb_log_buffer_size = 32M
      innodb_flush_log_at_trx_commit = 1
      innodb_file_per_table = On
      innodb_file_format = Barracuda
      # InnoDB Performance options:
      innodb_thread_concurrency = 8
      innodb_concurrency_tickets = 5000
      innodb_adaptive_hash_index = On
      innodb_adaptive_hash_index_parts = 16
       
      [mysqldump]
      quick
      quote-names
      max_allowed_packet      = 16M
       
      [mysql]
       
      [isamchk]
      key_buffer              = 16M

      Attachments

        Activity

          elenst Elena Stepanova added a comment - - edited

          Thanks for the report.

          Stack trace from 10.0 commit 74b1af19e9270a932f9e5f907eff53852984a7d3

          #2  0x000000000085a8c8 in handle_fatal_signal (sig=11) at /src/10.0/sql/signal_handler.cc:262
          #3  <signal handler called>
          #4  0x0000000000000000 in ?? ()
          #5  0x00007f3b174ee844 in GetTraceValue () at /src/10.0/storage/connect/ha_connect.cc:345
          #6  0x00007f3b175089c7 in PlugInit (Language=0x0, worksize=4306) at /src/10.0/storage/connect/plugutil.c:138
          #7  0x00007f3b1753b0d3 in JsonInit (initid=0x7f3b1a4d3360, args=0x7f3b1a4d3320, message=0x7f3b43658140 "@\202eC;\177", mbn=0 '\000', reslen=14, memlen=4306) at /src/10.0/storage/connect/jsonudf.cpp:1077
          #8  0x00007f3b1753cede in jsonvalue_init (initid=0x7f3b1a4d3360, args=0x7f3b1a4d3320, message=0x7f3b43658140 "@\202eC;\177") at /src/10.0/storage/connect/jsonudf.cpp:1703
          #9  0x00000000008c87a1 in udf_handler::fix_fields (this=0x7f3b1a4d3310, thd=0x7f3b291b52f0, func=0x7f3b1a4d3248, arg_count=1, arguments=0x7f3b1a4d32e0) at /src/10.0/sql/item_func.cc:3610
          #10 0x00000000008d3808 in Item_udf_func::fix_fields (this=0x7f3b1a4d3248, thd=0x7f3b291b52f0, ref=0x7f3b1a4d33b8) at /src/10.0/sql/item_func.h:1389
          #11 0x00000000006169b0 in setup_fields (thd=0x7f3b291b52f0, ref_pointer_array=0x7f3b1a4d39d0, fields=..., mark_used_columns=MARK_COLUMNS_READ, sum_func_list=0x7f3b1a4d37d0, allow_sum_func=true) at /src/10.0/sql/sql_base.cc:7918
          #12 0x00000000006a0b1d in JOIN::prepare (this=0x7f3b1a4d3488, rref_pointer_array=0x7f3b291b95c8, tables_init=0x0, wild_num=0, conds_init=0x0, og_num=0, order_init=0x0, skip_order_by=false, group_init=0x0, having_init=0x0, proc_param_init=0x0, select_lex_arg=0x7f3b291b9350, unit_arg=0x7f3b291b8c60) at /src/10.0/sql/sql_select.cc:785
          #13 0x00000000006a998b in mysql_select (thd=0x7f3b291b52f0, rref_pointer_array=0x7f3b291b95c8, tables=0x0, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0x7f3b1a4d3468, unit=0x7f3b291b8c60, select_lex=0x7f3b291b9350) at /src/10.0/sql/sql_select.cc:3286
          #14 0x000000000069fdb9 in handle_select (thd=0x7f3b291b52f0, lex=0x7f3b291b8b98, result=0x7f3b1a4d3468, setup_tables_done_option=0) at /src/10.0/sql/sql_select.cc:373
          #15 0x000000000067466c in execute_sqlcom_select (thd=0x7f3b291b52f0, all_tables=0x0) at /src/10.0/sql/sql_parse.cc:5304
          #16 0x000000000066ca55 in mysql_execute_command (thd=0x7f3b291b52f0) at /src/10.0/sql/sql_parse.cc:2562
          #17 0x00000000006771f3 in mysql_parse (thd=0x7f3b291b52f0, rawbuf=0x7f3b1a4d3088 "select jsonvalue(3.14)", length=22, parser_state=0x7f3b436596a0) at /src/10.0/sql/sql_parse.cc:6565
          #18 0x0000000000669cbf in dispatch_command (command=COM_QUERY, thd=0x7f3b291b52f0, packet=0x7f3b1a4b3e71 "select jsonvalue(3.14)", packet_length=22) at /src/10.0/sql/sql_parse.cc:1308
          #19 0x0000000000668f8c in do_command (thd=0x7f3b291b52f0) at /src/10.0/sql/sql_parse.cc:998
          #20 0x00000000007872c5 in do_handle_one_connection (thd_arg=0x7f3b291b52f0) at /src/10.0/sql/sql_connect.cc:1377
          #21 0x0000000000787037 in handle_one_connection (arg=0x7f3b291b52f0) at /src/10.0/sql/sql_connect.cc:1292
          #22 0x00007f3b432e90a4 in start_thread () from /lib64/libpthread.so.0
          #23 0x00007f3b4167d04d in clone () from /lib64/libc.so.6

          elenst Elena Stepanova added a comment - - edited Thanks for the report. Stack trace from 10.0 commit 74b1af19e9270a932f9e5f907eff53852984a7d3 #2 0x000000000085a8c8 in handle_fatal_signal (sig=11) at /src/10.0/sql/signal_handler.cc:262 #3 <signal handler called> #4 0x0000000000000000 in ?? () #5 0x00007f3b174ee844 in GetTraceValue () at /src/10.0/storage/connect/ha_connect.cc:345 #6 0x00007f3b175089c7 in PlugInit (Language=0x0, worksize=4306) at /src/10.0/storage/connect/plugutil.c:138 #7 0x00007f3b1753b0d3 in JsonInit (initid=0x7f3b1a4d3360, args=0x7f3b1a4d3320, message=0x7f3b43658140 "@\202eC;\177", mbn=0 '\000', reslen=14, memlen=4306) at /src/10.0/storage/connect/jsonudf.cpp:1077 #8 0x00007f3b1753cede in jsonvalue_init (initid=0x7f3b1a4d3360, args=0x7f3b1a4d3320, message=0x7f3b43658140 "@\202eC;\177") at /src/10.0/storage/connect/jsonudf.cpp:1703 #9 0x00000000008c87a1 in udf_handler::fix_fields (this=0x7f3b1a4d3310, thd=0x7f3b291b52f0, func=0x7f3b1a4d3248, arg_count=1, arguments=0x7f3b1a4d32e0) at /src/10.0/sql/item_func.cc:3610 #10 0x00000000008d3808 in Item_udf_func::fix_fields (this=0x7f3b1a4d3248, thd=0x7f3b291b52f0, ref=0x7f3b1a4d33b8) at /src/10.0/sql/item_func.h:1389 #11 0x00000000006169b0 in setup_fields (thd=0x7f3b291b52f0, ref_pointer_array=0x7f3b1a4d39d0, fields=..., mark_used_columns=MARK_COLUMNS_READ, sum_func_list=0x7f3b1a4d37d0, allow_sum_func=true) at /src/10.0/sql/sql_base.cc:7918 #12 0x00000000006a0b1d in JOIN::prepare (this=0x7f3b1a4d3488, rref_pointer_array=0x7f3b291b95c8, tables_init=0x0, wild_num=0, conds_init=0x0, og_num=0, order_init=0x0, skip_order_by=false, group_init=0x0, having_init=0x0, proc_param_init=0x0, select_lex_arg=0x7f3b291b9350, unit_arg=0x7f3b291b8c60) at /src/10.0/sql/sql_select.cc:785 #13 0x00000000006a998b in mysql_select (thd=0x7f3b291b52f0, rref_pointer_array=0x7f3b291b95c8, tables=0x0, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0x7f3b1a4d3468, unit=0x7f3b291b8c60, select_lex=0x7f3b291b9350) at /src/10.0/sql/sql_select.cc:3286 #14 0x000000000069fdb9 in handle_select (thd=0x7f3b291b52f0, lex=0x7f3b291b8b98, result=0x7f3b1a4d3468, setup_tables_done_option=0) at /src/10.0/sql/sql_select.cc:373 #15 0x000000000067466c in execute_sqlcom_select (thd=0x7f3b291b52f0, all_tables=0x0) at /src/10.0/sql/sql_parse.cc:5304 #16 0x000000000066ca55 in mysql_execute_command (thd=0x7f3b291b52f0) at /src/10.0/sql/sql_parse.cc:2562 #17 0x00000000006771f3 in mysql_parse (thd=0x7f3b291b52f0, rawbuf=0x7f3b1a4d3088 "select jsonvalue(3.14)", length=22, parser_state=0x7f3b436596a0) at /src/10.0/sql/sql_parse.cc:6565 #18 0x0000000000669cbf in dispatch_command (command=COM_QUERY, thd=0x7f3b291b52f0, packet=0x7f3b1a4b3e71 "select jsonvalue(3.14)", packet_length=22) at /src/10.0/sql/sql_parse.cc:1308 #19 0x0000000000668f8c in do_command (thd=0x7f3b291b52f0) at /src/10.0/sql/sql_parse.cc:998 #20 0x00000000007872c5 in do_handle_one_connection (thd_arg=0x7f3b291b52f0) at /src/10.0/sql/sql_connect.cc:1377 #21 0x0000000000787037 in handle_one_connection (arg=0x7f3b291b52f0) at /src/10.0/sql/sql_connect.cc:1292 #22 0x00007f3b432e90a4 in start_thread () from /lib64/libpthread.so.0 #23 0x00007f3b4167d04d in clone () from /lib64/libc.so.6

          Apparently the crash occurs in the function GetTraceValue() that is defined as:

          extern "C" int GetTraceValue(void) {return THDVAR(current_thd, xtrace);}

          It is used to get the value of the connect_xtrace variable. This function is used in ALL connect queries and until now has never failed.
          The only explanation I see is if a json udf is created and used without the CONNECT storage engine being installed. Unfortunately I cannot test this because I did not find a way to de-install a plugin engine and starting mysqld with --skip-connect does not work.
          Can you confirm this is the case so I can try to find a fix.
          In that case the turnaround is obvious: install CONNECT.

          bertrandop Olivier Bertrand added a comment - Apparently the crash occurs in the function GetTraceValue() that is defined as: extern "C" int GetTraceValue(void) {return THDVAR(current_thd, xtrace);} It is used to get the value of the connect_xtrace variable. This function is used in ALL connect queries and until now has never failed. The only explanation I see is if a json udf is created and used without the CONNECT storage engine being installed. Unfortunately I cannot test this because I did not find a way to de-install a plugin engine and starting mysqld with --skip-connect does not work. Can you confirm this is the case so I can try to find a fix. In that case the turnaround is obvious: install CONNECT.
          bertrandop Olivier Bertrand added a comment - - edited

          Downloading a new MariaDB from scratch, I could make it crash by creating and using a Json UDF while CONNECT was not installed.
          And verify that installing CONNECT fixes the problem.

          bertrandop Olivier Bertrand added a comment - - edited Downloading a new MariaDB from scratch, I could make it crash by creating and using a Json UDF while CONNECT was not installed. And verify that installing CONNECT fixes the problem.

          To make Json UDFs usable without CONNECT being installed, the way Json group size is handled was modified.
          The session variable connect_json_grp_size is now deprecated and will not be kept in future version.
          The new value called JsonGrpSize used by the Json aggregate functions is now set and can be queried using two new functions jsonset_grp_size and jsonget_grp_size.
          For older applications, the use of the session variable connect_json_grp_size can be temporarily continued by setting JsonGrpSize to 0 but only when CONNECT is installed.

          bertrandop Olivier Bertrand added a comment - To make Json UDFs usable without CONNECT being installed, the way Json group size is handled was modified. The session variable connect_json_grp_size is now deprecated and will not be kept in future version. The new value called JsonGrpSize used by the Json aggregate functions is now set and can be queried using two new functions jsonset_grp_size and jsonget_grp_size . For older applications, the use of the session variable connect_json_grp_size can be temporarily continued by setting JsonGrpSize to 0 but only when CONNECT is installed.

          People

            bertrandop Olivier Bertrand
            Norbert van Nobelen Norbert van Nobelen
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.