Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13650

Backport fix for MDEV-13060 (crash when both AWS plugin and server_audit are loaded) to 10.1

    XMLWordPrintable

Details

    Description

      All kinds of bad things happen if aws_key_management and server_audit plugins are loaded upon server startup.

      Example #1 (reported by community as MDEV-13624):

      • start server with --encrypt-tmp-files --plugin-load-add=server_audit --plugin-load-add=aws_key_management <valid aws options>
      • load schema from MDEV-13624
      • run percona-toolkit-3.0.2/bin/pt-table-checksum -uroot --no-check-plan --quiet --port=3306 --host=127.0.0.1 --databases test

        10.1 non-debug

        		 
        Program terminated with signal SIGSEGV, Segmentation fault.
        		 
        #0  0x0000559653fde946 in ha_innobase::innobase_get_index (this=0x7f3a844cc820, keynr=0) at /data/src/bb-10.1-wlad/storage/xtradb/handler/ha_innodb.cc:9994
        		 
        9994                            if (!key || ut_strcmp(index->name, key->name) != 0) {
        		 
         
        		 
        #0  0x0000559653fde946 in ha_innobase::innobase_get_index (this=0x7f3a844cc820, keynr=0) at /data/src/bb-10.1-wlad/storage/xtradb/handler/ha_innodb.cc:9994
        		 
        #1  0x0000559653fe84b1 in ha_innobase::change_active_index (this=this@entry=0x7f3a844cc820, keynr=0) at /data/src/bb-10.1-wlad/storage/xtradb/handler/ha_innodb.cc:10067
        		 
        #2  0x0000559653fe877b in ha_innobase::rnd_init (this=0x7f3a844cc820, scan=<optimized out>) at /data/src/bb-10.1-wlad/storage/xtradb/handler/ha_innodb.cc:10358
        		 
        #3  0x0000559653dbd036 in ha_rnd_init (scan=scan@entry=true, this=0x7f3a844cc820) at /data/src/bb-10.1-wlad/sql/handler.h:2796
        		 
        #4  handler::ha_rnd_init_with_error (this=0x7f3a844cc820, scan=scan@entry=true) at /data/src/bb-10.1-wlad/sql/handler.cc:2754
        		 
        #5  0x0000559653e9db1e in init_read_record (info=0x7f3a844278f8, thd=0x7f3ab47bd008, table=0x7f3a84495c08, select=0x7f3a84427cf8, use_record_cache=<optimized out>, print_error=<optimized out>, disable_rr_cache=<optimized out>) at /data/src/bb-10.1-wlad/sql/records.cc:282
        		 
        #6  0x0000559653c76e5f in join_init_read_record (tab=0x7f3a84427838) at /data/src/bb-10.1-wlad/sql/sql_select.cc:19305
        		 
        #7  0x0000559653c77036 in sub_select (join=0x7f3a84425d48, join_tab=0x7f3a84427838, end_of_records=<optimized out>) at /data/src/bb-10.1-wlad/sql/sql_select.cc:18395
        		 
        #8  0x0000559653c85c65 in do_select (join=0x7f3a84425d48, fields=0x7f3a84426148, table=0x0, procedure=0x0) at /data/src/bb-10.1-wlad/sql/sql_select.cc:18053
        		 
        #9  0x0000559653c97a9d in JOIN::exec_inner (this=0x7f3a84425d48) at /data/src/bb-10.1-wlad/sql/sql_select.cc:3225
        		 
        #10 0x0000559653c99644 in JOIN::exec (this=this@entry=0x7f3a84425d48) at /data/src/bb-10.1-wlad/sql/sql_select.cc:2512
        		 
        #11 0x0000559653c96117 in mysql_select (thd=thd@entry=0x7f3ab47bd008, rref_pointer_array=rref_pointer_array@entry=0x7f3ab47c12d8, tables=0x7f3a84425620, wild_num=<optimized out>, fields=..., conds=<optimized out>, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489958656, result=0x7f3a84425ca8, unit=0x7f3ab47c0948, select_lex=0x7f3ab47c1048) at /data/src/bb-10.1-wlad/sql/sql_select.cc:3449
        		 
        #12 0x0000559653c96b50 in handle_select (thd=0x7f3ab47bd008, lex=0x7f3ab47c0880, result=0x7f3a84425ca8, setup_tables_done_option=1073741824) at /data/src/bb-10.1-wlad/sql/sql_select.cc:384
        		 
        #13 0x0000559653c4c0c7 in mysql_execute_command (thd=0x7f3ab47bd008) at /data/src/bb-10.1-wlad/sql/sql_parse.cc:4032
        		 
        #14 0x0000559653c4d0ba in mysql_parse (thd=0x7f3ab47bd008, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /data/src/bb-10.1-wlad/sql/sql_parse.cc:7344
        		 
        #15 0x0000559653c4fd00 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7f3ab47bd008, packet=packet@entry=0x7f3a8d6c4009 "REPLACE INTO `percona`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT 'test', 'patient_medication', '1', NULL, NULL, NULL, COUNT(*) AS cnt, COALES"..., packet_length=packet_length@entry=465) at /data/src/bb-10.1-wlad/sql/sql_parse.cc:1484
        		 
        #16 0x0000559653c503f3 in do_command (thd=0x7f3ab47bd008) at /data/src/bb-10.1-wlad/sql/sql_parse.cc:1106
        		 
        #17 0x0000559653d0d19c in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f3ab47bd008) at /data/src/bb-10.1-wlad/sql/sql_connect.cc:1349
        		 
        #18 0x0000559653d0d357 in handle_one_connection (arg=0x7f3ab47bd008) at /data/src/bb-10.1-wlad/sql/sql_connect.cc:1261
        		 
        #19 0x00007f3ab6ba4494 in start_thread (arg=0x7f3ab6f00b00) at pthread_create.c:333
        		 
        #20 0x00007f3ab4cef93f in clone () from /lib/x86_64-linux-gnu/libc.so.6

      Example #2

      • start server with --encrypt-tmp-files --plugin-load-add=server_audit --plugin-load-add=aws_key_management <valid aws options>

        10.1 debug

        		 
        Error: Freeing overrun buffer  0x557be85646aa, 0x557be8551f6e, 0x557be7cdaba7, 0x557be7cdad0b, 0x557be7c7c55b, 0x557be7c7c9d6, 0x557be7dea3c1, 0x557be7bf59a8
        		 
        Allocated at 0x557be8552065, 0x557be7cda538, 0x557be7cda6b9, 0x557be7cda808, 0x7fa182df3fad, 0x7fa182df71b5, 0x557be7deec06, 0x557be7deecb6
        		 
        2017-08-26 17:50:41 140331719626880 [Note] Server socket created on IP: '::'.
        		 
        Error: Freeing overrun buffer  0x557be85646aa, 0x557be8551f6e, 0x557be7cdaba7, 0x557be7cdad0b, 0x557be7c7c55b, 0x557be7c7c9d6, 0x557be7c2aa53, 0x557be7bf62ef
        		 
        Allocated at 0x557be8552065, 0x557be7cda538, 0x557be7cda6b9, 0x557be7cda808, 0x7fa182df3fad, 0x7fa182df71b5, 0x557be7deec06, 0x557be7deecb6
        		 
        Error: Freeing overrun buffer  0x557be85646aa, 0x557be8551f6e, 0x557be7cdaba7, 0x557be7cdad0b, 0x557be7c7c55b, 0x557be7c7c9d6, 0x557be7dc1d8a, 0x557be7bf6317
        		 
        Allocated at 0x557be8552065, 0x557be7cda538, 0x557be7cda6b9, mysys/safemalloc.c:191, sql/sql_plugin.cc:3156, sql/sql_audit.cc:510, 0x557be7deec06, sql/sql_plugin.cc:3072
        		 
        Error: Freeing overrun buffer  mysys/safemalloc.c:191, mysys/my_malloc.c:219, sql/sql_plugin.cc:3156, sql/sql_plugin.cc:3183, sql/sql_class.cc:1663, sql/sql_class.cc:1710, sql/sql_acl.cc:6586, sql/mysqld.cc:5800
        		 
        Allocated at mysys/my_malloc.c:240, sql/sql_plugin.cc:2986, sql/sql_plugin.cc:3025, sql/sql_plugin.cc:3072, 0x7fa182df3fad, 0x7fa182df71b5, sql/sql_audit.cc:510, sql/sql_audit.cc:543
        		 
        Error: Freeing overrun buffer  mysys/safemalloc.c:191, mysys/my_malloc.c:219, sql/sql_plugin.cc:3156, sql/sql_plugin.cc:3183, sql/sql_class.cc:1663, sql/sql_class.cc:1710, sql/sql_udf.cc:262, sql/mysqld.cc:5802
        		 
        Allocated at mysys/my_malloc.c:240, sql/sql_plugin.cc:2986, sql/sql_plugin.cc:3025, sql/sql_plugin.cc:3072, 0x7fa182df3fad, 0x7fa182df71b5, sql/sql_audit.cc:510, sql/sql_audit.cc:543
        		 
        Error: Freeing overrun buffer  mysys/safemalloc.c:191, mysys/my_malloc.c:219, sql/sql_plugin.cc:3156, sql/sql_plugin.cc:3183, sql/sql_class.cc:1663, sql/sql_class.cc:1710, sql/events.cc:944, sql/mysqld.cc:5823
        		 
        Allocated at mysys/my_malloc.c:240, sql/sql_plugin.cc:2986, sql/sql_plugin.cc:3025, sql/sql_plugin.cc:3072, 0x7fa182df3fad, 0x7fa182df71b5, sql/sql_audit.cc:510, sql/sql_audit.cc:543
        		 
        2017-08-26 17:50:42 140331719626880 [Note] /data/bld/bb-10.1-wlad-debug/bin/mysqld: ready for connections.

      • shutdown the server

        10.1 debug

        		 
        Program terminated with signal SIGSEGV, Segmentation fault.
        		 
        #0  0x0000557be7cd5870 in intern_plugin_unlock (lex=0x0, plugin=0x7fa1838b62f0) at /data/src/bb-10.1-wlad/sql/sql_plugin.cc:1331
        		 
        1331      DBUG_ASSERT(pi->ref_count);
        		 
         
        		 
        #0  0x0000557be7cd5870 in intern_plugin_unlock (lex=0x0, plugin=0x7fa1838b62f0) at /data/src/bb-10.1-wlad/sql/sql_plugin.cc:1331
        		 
        #1  0x0000557be7cd59a8 in plugin_unlock (thd=0x0, plugin=0x7fa1838b62f0) at /data/src/bb-10.1-wlad/sql/sql_plugin.cc:1356
        		 
        #2  0x0000557be7da8881 in TABLE_SHARE::destroy (this=0x7fa183b11488) at /data/src/bb-10.1-wlad/sql/table.cc:428
        		 
        #3  0x0000557be7da8a10 in free_table_share (share=0x7fa183b11488) at /data/src/bb-10.1-wlad/sql/table.cc:469
        		 
        #4  0x0000557be7e80d93 in tdc_delete_share_from_hash (element=0x7fa183b2bf10) at /data/src/bb-10.1-wlad/sql/table_cache.cc:411
        		 
        #5  0x0000557be7e81d48 in tdc_release_share (share=0x7fa183b11488) at /data/src/bb-10.1-wlad/sql/table_cache.cc:790
        		 
        #6  0x0000557be7db077c in closefrm (table=0x7fa183b9d670, free_share=true) at /data/src/bb-10.1-wlad/sql/table.cc:3081
        		 
        #7  0x0000557be7c5241b in intern_close_table (table=0x7fa183b9d670) at /data/src/bb-10.1-wlad/sql/sql_base.cc:354
        		 
        #8  0x0000557be7e80589 in tc_purge (mark_flushed=true) at /data/src/bb-10.1-wlad/sql/table_cache.cc:204
        		 
        #9  0x0000557be7c5283e in close_cached_tables (thd=0x0, tables=0x0, wait_for_refresh=false, timeout=31536000) at /data/src/bb-10.1-wlad/sql/sql_base.cc:485
        		 
        #10 0x0000557be7e80f00 in tdc_start_shutdown () at /data/src/bb-10.1-wlad/sql/table_cache.cc:460
        		 
        #11 0x0000557be7bef1b2 in clean_up (print_message=true) at /data/src/bb-10.1-wlad/sql/mysqld.cc:2120
        		 
        #12 0x0000557be7beedd8 in unireg_end () at /data/src/bb-10.1-wlad/sql/mysqld.cc:2002
        		 
        #13 0x0000557be7beecef in kill_server (sig_ptr=0x0) at /data/src/bb-10.1-wlad/sql/mysqld.cc:1930
        		 
        #14 0x0000557be7beed17 in kill_server_thread (arg=0x7fa1864a0220) at /data/src/bb-10.1-wlad/sql/mysqld.cc:1953
        		 
        #15 0x00007fa1860b0494 in start_thread (arg=0x7fa1863c2b00) at pthread_create.c:333
        		 
        #16 0x00007fa1841fb93f in clone () from /lib/x86_64-linux-gnu/libc.so.6

      Some other observed consequences are in MDEV-13651 and MDEV-13652, both of which I'm closing as duplicates.

      Could not reproduce any of it on 10.2. Apparently, the root cause was fixed within the scope of MDEV-13060, but only in 10.2. Since we provide both plugins in 10.1, at least in sources, it should work there too.

      Attachments

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. MDEV-13624 pt-table-checksum crashes server

          • Major - Major loss of function.
          • Closed
          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MDEV-13651 [Draft] Server crashes upon startup with AWS, server_audit and encrypt-tmp-files

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-13652 [Draft] Server sigkills itself or OOMs with AWS

          • Major - Major loss of function.
          • Closed

          Activity

            People

              serg Sergei Golubchik
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.