Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13650

Backport fix for MDEV-13060 (crash when both AWS plugin and server_audit are loaded) to 10.1

    XMLWordPrintable

Details

    Description

      All kinds of bad things happen if aws_key_management and server_audit plugins are loaded upon server startup.

      Example #1 (reported by community as MDEV-13624):

      • start server with --encrypt-tmp-files --plugin-load-add=server_audit --plugin-load-add=aws_key_management <valid aws options>
      • load schema from MDEV-13624
      • run percona-toolkit-3.0.2/bin/pt-table-checksum -uroot --no-check-plan --quiet --port=3306 --host=127.0.0.1 --databases test

        10.1 non-debug

        Program terminated with signal SIGSEGV, Segmentation fault.
        #0  0x0000559653fde946 in ha_innobase::innobase_get_index (this=0x7f3a844cc820, keynr=0) at /data/src/bb-10.1-wlad/storage/xtradb/handler/ha_innodb.cc:9994
        9994                            if (!key || ut_strcmp(index->name, key->name) != 0) {
         
        #0  0x0000559653fde946 in ha_innobase::innobase_get_index (this=0x7f3a844cc820, keynr=0) at /data/src/bb-10.1-wlad/storage/xtradb/handler/ha_innodb.cc:9994
        #1  0x0000559653fe84b1 in ha_innobase::change_active_index (this=this@entry=0x7f3a844cc820, keynr=0) at /data/src/bb-10.1-wlad/storage/xtradb/handler/ha_innodb.cc:10067
        #2  0x0000559653fe877b in ha_innobase::rnd_init (this=0x7f3a844cc820, scan=<optimized out>) at /data/src/bb-10.1-wlad/storage/xtradb/handler/ha_innodb.cc:10358
        #3  0x0000559653dbd036 in ha_rnd_init (scan=scan@entry=true, this=0x7f3a844cc820) at /data/src/bb-10.1-wlad/sql/handler.h:2796
        #4  handler::ha_rnd_init_with_error (this=0x7f3a844cc820, scan=scan@entry=true) at /data/src/bb-10.1-wlad/sql/handler.cc:2754
        #5  0x0000559653e9db1e in init_read_record (info=0x7f3a844278f8, thd=0x7f3ab47bd008, table=0x7f3a84495c08, select=0x7f3a84427cf8, use_record_cache=<optimized out>, print_error=<optimized out>, disable_rr_cache=<optimized out>) at /data/src/bb-10.1-wlad/sql/records.cc:282
        #6  0x0000559653c76e5f in join_init_read_record (tab=0x7f3a84427838) at /data/src/bb-10.1-wlad/sql/sql_select.cc:19305
        #7  0x0000559653c77036 in sub_select (join=0x7f3a84425d48, join_tab=0x7f3a84427838, end_of_records=<optimized out>) at /data/src/bb-10.1-wlad/sql/sql_select.cc:18395
        #8  0x0000559653c85c65 in do_select (join=0x7f3a84425d48, fields=0x7f3a84426148, table=0x0, procedure=0x0) at /data/src/bb-10.1-wlad/sql/sql_select.cc:18053
        #9  0x0000559653c97a9d in JOIN::exec_inner (this=0x7f3a84425d48) at /data/src/bb-10.1-wlad/sql/sql_select.cc:3225
        #10 0x0000559653c99644 in JOIN::exec (this=this@entry=0x7f3a84425d48) at /data/src/bb-10.1-wlad/sql/sql_select.cc:2512
        #11 0x0000559653c96117 in mysql_select (thd=thd@entry=0x7f3ab47bd008, rref_pointer_array=rref_pointer_array@entry=0x7f3ab47c12d8, tables=0x7f3a84425620, wild_num=<optimized out>, fields=..., conds=<optimized out>, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489958656, result=0x7f3a84425ca8, unit=0x7f3ab47c0948, select_lex=0x7f3ab47c1048) at /data/src/bb-10.1-wlad/sql/sql_select.cc:3449
        #12 0x0000559653c96b50 in handle_select (thd=0x7f3ab47bd008, lex=0x7f3ab47c0880, result=0x7f3a84425ca8, setup_tables_done_option=1073741824) at /data/src/bb-10.1-wlad/sql/sql_select.cc:384
        #13 0x0000559653c4c0c7 in mysql_execute_command (thd=0x7f3ab47bd008) at /data/src/bb-10.1-wlad/sql/sql_parse.cc:4032
        #14 0x0000559653c4d0ba in mysql_parse (thd=0x7f3ab47bd008, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /data/src/bb-10.1-wlad/sql/sql_parse.cc:7344
        #15 0x0000559653c4fd00 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7f3ab47bd008, packet=packet@entry=0x7f3a8d6c4009 "REPLACE INTO `percona`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT 'test', 'patient_medication', '1', NULL, NULL, NULL, COUNT(*) AS cnt, COALES"..., packet_length=packet_length@entry=465) at /data/src/bb-10.1-wlad/sql/sql_parse.cc:1484
        #16 0x0000559653c503f3 in do_command (thd=0x7f3ab47bd008) at /data/src/bb-10.1-wlad/sql/sql_parse.cc:1106
        #17 0x0000559653d0d19c in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f3ab47bd008) at /data/src/bb-10.1-wlad/sql/sql_connect.cc:1349
        #18 0x0000559653d0d357 in handle_one_connection (arg=0x7f3ab47bd008) at /data/src/bb-10.1-wlad/sql/sql_connect.cc:1261
        #19 0x00007f3ab6ba4494 in start_thread (arg=0x7f3ab6f00b00) at pthread_create.c:333
        #20 0x00007f3ab4cef93f in clone () from /lib/x86_64-linux-gnu/libc.so.6
        

      Example #2

      • start server with --encrypt-tmp-files --plugin-load-add=server_audit --plugin-load-add=aws_key_management <valid aws options>

        10.1 debug

        Error: Freeing overrun buffer  0x557be85646aa, 0x557be8551f6e, 0x557be7cdaba7, 0x557be7cdad0b, 0x557be7c7c55b, 0x557be7c7c9d6, 0x557be7dea3c1, 0x557be7bf59a8
        Allocated at 0x557be8552065, 0x557be7cda538, 0x557be7cda6b9, 0x557be7cda808, 0x7fa182df3fad, 0x7fa182df71b5, 0x557be7deec06, 0x557be7deecb6
        2017-08-26 17:50:41 140331719626880 [Note] Server socket created on IP: '::'.
        Error: Freeing overrun buffer  0x557be85646aa, 0x557be8551f6e, 0x557be7cdaba7, 0x557be7cdad0b, 0x557be7c7c55b, 0x557be7c7c9d6, 0x557be7c2aa53, 0x557be7bf62ef
        Allocated at 0x557be8552065, 0x557be7cda538, 0x557be7cda6b9, 0x557be7cda808, 0x7fa182df3fad, 0x7fa182df71b5, 0x557be7deec06, 0x557be7deecb6
        Error: Freeing overrun buffer  0x557be85646aa, 0x557be8551f6e, 0x557be7cdaba7, 0x557be7cdad0b, 0x557be7c7c55b, 0x557be7c7c9d6, 0x557be7dc1d8a, 0x557be7bf6317
        Allocated at 0x557be8552065, 0x557be7cda538, 0x557be7cda6b9, mysys/safemalloc.c:191, sql/sql_plugin.cc:3156, sql/sql_audit.cc:510, 0x557be7deec06, sql/sql_plugin.cc:3072
        Error: Freeing overrun buffer  mysys/safemalloc.c:191, mysys/my_malloc.c:219, sql/sql_plugin.cc:3156, sql/sql_plugin.cc:3183, sql/sql_class.cc:1663, sql/sql_class.cc:1710, sql/sql_acl.cc:6586, sql/mysqld.cc:5800
        Allocated at mysys/my_malloc.c:240, sql/sql_plugin.cc:2986, sql/sql_plugin.cc:3025, sql/sql_plugin.cc:3072, 0x7fa182df3fad, 0x7fa182df71b5, sql/sql_audit.cc:510, sql/sql_audit.cc:543
        Error: Freeing overrun buffer  mysys/safemalloc.c:191, mysys/my_malloc.c:219, sql/sql_plugin.cc:3156, sql/sql_plugin.cc:3183, sql/sql_class.cc:1663, sql/sql_class.cc:1710, sql/sql_udf.cc:262, sql/mysqld.cc:5802
        Allocated at mysys/my_malloc.c:240, sql/sql_plugin.cc:2986, sql/sql_plugin.cc:3025, sql/sql_plugin.cc:3072, 0x7fa182df3fad, 0x7fa182df71b5, sql/sql_audit.cc:510, sql/sql_audit.cc:543
        Error: Freeing overrun buffer  mysys/safemalloc.c:191, mysys/my_malloc.c:219, sql/sql_plugin.cc:3156, sql/sql_plugin.cc:3183, sql/sql_class.cc:1663, sql/sql_class.cc:1710, sql/events.cc:944, sql/mysqld.cc:5823
        Allocated at mysys/my_malloc.c:240, sql/sql_plugin.cc:2986, sql/sql_plugin.cc:3025, sql/sql_plugin.cc:3072, 0x7fa182df3fad, 0x7fa182df71b5, sql/sql_audit.cc:510, sql/sql_audit.cc:543
        2017-08-26 17:50:42 140331719626880 [Note] /data/bld/bb-10.1-wlad-debug/bin/mysqld: ready for connections.
        

      • shutdown the server

        10.1 debug

        Program terminated with signal SIGSEGV, Segmentation fault.
        #0  0x0000557be7cd5870 in intern_plugin_unlock (lex=0x0, plugin=0x7fa1838b62f0) at /data/src/bb-10.1-wlad/sql/sql_plugin.cc:1331
        1331      DBUG_ASSERT(pi->ref_count);
         
        #0  0x0000557be7cd5870 in intern_plugin_unlock (lex=0x0, plugin=0x7fa1838b62f0) at /data/src/bb-10.1-wlad/sql/sql_plugin.cc:1331
        #1  0x0000557be7cd59a8 in plugin_unlock (thd=0x0, plugin=0x7fa1838b62f0) at /data/src/bb-10.1-wlad/sql/sql_plugin.cc:1356
        #2  0x0000557be7da8881 in TABLE_SHARE::destroy (this=0x7fa183b11488) at /data/src/bb-10.1-wlad/sql/table.cc:428
        #3  0x0000557be7da8a10 in free_table_share (share=0x7fa183b11488) at /data/src/bb-10.1-wlad/sql/table.cc:469
        #4  0x0000557be7e80d93 in tdc_delete_share_from_hash (element=0x7fa183b2bf10) at /data/src/bb-10.1-wlad/sql/table_cache.cc:411
        #5  0x0000557be7e81d48 in tdc_release_share (share=0x7fa183b11488) at /data/src/bb-10.1-wlad/sql/table_cache.cc:790
        #6  0x0000557be7db077c in closefrm (table=0x7fa183b9d670, free_share=true) at /data/src/bb-10.1-wlad/sql/table.cc:3081
        #7  0x0000557be7c5241b in intern_close_table (table=0x7fa183b9d670) at /data/src/bb-10.1-wlad/sql/sql_base.cc:354
        #8  0x0000557be7e80589 in tc_purge (mark_flushed=true) at /data/src/bb-10.1-wlad/sql/table_cache.cc:204
        #9  0x0000557be7c5283e in close_cached_tables (thd=0x0, tables=0x0, wait_for_refresh=false, timeout=31536000) at /data/src/bb-10.1-wlad/sql/sql_base.cc:485
        #10 0x0000557be7e80f00 in tdc_start_shutdown () at /data/src/bb-10.1-wlad/sql/table_cache.cc:460
        #11 0x0000557be7bef1b2 in clean_up (print_message=true) at /data/src/bb-10.1-wlad/sql/mysqld.cc:2120
        #12 0x0000557be7beedd8 in unireg_end () at /data/src/bb-10.1-wlad/sql/mysqld.cc:2002
        #13 0x0000557be7beecef in kill_server (sig_ptr=0x0) at /data/src/bb-10.1-wlad/sql/mysqld.cc:1930
        #14 0x0000557be7beed17 in kill_server_thread (arg=0x7fa1864a0220) at /data/src/bb-10.1-wlad/sql/mysqld.cc:1953
        #15 0x00007fa1860b0494 in start_thread (arg=0x7fa1863c2b00) at pthread_create.c:333
        #16 0x00007fa1841fb93f in clone () from /lib/x86_64-linux-gnu/libc.so.6
        

      Some other observed consequences are in MDEV-13651 and MDEV-13652, both of which I'm closing as duplicates.

      Could not reproduce any of it on 10.2. Apparently, the root cause was fixed within the scope of MDEV-13060, but only in 10.2. Since we provide both plugins in 10.1, at least in sources, it should work there too.

      Attachments

        Issue Links

          Activity

            People

              serg Sergei Golubchik
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.