[MDEV-13650] Backport fix for MDEV-13060 (crash when both AWS plugin and server_audit are loaded) to 10.1 Created: 2017-08-25  Updated: 2017-09-18  Resolved: 2017-09-18

Status: Closed
Project: MariaDB Server
Component/s: Plugin - Audit, Plugin - AWS key management, Plugins
Affects Version/s: 10.1
Fix Version/s: 10.1.27

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Blocks
blocks MDEV-13624 pt-table-checksum crashes server Closed
Duplicate
duplicates MDEV-13651 [Draft] Server crashes upon startup w... Closed
duplicates MDEV-13652 [Draft] Server sigkills itself or OOM... Closed

 Description   

All kinds of bad things happen if aws_key_management and server_audit plugins are loaded upon server startup.

Example #1 (reported by community as MDEV-13624):

  • start server with --encrypt-tmp-files --plugin-load-add=server_audit --plugin-load-add=aws_key_management <valid aws options>
  • load schema from MDEV-13624
  • run percona-toolkit-3.0.2/bin/pt-table-checksum -uroot --no-check-plan --quiet --port=3306 --host=127.0.0.1 --databases test

    10.1 non-debug

    		 
    Program terminated with signal SIGSEGV, Segmentation fault.
    		 
    #0  0x0000559653fde946 in ha_innobase::innobase_get_index (this=0x7f3a844cc820, keynr=0) at /data/src/bb-10.1-wlad/storage/xtradb/handler/ha_innodb.cc:9994
    		 
    9994                            if (!key || ut_strcmp(index->name, key->name) != 0) {
    		 
     
    		 
    #0  0x0000559653fde946 in ha_innobase::innobase_get_index (this=0x7f3a844cc820, keynr=0) at /data/src/bb-10.1-wlad/storage/xtradb/handler/ha_innodb.cc:9994
    		 
    #1  0x0000559653fe84b1 in ha_innobase::change_active_index (this=this@entry=0x7f3a844cc820, keynr=0) at /data/src/bb-10.1-wlad/storage/xtradb/handler/ha_innodb.cc:10067
    		 
    #2  0x0000559653fe877b in ha_innobase::rnd_init (this=0x7f3a844cc820, scan=<optimized out>) at /data/src/bb-10.1-wlad/storage/xtradb/handler/ha_innodb.cc:10358
    		 
    #3  0x0000559653dbd036 in ha_rnd_init (scan=scan@entry=true, this=0x7f3a844cc820) at /data/src/bb-10.1-wlad/sql/handler.h:2796
    		 
    #4  handler::ha_rnd_init_with_error (this=0x7f3a844cc820, scan=scan@entry=true) at /data/src/bb-10.1-wlad/sql/handler.cc:2754
    		 
    #5  0x0000559653e9db1e in init_read_record (info=0x7f3a844278f8, thd=0x7f3ab47bd008, table=0x7f3a84495c08, select=0x7f3a84427cf8, use_record_cache=<optimized out>, print_error=<optimized out>, disable_rr_cache=<optimized out>) at /data/src/bb-10.1-wlad/sql/records.cc:282
    		 
    #6  0x0000559653c76e5f in join_init_read_record (tab=0x7f3a84427838) at /data/src/bb-10.1-wlad/sql/sql_select.cc:19305
    		 
    #7  0x0000559653c77036 in sub_select (join=0x7f3a84425d48, join_tab=0x7f3a84427838, end_of_records=<optimized out>) at /data/src/bb-10.1-wlad/sql/sql_select.cc:18395
    		 
    #8  0x0000559653c85c65 in do_select (join=0x7f3a84425d48, fields=0x7f3a84426148, table=0x0, procedure=0x0) at /data/src/bb-10.1-wlad/sql/sql_select.cc:18053
    		 
    #9  0x0000559653c97a9d in JOIN::exec_inner (this=0x7f3a84425d48) at /data/src/bb-10.1-wlad/sql/sql_select.cc:3225
    		 
    #10 0x0000559653c99644 in JOIN::exec (this=this@entry=0x7f3a84425d48) at /data/src/bb-10.1-wlad/sql/sql_select.cc:2512
    		 
    #11 0x0000559653c96117 in mysql_select (thd=thd@entry=0x7f3ab47bd008, rref_pointer_array=rref_pointer_array@entry=0x7f3ab47c12d8, tables=0x7f3a84425620, wild_num=<optimized out>, fields=..., conds=<optimized out>, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489958656, result=0x7f3a84425ca8, unit=0x7f3ab47c0948, select_lex=0x7f3ab47c1048) at /data/src/bb-10.1-wlad/sql/sql_select.cc:3449
    		 
    #12 0x0000559653c96b50 in handle_select (thd=0x7f3ab47bd008, lex=0x7f3ab47c0880, result=0x7f3a84425ca8, setup_tables_done_option=1073741824) at /data/src/bb-10.1-wlad/sql/sql_select.cc:384
    		 
    #13 0x0000559653c4c0c7 in mysql_execute_command (thd=0x7f3ab47bd008) at /data/src/bb-10.1-wlad/sql/sql_parse.cc:4032
    		 
    #14 0x0000559653c4d0ba in mysql_parse (thd=0x7f3ab47bd008, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /data/src/bb-10.1-wlad/sql/sql_parse.cc:7344
    		 
    #15 0x0000559653c4fd00 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7f3ab47bd008, packet=packet@entry=0x7f3a8d6c4009 "REPLACE INTO `percona`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT 'test', 'patient_medication', '1', NULL, NULL, NULL, COUNT(*) AS cnt, COALES"..., packet_length=packet_length@entry=465) at /data/src/bb-10.1-wlad/sql/sql_parse.cc:1484
    		 
    #16 0x0000559653c503f3 in do_command (thd=0x7f3ab47bd008) at /data/src/bb-10.1-wlad/sql/sql_parse.cc:1106
    		 
    #17 0x0000559653d0d19c in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f3ab47bd008) at /data/src/bb-10.1-wlad/sql/sql_connect.cc:1349
    		 
    #18 0x0000559653d0d357 in handle_one_connection (arg=0x7f3ab47bd008) at /data/src/bb-10.1-wlad/sql/sql_connect.cc:1261
    		 
    #19 0x00007f3ab6ba4494 in start_thread (arg=0x7f3ab6f00b00) at pthread_create.c:333
    		 
    #20 0x00007f3ab4cef93f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Example #2

  • start server with --encrypt-tmp-files --plugin-load-add=server_audit --plugin-load-add=aws_key_management <valid aws options>

    10.1 debug

    		 
    Error: Freeing overrun buffer  0x557be85646aa, 0x557be8551f6e, 0x557be7cdaba7, 0x557be7cdad0b, 0x557be7c7c55b, 0x557be7c7c9d6, 0x557be7dea3c1, 0x557be7bf59a8
    		 
    Allocated at 0x557be8552065, 0x557be7cda538, 0x557be7cda6b9, 0x557be7cda808, 0x7fa182df3fad, 0x7fa182df71b5, 0x557be7deec06, 0x557be7deecb6
    		 
    2017-08-26 17:50:41 140331719626880 [Note] Server socket created on IP: '::'.
    		 
    Error: Freeing overrun buffer  0x557be85646aa, 0x557be8551f6e, 0x557be7cdaba7, 0x557be7cdad0b, 0x557be7c7c55b, 0x557be7c7c9d6, 0x557be7c2aa53, 0x557be7bf62ef
    		 
    Allocated at 0x557be8552065, 0x557be7cda538, 0x557be7cda6b9, 0x557be7cda808, 0x7fa182df3fad, 0x7fa182df71b5, 0x557be7deec06, 0x557be7deecb6
    		 
    Error: Freeing overrun buffer  0x557be85646aa, 0x557be8551f6e, 0x557be7cdaba7, 0x557be7cdad0b, 0x557be7c7c55b, 0x557be7c7c9d6, 0x557be7dc1d8a, 0x557be7bf6317
    		 
    Allocated at 0x557be8552065, 0x557be7cda538, 0x557be7cda6b9, mysys/safemalloc.c:191, sql/sql_plugin.cc:3156, sql/sql_audit.cc:510, 0x557be7deec06, sql/sql_plugin.cc:3072
    		 
    Error: Freeing overrun buffer  mysys/safemalloc.c:191, mysys/my_malloc.c:219, sql/sql_plugin.cc:3156, sql/sql_plugin.cc:3183, sql/sql_class.cc:1663, sql/sql_class.cc:1710, sql/sql_acl.cc:6586, sql/mysqld.cc:5800
    		 
    Allocated at mysys/my_malloc.c:240, sql/sql_plugin.cc:2986, sql/sql_plugin.cc:3025, sql/sql_plugin.cc:3072, 0x7fa182df3fad, 0x7fa182df71b5, sql/sql_audit.cc:510, sql/sql_audit.cc:543
    		 
    Error: Freeing overrun buffer  mysys/safemalloc.c:191, mysys/my_malloc.c:219, sql/sql_plugin.cc:3156, sql/sql_plugin.cc:3183, sql/sql_class.cc:1663, sql/sql_class.cc:1710, sql/sql_udf.cc:262, sql/mysqld.cc:5802
    		 
    Allocated at mysys/my_malloc.c:240, sql/sql_plugin.cc:2986, sql/sql_plugin.cc:3025, sql/sql_plugin.cc:3072, 0x7fa182df3fad, 0x7fa182df71b5, sql/sql_audit.cc:510, sql/sql_audit.cc:543
    		 
    Error: Freeing overrun buffer  mysys/safemalloc.c:191, mysys/my_malloc.c:219, sql/sql_plugin.cc:3156, sql/sql_plugin.cc:3183, sql/sql_class.cc:1663, sql/sql_class.cc:1710, sql/events.cc:944, sql/mysqld.cc:5823
    		 
    Allocated at mysys/my_malloc.c:240, sql/sql_plugin.cc:2986, sql/sql_plugin.cc:3025, sql/sql_plugin.cc:3072, 0x7fa182df3fad, 0x7fa182df71b5, sql/sql_audit.cc:510, sql/sql_audit.cc:543
    		 
    2017-08-26 17:50:42 140331719626880 [Note] /data/bld/bb-10.1-wlad-debug/bin/mysqld: ready for connections.

  • shutdown the server

    10.1 debug

    		 
    Program terminated with signal SIGSEGV, Segmentation fault.
    		 
    #0  0x0000557be7cd5870 in intern_plugin_unlock (lex=0x0, plugin=0x7fa1838b62f0) at /data/src/bb-10.1-wlad/sql/sql_plugin.cc:1331
    		 
    1331      DBUG_ASSERT(pi->ref_count);
    		 
     
    		 
    #0  0x0000557be7cd5870 in intern_plugin_unlock (lex=0x0, plugin=0x7fa1838b62f0) at /data/src/bb-10.1-wlad/sql/sql_plugin.cc:1331
    		 
    #1  0x0000557be7cd59a8 in plugin_unlock (thd=0x0, plugin=0x7fa1838b62f0) at /data/src/bb-10.1-wlad/sql/sql_plugin.cc:1356
    		 
    #2  0x0000557be7da8881 in TABLE_SHARE::destroy (this=0x7fa183b11488) at /data/src/bb-10.1-wlad/sql/table.cc:428
    		 
    #3  0x0000557be7da8a10 in free_table_share (share=0x7fa183b11488) at /data/src/bb-10.1-wlad/sql/table.cc:469
    		 
    #4  0x0000557be7e80d93 in tdc_delete_share_from_hash (element=0x7fa183b2bf10) at /data/src/bb-10.1-wlad/sql/table_cache.cc:411
    		 
    #5  0x0000557be7e81d48 in tdc_release_share (share=0x7fa183b11488) at /data/src/bb-10.1-wlad/sql/table_cache.cc:790
    		 
    #6  0x0000557be7db077c in closefrm (table=0x7fa183b9d670, free_share=true) at /data/src/bb-10.1-wlad/sql/table.cc:3081
    		 
    #7  0x0000557be7c5241b in intern_close_table (table=0x7fa183b9d670) at /data/src/bb-10.1-wlad/sql/sql_base.cc:354
    		 
    #8  0x0000557be7e80589 in tc_purge (mark_flushed=true) at /data/src/bb-10.1-wlad/sql/table_cache.cc:204
    		 
    #9  0x0000557be7c5283e in close_cached_tables (thd=0x0, tables=0x0, wait_for_refresh=false, timeout=31536000) at /data/src/bb-10.1-wlad/sql/sql_base.cc:485
    		 
    #10 0x0000557be7e80f00 in tdc_start_shutdown () at /data/src/bb-10.1-wlad/sql/table_cache.cc:460
    		 
    #11 0x0000557be7bef1b2 in clean_up (print_message=true) at /data/src/bb-10.1-wlad/sql/mysqld.cc:2120
    		 
    #12 0x0000557be7beedd8 in unireg_end () at /data/src/bb-10.1-wlad/sql/mysqld.cc:2002
    		 
    #13 0x0000557be7beecef in kill_server (sig_ptr=0x0) at /data/src/bb-10.1-wlad/sql/mysqld.cc:1930
    		 
    #14 0x0000557be7beed17 in kill_server_thread (arg=0x7fa1864a0220) at /data/src/bb-10.1-wlad/sql/mysqld.cc:1953
    		 
    #15 0x00007fa1860b0494 in start_thread (arg=0x7fa1863c2b00) at pthread_create.c:333
    		 
    #16 0x00007fa1841fb93f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Some other observed consequences are in MDEV-13651 and MDEV-13652, both of which I'm closing as duplicates.

Could not reproduce any of it on 10.2. Apparently, the root cause was fixed within the scope of MDEV-13060, but only in 10.2. Since we provide both plugins in 10.1, at least in sources, it should work there too.
Generated at Thu Feb 08 08:07:15 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.