Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-8066

Crash on unloading semisync_master plugin

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.0.15, 10.0(EOL), 10.1(EOL)
    • 10.0.23, 10.1.9
    • Replication, Tests
    • None
    • Linux

    Description

      This has so far only been observed once and was not reproducible so far
      (even with several clients doing transactions in parallel for days while the plugin was unloaded and then installed again every second):

      150331 9:19:22 [Note] Semi-sync replication switched OFF.
      		 
      150331 9:19:22 [Note] Semi-sync replication disabled on the master.
      		 
      150331 9:19:22 [ERROR] mysqld got signal 11 ;

      [...]

      /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xb70d4b]
      		 
      /usr/sbin/mysqld(handle_fatal_signal+0x398)[0x7257b8]
      		 
      /lib64/libpthread.so.0[0x377040f710]
      		 
      /usr/lib64/mysql/plugin/semisync_master.so(ActiveTranx::is_tranx_end_pos(char const*, unsigned long long)+0x24)[0x7fe91a5f9fe4]
      		 
      /usr/lib64/mysql/plugin/semisync_master.so(ReplSemiSyncMaster::commitTrx(char const*, unsigned long long)+0x19e)[0x7fe91a5fab3e]
      		 
      /usr/sbin/mysqld(Trans_delegate::after_commit(THD*, bool)+0xa2)[0x69b612]
      		 
      /usr/sbin/mysqld(ha_commit_trans(THD*, bool)+0x222)[0x728872]
      		 
      /usr/sbin/mysqld(trans_commit_stmt(THD*)+0x1b)[0x6a350b]
      		 
      /usr/sbin/mysqld(mysql_execute_command(THD*)+0x514)[0x5d16d4]
      		 
      /usr/sbin/mysqld[0x5d79d2]
      		 
      /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1b20)[0x5d9b90]
      		 
      /usr/sbin/mysqld(do_handle_one_connection(THD*)+0x453)[0x6956a3]
      		 
      /usr/sbin/mysqld(handle_one_connection+0x42)[0x695772]
      		 
      /lib64/libpthread.so.0[0x37704079d1]
      		 
      /lib64/libc.so.6(clone+0x6d)[0x37700e8b6d]

      My blind educated guess is a race condition between plugin callbacks and plugin teardown code. More specific: I think that ReplSemiSyncMaster::commitTrx() is still registered as an after_commit callback, but either at the time it gets called the plugins transaction hash table has just been freed, or it got freed just at the "right" time between invoking the callback and actually processing it?

      Attachments

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. MDEV-7096 In THD::cleanup(), apc_target.destroy() has assertion failure.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            hholzgra Hartmut Holzgraefe created issue -
            hholzgra Hartmut Holzgraefe made changes -
            Field Original Value New Value
            Description This has so far only been observed once and was not reproducible so far
            (even with several clients doing transactions in parallel for hours while the plugin was unloaded and then installed again every second):

            {noformat}
            150331 9:19:22 [Note] Semi-sync replication switched OFF.
            150331 9:19:22 [Note] Semi-sync replication disabled on the master.
            150331 9:19:22 [ERROR] mysqld got signal 11 ;
            {noformat}
            [...]
            {noformat}
            /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xb70d4b]
            /usr/sbin/mysqld(handle_fatal_signal+0x398)[0x7257b8]
            /lib64/libpthread.so.0[0x377040f710]
            /usr/lib64/mysql/plugin/semisync_master.so(ActiveTranx::is_tranx_end_pos(char const*, unsigned long long)+0x24)[0x7fe91a5f9fe4]
            /usr/lib64/mysql/plugin/semisync_master.so(ReplSemiSyncMaster::commitTrx(char const*, unsigned long long)+0x19e)[0x7fe91a5fab3e]
            /usr/sbin/mysqld(Trans_delegate::after_commit(THD*, bool)+0xa2)[0x69b612]
            /usr/sbin/mysqld(ha_commit_trans(THD*, bool)+0x222)[0x728872]
            /usr/sbin/mysqld(trans_commit_stmt(THD*)+0x1b)[0x6a350b]
            /usr/sbin/mysqld(mysql_execute_command(THD*)+0x514)[0x5d16d4]
            /usr/sbin/mysqld[0x5d79d2]
            /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1b20)[0x5d9b90]
            /usr/sbin/mysqld(do_handle_one_connection(THD*)+0x453)[0x6956a3]
            /usr/sbin/mysqld(handle_one_connection+0x42)[0x695772]
            /lib64/libpthread.so.0[0x37704079d1]
            /lib64/libc.so.6(clone+0x6d)[0x37700e8b6d]
            {noformat}

            My blind educated guess is a race condition between plugin callbacks and plugin teardown code. More specific: I think that ReplSemiSyncMaster::commitTrx() is still registered as an after_commit callback, but either at the time it gets called the plugins transaction hash table has just been freed, or it got freed just at the "right" time between invoking the callback and actually processing it?
            		 This has so far only been observed once and was not reproducible so far
            (even with several clients doing transactions in parallel for days while the plugin was unloaded and then installed again every second):

            {noformat}
            150331 9:19:22 [Note] Semi-sync replication switched OFF.
            150331 9:19:22 [Note] Semi-sync replication disabled on the master.
            150331 9:19:22 [ERROR] mysqld got signal 11 ;
            {noformat}
            [...]
            {noformat}
            /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xb70d4b]
            /usr/sbin/mysqld(handle_fatal_signal+0x398)[0x7257b8]
            /lib64/libpthread.so.0[0x377040f710]
            /usr/lib64/mysql/plugin/semisync_master.so(ActiveTranx::is_tranx_end_pos(char const*, unsigned long long)+0x24)[0x7fe91a5f9fe4]
            /usr/lib64/mysql/plugin/semisync_master.so(ReplSemiSyncMaster::commitTrx(char const*, unsigned long long)+0x19e)[0x7fe91a5fab3e]
            /usr/sbin/mysqld(Trans_delegate::after_commit(THD*, bool)+0xa2)[0x69b612]
            /usr/sbin/mysqld(ha_commit_trans(THD*, bool)+0x222)[0x728872]
            /usr/sbin/mysqld(trans_commit_stmt(THD*)+0x1b)[0x6a350b]
            /usr/sbin/mysqld(mysql_execute_command(THD*)+0x514)[0x5d16d4]
            /usr/sbin/mysqld[0x5d79d2]
            /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1b20)[0x5d9b90]
            /usr/sbin/mysqld(do_handle_one_connection(THD*)+0x453)[0x6956a3]
            /usr/sbin/mysqld(handle_one_connection+0x42)[0x695772]
            /lib64/libpthread.so.0[0x37704079d1]
            /lib64/libc.so.6(clone+0x6d)[0x37700e8b6d]
            {noformat}

            My blind educated guess is a race condition between plugin callbacks and plugin teardown code. More specific: I think that ReplSemiSyncMaster::commitTrx() is still registered as an after_commit callback, but either at the time it gets called the plugins transaction hash table has just been freed, or it got freed just at the "right" time between invoking the callback and actually processing it?
            elenst Elena Stepanova made changes -
            Fix Version/s 10.0 [ 16000 ]
            Assignee Kristian Nielsen [ knielsen ]
            ratzpo Rasmus Johansson (Inactive) made changes -
            Workflow MariaDB v2 [ 60712 ] MariaDB v3 [ 65960 ]
            elenst Elena Stepanova made changes -
            Status Open [ 1 ] Confirmed [ 10101 ]
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            Component/s Tests [ 10800 ]
            Fix Version/s 10.1 [ 16100 ]
            Affects Version/s 10.0 [ 16000 ]
            Affects Version/s 10.1 [ 16100 ]
            monty Michael Widenius made changes -
            Assignee Kristian Nielsen [ knielsen ] Michael Widenius [ monty ]
            monty Michael Widenius made changes -
            Status Confirmed [ 10101 ] In Progress [ 3 ]
            monty Michael Widenius made changes -
            Status In Progress [ 3 ] Stalled [ 10000 ]
            monty Michael Widenius made changes -
            Fix Version/s 10.0.23 [ 20401 ]
            Fix Version/s 10.1.9 [ 20301 ]
            Fix Version/s 10.0 [ 16000 ]
            Fix Version/s 10.1 [ 16100 ]
            Resolution Fixed [ 1 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 65960 ] MariaDB v4 [ 149102 ]

            People

              monty Michael Widenius
              hholzgra Hartmut Holzgraefe
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.