[MDEV-6162] TokuDB: multiple locks and unlock of not locked TOKUDB_SHARE::num_DBs_lock Created: 2014-04-24  Updated: 2014-05-06  Resolved: 2014-05-06

Status: Closed
Project: MariaDB Server
Component/s: None
Affects Version/s: 5.5.37
Fix Version/s: 5.5.38

Type: Bug Priority: Major
Reporter: Sergey Vojtovich Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: tokudb, upstream

Attachments: Text File num_DBs_lock.patch    
Issue Links:
PartOf
is part of MDEV-5089 possible deadlocks between rwlocks an... Closed

 Description   

Found while analyzing test failures for MDEV-5089.

To reproduce these problems compile MariaDB as described in MDEV-5089 with -DUSE_MUTEX_INSTEAD_OF_RW_LOCKS. Run tokudb and tokudb_bugs test suites.

First, there are attempts to lock read-locked num_DBs_lock. It is locked in ha_tokudb::start_bulk_insert() and then in ha_tokudb::update_row() or ha_tokudb::acquire_table_lock(). It may be not a problem, but as this situation is handled in ha_tokudb::write_row() it should be checked at least.

Second, ha_tokudb::write_row() unlocks not locked num_DBs_lock (in case of error in the very beginning of the function). This should be a problem according to the pthread_rwlock_unlock manual: results are undefined if the read-write lock rwlock is not held by the calling thread.

A rough patching solving the above issues attached.



 Comments   
Comment by Rich Prohaska [ 2014-04-29 ]

Can I define a cmake variable or do I need to hack the source to define USE_MUTEX_INSTEAD_OF_RW_LOCKS?

Comment by Sergey Vojtovich [ 2014-04-29 ]

Hi Rich,

there seem to be no corresponding cmake variable. Either hack sources (include/my_pthread.h) or compile with CFLAGS=-DUSE_MUTEX_INSTEAD_OF_RW_LOCKS.

Comment by Rich Prohaska [ 2014-04-29 ]

I want to reproduce case (1). If you know of a particular tokudb test that exposes these locking problems, please let me know. Otherwise, I will run all of the test and try to discover one.

The error cleanup in described in case (2) is unfortunate. Will fix that one.

Comment by Sergey Vojtovich [ 2014-04-29 ]

IIRC any of the following tests should expose the problem:
tokudb_bugs.1795
tokudb_bugs.1833
tokudb_bugs.2548
tokudb_bugs.5003
tokudb_bugs.tokudb_mrr
tokudb_bugs.tokudb_mrr2
tokudb.auto_increment
tokudb.auto_increment_boundary
tokudb.auto_increment_boundary_traditional
tokudb.i_s_tokudb_lock_waits_released
tokudb.i_s_tokudb_locks_released
tokudb.nested_txn_autocommit
tokudb.nested_txn_begin
tokudb.type_timestamp

Comment by Rich Prohaska [ 2014-04-29 ]

I hit this problem first. Is there is patch for it?
Version: '5.5.37-MariaDB-debug' socket: '/tmp/rfp.sock' port: 30000 Source distribution
[New Thread 0x7ffff7b16700 (LWP 11193)]
safe_mutex: Found wrong usage of mutex '(&that->m_rwlock)' and 'LOCK_global_system_variables'

#2 0x0000000000cb87d4 in safe_mutex_lock (mp=0x15605e0 <LOCK_system_variables_hash>, my_flags=0, file=0xd61ec0 "/home/prohaska/mdev6162/mariadb-5.5.37/include/mysql/psi/mysql_thread.h", line=761) at /ho\
me/prohaska/mdev6162/mariadb-5.5.37/mysys/thr_mutex.c:266
#3 0x000000000060ebc2 in inline_mysql_rwlock_rdlock (that=0x15605e0 <LOCK_system_variables_hash>, src_file=0xd621f0 "/home/prohaska/mdev6162/mariadb-5.5.37/sql/sql_plugin.cc", src_line=2916) at /home/pr\
ohaska/mdev6162/mariadb-5.5.37/include/mysql/psi/mysql_thread.h:761
#4 0x0000000000615d97 in intern_sys_var_ptr (thd=0x7ffdb6502060, offset=24, global_lock=false) at /home/prohaska/mdev6162/mariadb-5.5.37/sql/sql_plugin.cc:2916
#5 0x0000000000616956 in sys_var_pluginvar::real_value_ptr (this=0x7ffff7120ad8, thd=0x7ffdb6502060, type=OPT_SESSION) at /home/prohaska/mdev6162/mariadb-5.5.37/sql/sql_plugin.cc:3226
#6 0x0000000000616a1e in sys_var_pluginvar::do_value_ptr (this=0x7ffff7120ad8, thd=0x7ffdb6502060, type=OPT_SESSION, base=0x7ffff7b14330) at /home/prohaska/mdev6162/mariadb-5.5.37/sql/sql_plugin.cc:3255
#7 0x0000000000618fa8 in sys_var_pluginvar::session_value_ptr (this=0x7ffff7120ad8, thd=0x7ffdb6502060, base=0x7ffff7b14330) at /home/prohaska/mdev6162/mariadb-5.5.37/sql/sql_plugin.cc:289
#8 0x000000000056d6ae in sys_var::value_ptr (this=0x7ffff7120ad8, thd=0x7ffdb6502060, type=OPT_SESSION, base=0x7ffff7b14330) at /home/prohaska/mdev6162/mariadb-5.5.37/sql/set_var.cc:254
#9 0x00000000006782e4 in show_status_array (thd=0x7ffdb6502060, wild=0x0, variables=0x7ffd8587e1c8, value_type=OPT_SESSION, status_var=0x0, prefix=0xd6d6c3 "", table=0x7ffd8584c078, ucase_names=false, c\
ond=0x0) at /home/prohaska/mdev6162/mariadb-5.5.37/sql/sql_show.cc:2659
#10 0x00000000006878db in fill_variables (thd=0x7ffdb6502060, tables=0x7ffd858453a0, cond=0x0) at /home/prohaska/mdev6162/mariadb-5.5.37/sql/sql_show.cc:6943

Comment by Sergey Vojtovich [ 2014-04-29 ]

Yes, see first comment in MDEV-5089.

A workaround:

=== modified file 'sql/sql_plugin.cc'
--- sql/sql_plugin.cc	2014-04-09 12:28:07 +0000
+++ sql/sql_plugin.cc	2014-04-24 10:14:36 +0000
@@ -2913,7 +2913,10 @@ static uchar *intern_sys_var_ptr(THD* th
   {
     uint idx;
 
-    mysql_rwlock_rdlock(&LOCK_system_variables_hash);
+#undef pthread_mutex_lock
+#undef pthread_mutex_unlock
+//  mysql_rwlock_rdlock(&LOCK_system_variables_hash);
+//    pthread_mutex_lock(&LOCK_system_variables_hash.m_rwlock.mutex);
 
     thd->variables.dynamic_variables_ptr= (char*)
       my_realloc(thd->variables.dynamic_variables_ptr,
@@ -2973,7 +2976,8 @@ static uchar *intern_sys_var_ptr(THD* th
     thd->variables.dynamic_variables_size=
            global_system_variables.dynamic_variables_size;
 
-    mysql_rwlock_unlock(&LOCK_system_variables_hash);
+//    pthread_mutex_unlock(&LOCK_system_variables_hash.m_rwlock.mutex);
+//    mysql_rwlock_unlock(&LOCK_system_variables_hash);
   }
   DBUG_RETURN((uchar*)thd->variables.dynamic_variables_ptr + offset);
 }
 
=== modified file 'sql/sql_show.cc'
--- sql/sql_show.cc	2014-04-15 15:41:08 +0000
+++ sql/sql_show.cc	2014-04-24 10:14:36 +0000
@@ -6889,11 +6889,15 @@ int fill_variables(THD *thd, TABLE_LIST
 
   COND *partial_cond= make_cond_for_info_schema(cond, tables);
 
-  mysql_rwlock_rdlock(&LOCK_system_variables_hash);
+#undef pthread_mutex_lock
+#undef pthread_mutex_unlock
+//  mysql_rwlock_rdlock(&LOCK_system_variables_hash);
+  pthread_mutex_lock(&LOCK_system_variables_hash.m_rwlock.mutex);
   res= show_status_array(thd, wild, enumerate_sys_vars(thd, sorted_vars, option_type),
                          option_type, NULL, "", tables->table,
                          upper_case_names, partial_cond);
-  mysql_rwlock_unlock(&LOCK_system_variables_hash);
+  pthread_mutex_unlock(&LOCK_system_variables_hash.m_rwlock.mutex);
+//  mysql_rwlock_unlock(&LOCK_system_variables_hash);
   DBUG_RETURN(res);
 }
 
 

Comment by Rich Prohaska [ 2014-04-29 ]

tokudb 7.1.6 will include these bug fixes. Thanks.

Generated at Thu Feb 08 07:09:50 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.