Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13444

assertion mtflush_io != NULL on exit with innodb-use-mtflush

Details

    Description

      10.2 488f46f3de51070fa91e5eadcc215b6a4e343f7b

       ( cd mysql-test/ ; ./mtr --mysqld=--innodb-use-mtflush --mem innodb.log_file_name )
       
      Thread 1 (Thread 0x3fff91f5d3a0 (LWP 31777)):
      #0  0x00003fff918f1964 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:58
      #1  0x0000000029349fe8 in my_write_core (sig=<optimized out>) at /home/danielgb/mariadb-server-10.2/mysys/stacktrace.c:477
      #2  0x0000000028d070bc in handle_fatal_signal (sig=<optimized out>) at /home/danielgb/mariadb-server-10.2/sql/signal_handler.cc:299
      #3  <signal handler called>
      #4  0x00003fff9141edb0 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:54
      #5  0x00003fff91421270 in __GI_abort () at abort.c:89
      #6  0x0000000028a4e298 in ut_dbg_assertion_failed (expr=0x29494668 "mtflush_io != NULL", file=0x29494620 "/home/danielgb/mariadb-server-10.2/storage/innobase/buf/buf0mtflu.cc", line=<optimized out>) at /home/danielgb/mariadb-server-10.2/storage/innobase/ut/ut0dbg.cc:60
      #7  0x000000002910adb0 in buf_mtflu_io_thread_exit () at /home/danielgb/mariadb-server-10.2/storage/innobase/buf/buf0mtflu.cc:402
      #8  0x00000000290335f4 in srv_shutdown_all_bg_threads () at /home/danielgb/mariadb-server-10.2/storage/innobase/srv/srv0start.cc:1301
      #9  0x00000000290339b4 in srv_init_abort_low (create_new_db=<optimized out>, err=DB_ERROR) at /home/danielgb/mariadb-server-10.2/storage/innobase/srv/srv0start.cc:1370
      #10 0x000000002903c620 in innobase_start_or_create_for_mysql () at /home/danielgb/mariadb-server-10.2/storage/innobase/srv/srv0start.cc:2604
      #11 0x0000000028ef9c54 in innobase_init (p=<optimized out>) at /home/danielgb/mariadb-server-10.2/storage/innobase/handler/ha_innodb.cc:4422
      #12 0x0000000028d0a918 in ha_initialize_handlerton (plugin=0x100243a33a0) at /home/danielgb/mariadb-server-10.2/sql/handler.cc:512
      #13 0x0000000028b33a5c in plugin_initialize (tmp_root=0x3fffe82001c8, plugin=0x100243a33a0, argc=0x29a12ec8 <remaining_argc>, argv=0x10024219848, options_only=false) at /home/danielgb/mariadb-server-10.2/sql/sql_plugin.cc:1413
      #14 0x0000000028b3453c in plugin_init (argc=0x29a12ec8 <remaining_argc>, argv=0x10024219848, flags=<optimized out>) at /home/danielgb/mariadb-server-10.2/sql/sql_plugin.cc:1694
      #15 0x0000000028a74980 in init_server_components () at /home/danielgb/mariadb-server-10.2/sql/mysqld.cc:5253
      #16 0x0000000028a7a960 in mysqld_main (argc=145, argv=0x10024219848) at /home/danielgb/mariadb-server-10.2/sql/mysqld.cc:5845
      #17 0x0000000028a52068 in main (argc=<optimized out>, argv=<optimized out>) at /home/danielgb/mariadb-server-10.2/sql/main.cc:25
      

      Found by GSoC student: Sumit Lakra

      Attachments

        Issue Links

          Activity

            danblack Daniel Black added a comment - - edited

            10.1 8b019f87dd51b1d2755655d2f136d5429fc47e1f

            ( cd mysql-test/ ; ./mtr --mysqld=--innodb-use-mtflush --mem  --parallel=50 --max-test-fail=50 --force  innodb.101_compatibility  )
             
            Thread 1 (Thread 0x7fc1b0601b00 (LWP 15814)):
            #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
            #1  0x000055bd53c34cba in my_write_core (sig=sig@entry=6) at /home/danielgb/mariadb-server-10.1/mysys/stacktrace.c:477
            #2  0x000055bd5381c1b8 in handle_fatal_signal (sig=6) at /home/danielgb/mariadb-server-10.1/sql/signal_handler.cc:296
            #3  <signal handler called>
            #4  0x00007fc1adf3e428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
            #5  0x00007fc1adf4002a in __GI_abort () at abort.c:89
            #6  0x000055bd53b89b8d in buf_mtflu_io_thread_exit () at /home/danielgb/mariadb-server-10.1/storage/xtradb/buf/buf0mtflu.cc:371
            #7  0x000055bd53b119ce in innodb_shutdown () at /home/danielgb/mariadb-server-10.1/storage/xtradb/srv/srv0start.cc:3254
            #8  0x000055bd53a3ae07 in innobase_end () at /home/danielgb/mariadb-server-10.1/storage/xtradb/handler/ha_innodb.cc:4531
            #9  0x000055bd5381e78b in ha_finalize_handlerton (plugin=0x7fc1a83f6818) at /home/danielgb/mariadb-server-10.1/sql/handler.cc:452
            #10 0x000055bd536bb7d9 in plugin_deinitialize (plugin=0x7fc1a83f6818, ref_check=ref_check@entry=true) at /home/danielgb/mariadb-server-10.1/sql/sql_plugin.cc:1207
            #11 0x000055bd536bf4f6 in reap_plugins () at /home/danielgb/mariadb-server-10.1/sql/sql_plugin.cc:1285
            #12 0x000055bd536bfdf5 in plugin_shutdown () at /home/danielgb/mariadb-server-10.1/sql/sql_plugin.cc:1938
            #13 0x000055bd5362794c in clean_up (print_message=print_message@entry=true) at /home/danielgb/mariadb-server-10.1/sql/mysqld.cc:2120
            #14 0x000055bd53628f2b in clean_up (print_message=true) at /home/danielgb/mariadb-server-10.1/sql/mysqld.cc:2007
            #15 unireg_end () at /home/danielgb/mariadb-server-10.1/sql/mysqld.cc:2001
            #16 0x000055bd5362e590 in kill_server (sig_ptr=0x0) at /home/danielgb/mariadb-server-10.1/sql/mysqld.cc:1929
            #17 0x000055bd5362e84e in kill_server_thread (arg=arg@entry=0x7fc1b06df008) at /home/danielgb/mariadb-server-10.1/sql/mysqld.cc:1952
            #18 0x000055bd53973eb4 in pfs_spawn_thread (arg=0x7fc1a1825208) at /home/danielgb/mariadb-server-10.1/storage/perfschema/pfs.cc:1860
            #19 0x00007fc1ae96d6ba in start_thread (arg=0x7fc1b0601b00) at pthread_create.c:333
            

            danblack Daniel Black added a comment - - edited 10.1 8b019f87dd51b1d2755655d2f136d5429fc47e1f ( cd mysql-test/ ; ./mtr --mysqld=--innodb-use-mtflush --mem --parallel=50 --max-test-fail=50 --force innodb.101_compatibility )   Thread 1 (Thread 0x7fc1b0601b00 (LWP 15814)): #0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62 #1 0x000055bd53c34cba in my_write_core (sig=sig@entry=6) at /home/danielgb/mariadb-server-10.1/mysys/stacktrace.c:477 #2 0x000055bd5381c1b8 in handle_fatal_signal (sig=6) at /home/danielgb/mariadb-server-10.1/sql/signal_handler.cc:296 #3 <signal handler called> #4 0x00007fc1adf3e428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 #5 0x00007fc1adf4002a in __GI_abort () at abort.c:89 #6 0x000055bd53b89b8d in buf_mtflu_io_thread_exit () at /home/danielgb/mariadb-server-10.1/storage/xtradb/buf/buf0mtflu.cc:371 #7 0x000055bd53b119ce in innodb_shutdown () at /home/danielgb/mariadb-server-10.1/storage/xtradb/srv/srv0start.cc:3254 #8 0x000055bd53a3ae07 in innobase_end () at /home/danielgb/mariadb-server-10.1/storage/xtradb/handler/ha_innodb.cc:4531 #9 0x000055bd5381e78b in ha_finalize_handlerton (plugin=0x7fc1a83f6818) at /home/danielgb/mariadb-server-10.1/sql/handler.cc:452 #10 0x000055bd536bb7d9 in plugin_deinitialize (plugin=0x7fc1a83f6818, ref_check=ref_check@entry=true) at /home/danielgb/mariadb-server-10.1/sql/sql_plugin.cc:1207 #11 0x000055bd536bf4f6 in reap_plugins () at /home/danielgb/mariadb-server-10.1/sql/sql_plugin.cc:1285 #12 0x000055bd536bfdf5 in plugin_shutdown () at /home/danielgb/mariadb-server-10.1/sql/sql_plugin.cc:1938 #13 0x000055bd5362794c in clean_up (print_message=print_message@entry=true) at /home/danielgb/mariadb-server-10.1/sql/mysqld.cc:2120 #14 0x000055bd53628f2b in clean_up (print_message=true) at /home/danielgb/mariadb-server-10.1/sql/mysqld.cc:2007 #15 unireg_end () at /home/danielgb/mariadb-server-10.1/sql/mysqld.cc:2001 #16 0x000055bd5362e590 in kill_server (sig_ptr=0x0) at /home/danielgb/mariadb-server-10.1/sql/mysqld.cc:1929 #17 0x000055bd5362e84e in kill_server_thread (arg=arg@entry=0x7fc1b06df008) at /home/danielgb/mariadb-server-10.1/sql/mysqld.cc:1952 #18 0x000055bd53973eb4 in pfs_spawn_thread (arg=0x7fc1a1825208) at /home/danielgb/mariadb-server-10.1/storage/perfschema/pfs.cc:1860 #19 0x00007fc1ae96d6ba in start_thread (arg=0x7fc1b0601b00) at pthread_create.c:333

            Reassigning, because I have done very little about the multi-threaded flushing.
            On a quick look, this code in buf0mtflu.cc screams ‘race condition’:

            	/* Wait about 1/2 sec to allow threads really exit */
            	os_thread_sleep(MT_WAIT_IN_USECS);
             
            	/* Make sure that work queue is empty */
            	while(!ib_wqueue_is_empty(mtflush_io->wq))
            	{
            		ib_wqueue_nowait(mtflush_io->wq);
            	}
             
            	mtflush_ctx->~thread_sync_t();
            	mtflush_ctx = NULL;
            

            The code is a little obfuscated, but it looks like mtflush_ctx and mtflush_io could be the same thing.
            The sleep should be replaced with something that really ensures that each thread has exited. An atomic counter of threads, perhaps? We already have os_thread_count which might be usable here (the I/O threads are shut down last).

            marko Marko Mäkelä added a comment - Reassigning, because I have done very little about the multi-threaded flushing. On a quick look, this code in buf0mtflu.cc screams ‘race condition’: /* Wait about 1/2 sec to allow threads really exit */ os_thread_sleep(MT_WAIT_IN_USECS);   /* Make sure that work queue is empty */ while(!ib_wqueue_is_empty(mtflush_io->wq)) { ib_wqueue_nowait(mtflush_io->wq); }   mtflush_ctx->~thread_sync_t(); mtflush_ctx = NULL; The code is a little obfuscated, but it looks like mtflush_ctx and mtflush_io could be the same thing. The sleep should be replaced with something that really ensures that each thread has exited. An atomic counter of threads, perhaps? We already have os_thread_count which might be usable here (the I/O threads are shut down last).

            In MDEV-25003, I wrote:

            The innodb_use_mtflush interface was something that had been copied from Percona XtraDB server. MySQL 5.7 introduced a different implementation of multi-threaded page cleaner that should not have this problem. The XtraDB mechanism was removed from MariaDB Server 10.3 as part of MDEV-13690.

            In MDEV-23855, we switched MariaDB Server 10.5.7 back to single-threaded flushing, using asynchronous I/O also for the doublewrite buffer.

            Furthermore, 10.2 will reach its end of life soon.

            marko Marko Mäkelä added a comment - In MDEV-25003 , I wrote: The innodb_use_mtflush interface was something that had been copied from Percona XtraDB server. MySQL 5.7 introduced a different implementation of multi-threaded page cleaner that should not have this problem. The XtraDB mechanism was removed from MariaDB Server 10.3 as part of MDEV-13690 . In MDEV-23855 , we switched MariaDB Server 10.5.7 back to single-threaded flushing, using asynchronous I/O also for the doublewrite buffer. Furthermore, 10.2 will reach its end of life soon.

            People

              jplindst Jan Lindström (Inactive)
              danblack Daniel Black
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.