[MDEV-12747] main.mysqld_option_err fails in buildbot with timeout Created: 2017-05-09  Updated: 2019-01-22  Resolved: 2019-01-22

Status: Closed
Project: MariaDB Server
Component/s: Tests
Affects Version/s: 10.2
Fix Version/s: 10.4.2, 10.2.22, 10.3.13

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Sergey Vojtovich
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Duplicate
is duplicated by MDEV-12728 perfschema.bad_option_3 failed on Pow... Closed
is duplicated by MDEV-13892 perfschema.bad_option_1 fails in buil... Closed
is duplicated by MDEV-14197 perfschema.bad_option_5 failed in bui... Closed

 Description   

http://buildbot.askmonty.org/buildbot/builders/p8-trusty-bintar/builds/711/steps/test/logs/stdio
http://buildbot.askmonty.org/buildbot/builders/kvm-deb-wheezy-x86/builds/4351/steps/mtr/logs/stdio

main.mysqld_option_err                   w3 [ fail ]  timeout after 900 seconds
        Test ended at 2017-05-07 04:55:48
 
Test case timeout after 900 seconds
 
== /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/var/3/log/mysqld_option_err.log == 
Test that unknown option is not silently ignored.
Test bad binlog format.
Test bad default storage engine.
 
 == /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/var/3/tmp/analyze-timeout-mysqld.1.err ==
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/' (111)
 
 
 - saving '/var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/var/3/log/main.mysqld_option_err/' to '/var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/var/log/main.mysqld_option_err/'
 - found 'core' (1/5)
 
Trying 'dbx' to get a backtrace
worker[3] > Restart  - not started
 
Trying 'gdb' to get a backtrace from coredump /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/var/log/main.mysqld_option_err/mysqld.1/data/core
Core generated by '/var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 29931]
[New LWP 30037]
[New LWP 29932]
[New LWP 29933]
[New LWP 29934]
[New LWP 29935]
 
warning: Could not load shared library symbols for linux-vdso64.so.1.
Do you need "set solib-search-path" or "set sysroot"?
 
warning: File "/opt/at8.0/lib64/power8/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
	add-auto-load-safe-path /opt/at8.0/lib64/power8/libthread_db-1.0.so
line to your configuration file "/var/lib/buildbot/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/var/lib/buildbot/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 
warning: File "/opt/at8.0/lib64/power8/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `/var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/mysqld --defaults-grou'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00003fff79a50ef0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60
60	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
#0  0x00003fff79a50ef0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60
#1  0x000000004882e860 in my_write_core (sig=<optimized out>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysys/stacktrace.c:477
#2  0x000000004810645c in handle_fatal_signal (sig=<optimized out>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/signal_handler.cc:299
#3  <signal handler called>
#4  0x00003fff793a1438 in poll () at ../sysdeps/unix/syscall-template.S:81
#5  0x00003fff793ca0c4 in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27
#6  0x0000000047e25460 in poll (__timeout=-1, __nfds=2, __fds=<optimized out>) at /opt/at8.0/include/bits/poll2.h:41
#7  handle_connections_sockets () at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/mysqld.cc:6557
#8  0x0000000047e2facc in mysqld_main (argc=128, argv=0x1001e0c9478) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/mysqld.cc:6023
#9  0x0000000047e05160 in main (argc=<optimized out>, argv=<optimized out>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/main.cc:25
 
Thread 6 (LWP 29935):
#0  0x00003fff79a4f7d0 in __pthread_cond_wait (cond=0x48f5a7c0 <COND_slave_background>, mutex=0x48f58000 <LOCK_slave_background>) at pthread_cond_wait.c:188
#1  0x0000000047e4d200 in inline_mysql_cond_wait (that=<optimized out>, mutex=<optimized out>, src_file=0x488eaa80 "/var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/slave.cc", src_line=334) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/include/mysql/psi/mysql_thread.h:1138
#2  handle_slave_background (arg=<optimized out>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/slave.cc:334
#3  0x00000000487c67e0 in pfs_spawn_thread (arg=0x1001e3813c8) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/storage/perfschema/pfs.cc:1862
#4  0x00003fff79a47cec in start_thread (arg=0x3fff74108180) at pthread_create.c:312
#5  0x00003fff793b1140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
 
Thread 5 (LWP 29934):
#0  0x00003fff79a55714 in do_sigwait (sig=0x3fff74167630, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:61
#1  __sigwait (set=<optimized out>, sig=0x3fff74167630) at ../sysdeps/unix/sysv/linux/sigwait.c:98
#2  0x0000000047e229cc in signal_hand (arg=<optimized out>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/mysqld.cc:3488
#3  0x00000000487c67e0 in pfs_spawn_thread (arg=0x1001e330948) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/storage/perfschema/pfs.cc:1862
#4  0x00003fff79a47cec in start_thread (arg=0x3fff74168180) at pthread_create.c:312
#5  0x00003fff793b1140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
 
Thread 4 (LWP 29933):
#0  0x00003fff79a4fcc0 in __pthread_cond_timedwait (cond=0x497ecd38 <COND_checkpoint>, mutex=0x497ecd00 <LOCK_checkpoint>, abstime=0x3fff6f84e5a8) at pthread_cond_timedwait.c:199
#1  0x00000000486b890c in inline_mysql_cond_timedwait (src_file=0x489bb490 "/var/lib/buildbot/maria-slave/p8-trusty-bintar/build/storage/maria/ma_servicethread.c", src_line=116, abstime=<optimized out>, mutex=0x497ecd00 <LOCK_checkpoint>, that=0x497ecd38 <COND_checkpoint>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/include/mysql/psi/mysql_thread.h:1175
#2  my_service_thread_sleep (control=0x48ed6fc0 <checkpoint_control>, sleep_time=<optimized out>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/storage/maria/ma_servicethread.c:115
#3  0x00000000486accf4 in ma_checkpoint_background (arg=0x1e) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/storage/maria/ma_checkpoint.c:708
#4  0x00000000487c67e0 in pfs_spawn_thread (arg=0x1001e2305f8) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/storage/perfschema/pfs.cc:1862
#5  0x00003fff79a47cec in start_thread (arg=0x3fff6f84f180) at pthread_create.c:312
#6  0x00003fff793b1140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
 
Thread 3 (LWP 29932):
#0  0x00003fff79a4fcc0 in __pthread_cond_timedwait (cond=0x497fcbb8 <COND_timer>, mutex=0x497fcb50 <LOCK_timer>, abstime=0x3fff74d8e6a8) at pthread_cond_timedwait.c:199
#1  0x00000000488352e8 in inline_mysql_cond_timedwait (that=0x497fcbb8 <COND_timer>, mutex=0x497fcb50 <LOCK_timer>, src_file=0x489cf6c8 "/var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysys/thr_timer.c", src_line=292, abstime=0x3fff74d8e6a8) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/include/mysql/psi/mysql_thread.h:1175
#2  timer_handler (arg=<optimized out>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysys/thr_timer.c:292
#3  0x00000000487c67e0 in pfs_spawn_thread (arg=0x1001e222758) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/storage/perfschema/pfs.cc:1862
#4  0x00003fff79a47cec in start_thread (arg=0x3fff74d8f180) at pthread_create.c:312
#5  0x00003fff793b1140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
 
Thread 2 (LWP 30037):
#0  0x00003fff79a4fcc0 in __pthread_cond_timedwait (cond=0x48f5a7f8 <COND_thread_cache>, mutex=0x48f582a0 <LOCK_thread_cache>, abstime=0x3fff6ef275e8) at pthread_cond_timedwait.c:199
#1  0x0000000047e241e0 in inline_mysql_cond_timedwait (src_file=0x488e1710 "/var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/mysqld.cc", src_line=2951, abstime=0x3fff6ef275e8, mutex=0x48f582a0 <LOCK_thread_cache>, that=0x48f5a7f8 <COND_thread_cache>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/include/mysql/psi/mysql_thread.h:1175
#2  cache_thread (thd=0x3fff480009a8) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/mysqld.cc:2950
#3  one_thread_per_connection_end (thd=0x3fff480009a8, put_in_cache=<optimized out>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/mysqld.cc:3039
#4  0x0000000047ff4dd8 in do_handle_one_connection (connect=<optimized out>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/sql_connect.cc:1373
#5  0x0000000047ff5104 in handle_one_connection (arg=0x1001e39d268) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/sql_connect.cc:1260
#6  0x00000000487c67e0 in pfs_spawn_thread (arg=0x1001e36f028) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/storage/perfschema/pfs.cc:1862
#7  0x00003fff79a47cec in start_thread (arg=0x3fff6ef28180) at pthread_create.c:312
#8  0x00003fff793b1140 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
 
Thread 1 (LWP 29931):
#0  0x00003fff79a50ef0 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60
#1  0x000000004882e860 in my_write_core (sig=<optimized out>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysys/stacktrace.c:477
#2  0x000000004810645c in handle_fatal_signal (sig=<optimized out>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/signal_handler.cc:299
#3  <signal handler called>
#4  0x00003fff793a1438 in poll () at ../sysdeps/unix/syscall-template.S:81
#5  0x00003fff793ca0c4 in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27
#6  0x0000000047e25460 in poll (__timeout=-1, __nfds=2, __fds=<optimized out>) at /opt/at8.0/include/bits/poll2.h:41
#7  handle_connections_sockets () at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/mysqld.cc:6557
#8  0x0000000047e2facc in mysqld_main (argc=128, argv=0x1001e0c9478) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/mysqld.cc:6023
#9  0x0000000047e05160 in main (argc=<optimized out>, argv=<optimized out>) at /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/sql/main.cc:25
 
Compressed file /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/var/log/main.mysqld_option_err/mysqld.1/data/core
***Warnings generated in error logs during shutdown after running tests: main.mysql_locale_posix main.mysql_not_windows main.myisam_mrr main.myisam_icp main.mysql-bug45236 main.mysqld_option_err main.myisam_icp_notembedded main.myisam_optimize main.mysql_protocols main.mysql5613mysql main.mysql_upgrade_no_innodb main.mysql_comments main.mysql_cp932 main.mysqld--defaults-file main.mysql57_virtual main.myisampack main.mysql-bug41486 main.mysqld_help_crash-9183 main.mysqladmin main.mysql main.mysqld--help main.mysql_tzinfo_to_sql_symlink main.myisam_explain_non_select_all

main.mysqld_option_err                   w1 [ fail ]  timeout after 900 seconds
        Test ended at 2017-04-06 16:17:37
 
Test case timeout after 900 seconds
 
== /run/shm/var/1/log/mysqld_option_err.log == 
Test that unknown option is not silently ignored.
Test bad binlog format.
Test bad default storage engine.



 Comments   
Comment by Alice Sherepa [ 2017-10-23 ]

on 10.3 http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/10084/steps/test_2/logs/stdio

Comment by Sergey Vojtovich [ 2019-01-21 ]

This patch makes it hang reliably for me:

diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc
index 423e731..dfb9b06 100644
--- a/storage/innobase/handler/ha_innodb.cc
+++ b/storage/innobase/handler/ha_innodb.cc
@@ -285,7 +285,10 @@ thd_destructor_proxy(void *)
        srv_running.store(myvar, std::memory_order_relaxed);
        /* wait until the server wakes the THD to abort and die */
        while (!myvar->abort)
+       {
+               sleep(1);
                mysql_cond_wait(&thd_destructor_cond, &thd_destructor_mutex);
+       }
        mysql_mutex_unlock(&thd_destructor_mutex);
        srv_running.store(NULL, std::memory_order_relaxed);

Generated at Thu Feb 08 08:00:09 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.