Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-7098

spider/bg.spider_fixes failed in buildbot with safe_mutex: Trying to unlock mutex conn->mta_conn_mutex that wasn't locked at storage/spider/spd_db_conn.cc, line 671

    Details

      Description

      It's a one-time failure so far, but it looks nasty.
      http://buildbot.askmonty.org/buildbot/builders/bintar-rhel6-p8-debug/builds/218/steps/test/logs/stdio

      CURRENT_TEST: spider/bg.spider_fixes
      mysqltest: At line 1117: query 'SELECT MAX(id) FROM t1' failed: 2013: Lost connection to MySQL server during query
       
      The result from queries just before the failure was:
      < snip >
      1554
      2331
      3108
      3885
      4662
      5439
      6216
      SET INSERT_ID=5000;
      INSERT INTO t1 () VALUES ();
      SELECT LAST_INSERT_ID();
      LAST_INSERT_ID()
      5000
      SELECT MAX(id) FROM t1;
      MAX(id)
      6216
      INSERT INTO t1 (id) VALUES (10000);
      SELECT LAST_INSERT_ID();
      LAST_INSERT_ID()
      5000
      SELECT MAX(id) FROM t1;

      Server [mysqld.1.1 - pid: 27307, winpid: 27307, exit: 256] failed during test run
      Server log from this test:
      ----------SERVER LOG START-----------
      safe_mutex: Trying to unlock mutex conn->mta_conn_mutex that wasn't locked at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_db_conn.cc, line 671
      Last used at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_conn.cc, line: 2379
      141112 11:36:49 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.0.15-MariaDB-debug-log
      key_buffer_size=1048576
      read_buffer_size=131072
      max_used_connections=4
      max_threads=153
      thread_count=3
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62987 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0xfff896ac070
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0xfff77ff6680 thread_stack 0x48000
      :0(00000017.plt_call.strsignal@@GLIBC_2.3)[0x10d25084]
      sql/signal_handler.cc:153(00000017.plt_call.strsignal@@GLIBC_2.3)[0x1050f618]
      linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0xfff8ab30448]
      /opt/at7.0/lib64/power7/libc.so.6(gsignal-0x16f708)[0xfff8a26f8f0]
      /opt/at7.0/lib64/power7/libc.so.6(abort-0x16dab4)[0xfff8a2719c4]
      :0(00000017.plt_call.strsignal@@GLIBC_2.3)[0x10d2bae0]
      /home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/3/plugins/ha_spider.so(+0x74dc4)[0xfff85e34dc4]
      /home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/3/plugins/ha_spider.so(_Z17spider_db_errornoP14st_spider_conn-0x135b74)[0xfff85e38984]
      /home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/3/plugins/ha_spider.so(_ZN15spider_db_mysql6commitEPi-0x7503c)[0xfff85efe8bc]
      /home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/3/plugins/ha_spider.so(_Z16spider_db_commitP14st_spider_conn-0x133534)[0xfff85e3b09c]
      /home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/3/plugins/ha_spider.so(_Z13spider_commitP10handlertonP3THDb-0x13cae0)[0xfff85e31580]
      sql/handler.cc:1527(00000017.plt_call.strsignal@@GLIBC_2.3)[0x10514548]
      sql/handler.cc:1508(00000017.plt_call.strsignal@@GLIBC_2.3)[0x105143e8]
      sql/handler.cc:1385(00000017.plt_call.strsignal@@GLIBC_2.3)[0x10513c38]
      sql/transaction.cc:394(00000017.plt_call.strsignal@@GLIBC_2.3)[0x1040f35c]
      sql/sql_parse.cc:5126(00000017.plt_call.strsignal@@GLIBC_2.3)[0x102914e8]
      sql/sql_parse.cc:6407(00000017.plt_call.strsignal@@GLIBC_2.3)[0x10294bb0]
      sql/sql_parse.cc:1299(00000017.plt_call.strsignal@@GLIBC_2.3)[0x1028655c]
      sql/sql_parse.cc:996(00000017.plt_call.strsignal@@GLIBC_2.3)[0x1028572c]
      sql/sql_connect.cc:1379(00000017.plt_call.strsignal@@GLIBC_2.3)[0x103f3af0]
      sql/sql_connect.cc:1293(00000017.plt_call.strsignal@@GLIBC_2.3)[0x103f3754]
      /opt/at7.0/lib64/power7/libpthread.so.0(+0xc548)[0xfff8aafc548]
      /opt/at7.0/lib64/power7/libc.so.6(clone-0xa3978)[0xfff8a343c90]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0xfff77410888): SELECT MAX(id) FROM t1
      Connection ID (thread ID): 21
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
       
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      Writing a core file
      ----------SERVER LOG END-------------
       
       
       - saving '/home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/3/log/spider/bg.spider_fixes/' to '/home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/log/bg.spider_fixes/'
       - found 'core.27308' (0/5)
       
      Trying 'dbx' to get a backtrace
       
      Trying 'gdb' to get a backtrace
      Core generated by '/home/buildbot/maria-slave/power8-vlp01-debug/build/sql/mysqld'
      Output from gdb follows. The first stack trace is from the failing thread.
      The following stack traces are from all threads (so the failing one is
      duplicated).
      --------------------------
      [New Thread 27698]
      [New Thread 27310]
      [New Thread 27316]
      [New Thread 27382]
      [New Thread 27708]
      [New Thread 27315]
      [New Thread 27380]
      [New Thread 27314]
      [New Thread 27713]
      [New Thread 27312]
      [New Thread 27385]
      [New Thread 27384]
      [New Thread 27386]
      [New Thread 27388]
      [New Thread 27383]
      [New Thread 27317]
      [New Thread 27710]
      [New Thread 27308]
      [New Thread 27387]
      [New Thread 27714]
      [New Thread 27389]
      [New Thread 27390]
      [New Thread 27397]
      [New Thread 27697]
      [New Thread 27381]
      [New Thread 27313]
       
      warning: Ignoring non-absolute filename: <linux-vdso64.so.1>
      Missing separate debuginfo for linux-vdso64.so.1
      Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/05/920648c05dcd5d47af1fe7b0befecb469a256d
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
      To enable execution of this file add
      	add-auto-load-safe-path /opt/at7.0/lib64/libthread_db-1.0.so
      line to your configuration file "/home/buildbot/.gdbinit".
      To completely disable this security protection add
      	set auto-load safe-path /
      line to your configuration file "/home/buildbot/.gdbinit".
      For more information about this security protection see the
      "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
      	info "(gdb)Auto-loading safe path"
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
       
      warning: File "/opt/at7.0/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
       
      warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
      Core was generated by `/home/buildbot/maria-slave/power8-vlp01-debug/build/sql/mysqld --defaults-group'.
      Program terminated with signal 6, Aborted.
      #0  0x00000fff8ab03a40 in .pthread_kill () from /opt/at7.0/lib64/power7/libpthread.so.0
      #0  0x00000fff8ab03a40 in .pthread_kill () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x0000000010d251d0 in .my_write_core ()
      #2  0x000000001050fa48 in .handle_fatal_signal () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/signal_handler.cc:262
      #3  <signal handler called>
      #4  0x00000fff8a26f8f0 in .raise () from /opt/at7.0/lib64/power7/libc.so.6
      #5  0x00000fff8a2718f4 in .abort () from /opt/at7.0/lib64/power7/libc.so.6
      #6  0x0000000010d2bae0 in .safe_mutex_unlock ()
      #7  0x00000fff85e34dc4 in .inline_mysql_mutex_unlock () at /home/buildbot/maria-slave/power8-vlp01-debug/build/include/mysql/psi/mysql_thread.h:761
      #8  0x00000fff85e38984 in ._Z17spider_db_errornoP14st_spider_conn () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_db_conn.cc:671
      #9  0x00000fff85efe8bc in ._ZN15spider_db_mysql6commitEPi () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_db_mysql.cc:2087
      #10 0x00000fff85e3b09c in ._Z16spider_db_commitP14st_spider_conn () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_db_conn.cc:1148
      #11 0x00000fff85e31580 in ._Z13spider_commitP10handlertonP3THDb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_trx.cc:3327
      #12 0x0000000010514548 in ._ZL18commit_one_phase_2P3THDbP9THD_TRANSb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/handler.cc:1527
      #13 0x00000000105143e8 in ._Z19ha_commit_one_phaseP3THDb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/handler.cc:1508
      #14 0x0000000010513c38 in ._Z15ha_commit_transP3THDb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/handler.cc:1385
      #15 0x000000001040f35c in ._Z17trans_commit_stmtP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/transaction.cc:394
      #16 0x00000000102914e8 in ._Z21mysql_execute_commandP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:5126
      #17 0x0000000010294bb0 in ._Z11mysql_parseP3THDPcjP12Parser_state () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:6407
      #18 0x000000001028655c in ._Z16dispatch_command19enum_server_commandP3THDPcj () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:1299
      #19 0x000000001028572c in ._Z10do_commandP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:996
      #20 0x00000000103f3af0 in ._Z24do_handle_one_connectionP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1379
      #21 0x00000000103f3754 in .handle_one_connection () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1293
      #22 0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #23 0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 26 (Thread 27313):
      #0  0x0000008078cd0958 in ?? () from /lib64/libaio.so.1
      #1  0x0000008078cd09bc in .io_getevents () from /lib64/libaio.so.1
      #2  0x000000001092b5e8 in ._ZL20os_aio_linux_collectP14os_aio_array_tmm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5098
      #3  0x000000001092ba6c in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5244
      #4  0x0000000010b71118 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/fil/fil0fil.cc:5662
      #5  0x0000000010a2edb0 in .io_handler_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0start.cc:499
      #6  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #7  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 25 (Thread 27381):
      #0  0x00000fff8ab025a4 in .pthread_cond_timedwait () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x000000001092e64c in ._ZL18os_cond_wait_timedP14pthread_cond_tP15os_fast_mutex_tPK8timespec () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:177
      #2  0x000000001092f220 in ._Z22os_event_wait_time_lowP8os_eventml () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:723
      #3  0x0000000010a27d18 in .srv_error_monitor_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0srv.cc:2241
      #4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 24 (Thread 27697):
      #0  0x00000fff8a334dd0 in .__GI___poll () at ../sysdeps/unix/syscall-template.S:81
      #1  0x0000000010d931fc in .vio_io_wait ()
      #2  0x0000000010d91414 in .vio_socket_io_wait ()
      #3  0x0000000010d91660 in .vio_read ()
      #4  0x000000001019ea48 in ._ZL12my_real_readP6st_netPmc ()
      #5  0x000000001019f4c0 in .my_net_read_packet ()
      #6  0x0000000010285440 in ._Z10do_commandP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:929
      #7  0x00000000103f3af0 in ._Z24do_handle_one_connectionP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1379
      #8  0x00000000103f3754 in .handle_one_connection () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1293
      #9  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #10 0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 23 (Thread 27397):
      #0  0x00000fff8ab07f18 in .__libc_sigwait () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x000000001018c8e4 in .signal_hand () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/mysqld.cc:3343
      #2  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #3  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 22 (Thread 27390):
      #0  0x00000fff8ab02130 in .pthread_cond_wait () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x0000000010d2bef4 in .safe_cond_wait ()
      #2  0x0000000010643920 in .inline_mysql_cond_wait () at /home/buildbot/maria-slave/power8-vlp01-debug/build/include/mysql/psi/mysql_thread.h:1165
      #3  0x0000000010660694 in .binlog_background_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/log.cc:9133
      #4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 21 (Thread 27389):
      #0  0x00000fff8ab025a4 in .pthread_cond_timedwait () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x000000001092e64c in ._ZL18os_cond_wait_timedP14pthread_cond_tP15os_fast_mutex_tPK8timespec () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:177
      #2  0x000000001092f220 in ._Z22os_event_wait_time_lowP8os_eventml () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:723
      #3  0x0000000010a85030 in ._Z19ib_wqueue_timedwaitP11ib_wqueue_tl () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/ut/ut0wqueue.cc:154
      #4  0x0000000010bab2a8 in ._Z19fts_optimize_threadPv () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/fts/fts0opt.cc:2996
      #5  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #6  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 20 (Thread 27714):
      #0  0x00000fff8ab02130 in .pthread_cond_wait () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x0000000010d2bef4 in .safe_cond_wait ()
      #2  0x00000fff85e5f330 in .inline_mysql_cond_wait () at /home/buildbot/maria-slave/power8-vlp01-debug/build/include/mysql/psi/mysql_thread.h:1165
      #3  0x00000fff85e6656c in ._Z21spider_bg_conn_actionPv () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_conn.cc:2282
      #4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 19 (Thread 27387):
      #0  0x00000fff8ab02130 in .pthread_cond_wait () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x000000001092e7a8 in ._ZL12os_cond_waitP14pthread_cond_tP15os_fast_mutex_t () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:214
      #2  0x000000001092efec in ._Z17os_event_wait_lowP8os_eventl () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:610
      #3  0x0000000010aefae4 in .buf_dump_thread ()
      #4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 18 (Thread 27308):
      #0  0x00000fff8a334dd0 in .__GI___poll () at ../sysdeps/unix/syscall-template.S:81
      #1  0x00000000101931dc in ._Z26handle_connections_socketsv () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/mysqld.cc:6103
      #2  0x0000000010191f04 in ._Z11mysqld_mainiPPc () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/mysqld.cc:5535
      #3  0x0000000010184208 in .main ()
       
      Thread 17 (Thread 27710):
      #0  0x00000fff8ab02130 in .pthread_cond_wait () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x0000000010d2bef4 in .safe_cond_wait ()
      #2  0x00000fff85e5f330 in .inline_mysql_cond_wait () at /home/buildbot/maria-slave/power8-vlp01-debug/build/include/mysql/psi/mysql_thread.h:1165
      #3  0x00000fff85e6656c in ._Z21spider_bg_conn_actionPv () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_conn.cc:2282
      #4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 16 (Thread 27317):
      #0  0x0000008078cd0958 in ?? () from /lib64/libaio.so.1
      #1  0x0000008078cd09bc in .io_getevents () from /lib64/libaio.so.1
      #2  0x000000001092b5e8 in ._ZL20os_aio_linux_collectP14os_aio_array_tmm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5098
      #3  0x000000001092ba6c in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5244
      #4  0x0000000010b71118 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/fil/fil0fil.cc:5662
      #5  0x0000000010a2edb0 in .io_handler_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0start.cc:499
      #6  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #7  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 15 (Thread 27383):
      #0  0x00000fff8a337784 in .__select () at ../sysdeps/unix/syscall-template.S:81
      #1  0x00000000109302f8 in ._Z15os_thread_sleepm ()
      #2  0x0000000010a29b44 in ._ZL16srv_master_sleepv () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0srv.cc:2969
      #3  0x0000000010a29d5c in .srv_master_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0srv.cc:3017
      #4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 14 (Thread 27388):
      #0  0x00000fff8ab025a4 in .pthread_cond_timedwait () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x000000001092e64c in ._ZL18os_cond_wait_timedP14pthread_cond_tP15os_fast_mutex_tPK8timespec () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:177
      #2  0x000000001092f220 in ._Z22os_event_wait_time_lowP8os_eventml () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:723
      #3  0x0000000010b59d24 in .dict_stats_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/dict/dict0stats_bg.cc:349
      #4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 13 (Thread 27386):
      #0  0x00000fff8a337784 in .__select () at ../sysdeps/unix/syscall-template.S:81
      #1  0x00000000109302f8 in ._Z15os_thread_sleepm ()
      #2  0x0000000010afa280 in ._ZL28page_cleaner_sleep_if_neededm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/buf/buf0flu.cc:2578
      #3  0x0000000010afacb8 in .buf_flush_lru_manager_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/buf/buf0flu.cc:2849
      #4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 12 (Thread 27384):
      #0  0x00000fff8ab02130 in .pthread_cond_wait () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x000000001092e7a8 in ._ZL12os_cond_waitP14pthread_cond_tP15os_fast_mutex_t () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:214
      #2  0x000000001092efec in ._Z17os_event_wait_lowP8os_eventl () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:610
      #3  0x0000000010a2ab00 in ._ZL29srv_purge_coordinator_suspendP10srv_slot_tm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0srv.cc:3326
      #4  0x0000000010a2b1e8 in .srv_purge_coordinator_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0srv.cc:3448
      #5  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #6  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 11 (Thread 27385):
      #0  0x00000fff8a337784 in .__select () at ../sysdeps/unix/syscall-template.S:81
      #1  0x00000000109302f8 in ._Z15os_thread_sleepm ()
      #2  0x0000000010afa280 in ._ZL28page_cleaner_sleep_if_neededm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/buf/buf0flu.cc:2578
      #3  0x0000000010afa698 in .buf_flush_page_cleaner_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/buf/buf0flu.cc:2697
      #4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 10 (Thread 27312):
      #0  0x0000008078cd0958 in ?? () from /lib64/libaio.so.1
      #1  0x0000008078cd09bc in .io_getevents () from /lib64/libaio.so.1
      #2  0x000000001092b5e8 in ._ZL20os_aio_linux_collectP14os_aio_array_tmm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5098
      #3  0x000000001092ba6c in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5244
      #4  0x0000000010b71118 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/fil/fil0fil.cc:5662
      #5  0x0000000010a2edb0 in .io_handler_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0start.cc:499
      #6  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #7  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 9 (Thread 27713):
      #0  0x00000fff8ab02130 in .pthread_cond_wait () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x0000000010d2bef4 in .safe_cond_wait ()
      #2  0x00000fff85e5f330 in .inline_mysql_cond_wait () at /home/buildbot/maria-slave/power8-vlp01-debug/build/include/mysql/psi/mysql_thread.h:1165
      #3  0x00000fff85e6656c in ._Z21spider_bg_conn_actionPv () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_conn.cc:2282
      #4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 8 (Thread 27314):
      #0  0x0000008078cd0958 in ?? () from /lib64/libaio.so.1
      #1  0x0000008078cd09bc in .io_getevents () from /lib64/libaio.so.1
      #2  0x000000001092b5e8 in ._ZL20os_aio_linux_collectP14os_aio_array_tmm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5098
      #3  0x000000001092ba6c in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5244
      #4  0x0000000010b71118 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/fil/fil0fil.cc:5662
      #5  0x0000000010a2edb0 in .io_handler_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0start.cc:499
      #6  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #7  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 7 (Thread 27380):
      #0  0x00000fff8ab025a4 in .pthread_cond_timedwait () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x000000001092e64c in ._ZL18os_cond_wait_timedP14pthread_cond_tP15os_fast_mutex_tPK8timespec () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:177
      #2  0x000000001092f220 in ._Z22os_event_wait_time_lowP8os_eventml () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:723
      #3  0x00000000108f509c in .lock_wait_timeout_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/lock/lock0wait.cc:499
      #4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 6 (Thread 27315):
      #0  0x0000008078cd0958 in ?? () from /lib64/libaio.so.1
      #1  0x0000008078cd09bc in .io_getevents () from /lib64/libaio.so.1
      #2  0x000000001092b5e8 in ._ZL20os_aio_linux_collectP14os_aio_array_tmm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5098
      #3  0x000000001092ba6c in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5244
      #4  0x0000000010b71118 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/fil/fil0fil.cc:5662
      #5  0x0000000010a2edb0 in .io_handler_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0start.cc:499
      #6  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #7  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 5 (Thread 27708):
      #0  0x00000fff8ab025a4 in .pthread_cond_timedwait () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x0000000010d2c238 in .safe_cond_timedwait ()
      #2  0x0000000010643aa0 in .inline_mysql_cond_timedwait () at /home/buildbot/maria-slave/power8-vlp01-debug/build/include/mysql/psi/mysql_thread.h:1202
      #3  0x000000001065bf98 in ._ZN13MYSQL_BIN_LOG23wait_for_update_bin_logEP3THDPK8timespec () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/log.cc:7636
      #4  0x00000000102c51f8 in ._Z17mysql_binlog_sendP3THDPcyt () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_repl.cc:2547
      #5  0x0000000010287194 in ._Z16dispatch_command19enum_server_commandP3THDPcj () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:1524
      #6  0x000000001028572c in ._Z10do_commandP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:996
      #7  0x00000000103f3af0 in ._Z24do_handle_one_connectionP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1379
      #8  0x00000000103f3754 in .handle_one_connection () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1293
      #9  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #10 0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 4 (Thread 27382):
      #0  0x00000fff8ab025a4 in .pthread_cond_timedwait () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x000000001092e64c in ._ZL18os_cond_wait_timedP14pthread_cond_tP15os_fast_mutex_tPK8timespec () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:177
      #2  0x000000001092f220 in ._Z22os_event_wait_time_lowP8os_eventml () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0sync.cc:723
      #3  0x0000000010a27320 in .srv_monitor_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0srv.cc:1977
      #4  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #5  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 3 (Thread 27316):
      #0  0x0000008078cd0958 in ?? () from /lib64/libaio.so.1
      #1  0x0000008078cd09bc in .io_getevents () from /lib64/libaio.so.1
      #2  0x000000001092b5e8 in ._ZL20os_aio_linux_collectP14os_aio_array_tmm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5098
      #3  0x000000001092ba6c in ._Z19os_aio_linux_handlemPP10fil_node_tPPvPmS4_ () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/os/os0file.cc:5244
      #4  0x0000000010b71118 in ._Z12fil_aio_waitm () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/fil/fil0fil.cc:5662
      #5  0x0000000010a2edb0 in .io_handler_thread () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/xtradb/srv/srv0start.cc:499
      #6  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #7  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 2 (Thread 27310):
      #0  0x00000fff8ab025a4 in .pthread_cond_timedwait () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x0000000010d2c238 in .safe_cond_timedwait ()
      #2  0x00000000107c5598 in .inline_mysql_cond_timedwait ()
      #3  0x00000000107c5bb8 in .my_service_thread_sleep ()
      #4  0x00000000107b63dc in .ma_checkpoint_background ()
      #5  0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #6  0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Thread 1 (Thread 27698):
      #0  0x00000fff8ab03a40 in .pthread_kill () from /opt/at7.0/lib64/power7/libpthread.so.0
      #1  0x0000000010d251d0 in .my_write_core ()
      #2  0x000000001050fa48 in .handle_fatal_signal () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/signal_handler.cc:262
      #3  <signal handler called>
      #4  0x00000fff8a26f8f0 in .raise () from /opt/at7.0/lib64/power7/libc.so.6
      #5  0x00000fff8a2718f4 in .abort () from /opt/at7.0/lib64/power7/libc.so.6
      #6  0x0000000010d2bae0 in .safe_mutex_unlock ()
      #7  0x00000fff85e34dc4 in .inline_mysql_mutex_unlock () at /home/buildbot/maria-slave/power8-vlp01-debug/build/include/mysql/psi/mysql_thread.h:761
      #8  0x00000fff85e38984 in ._Z17spider_db_errornoP14st_spider_conn () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_db_conn.cc:671
      #9  0x00000fff85efe8bc in ._ZN15spider_db_mysql6commitEPi () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_db_mysql.cc:2087
      #10 0x00000fff85e3b09c in ._Z16spider_db_commitP14st_spider_conn () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_db_conn.cc:1148
      #11 0x00000fff85e31580 in ._Z13spider_commitP10handlertonP3THDb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/storage/spider/spd_trx.cc:3327
      #12 0x0000000010514548 in ._ZL18commit_one_phase_2P3THDbP9THD_TRANSb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/handler.cc:1527
      #13 0x00000000105143e8 in ._Z19ha_commit_one_phaseP3THDb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/handler.cc:1508
      #14 0x0000000010513c38 in ._Z15ha_commit_transP3THDb () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/handler.cc:1385
      #15 0x000000001040f35c in ._Z17trans_commit_stmtP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/transaction.cc:394
      #16 0x00000000102914e8 in ._Z21mysql_execute_commandP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:5126
      #17 0x0000000010294bb0 in ._Z11mysql_parseP3THDPcjP12Parser_state () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:6407
      #18 0x000000001028655c in ._Z16dispatch_command19enum_server_commandP3THDPcj () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:1299
      #19 0x000000001028572c in ._Z10do_commandP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_parse.cc:996
      #20 0x00000000103f3af0 in ._Z24do_handle_one_connectionP3THD () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1379
      #21 0x00000000103f3754 in .handle_one_connection () at /home/buildbot/maria-slave/power8-vlp01-debug/build/sql/sql_connect.cc:1293
      #22 0x00000fff8aafc548 in .start_thread () from /opt/at7.0/lib64/power7/libpthread.so.0
      #23 0x00000fff8a343c90 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:106
       
      Compressed file /home/buildbot/maria-slave/power8-vlp01-debug/build/mysql-test/var/log/bg.spider_fixes/mysqld.1.1/data/core.27308

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Kentoku Kentoku Shiba
                Reporter:
                elenst Elena Stepanova
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: