Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24888

galera.galera_nopk_unicode MTR failed: Crash with SIGABRT

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 10.2.37
    • Fix Version/s: 10.2
    • Component/s: Galera, Tests
    • Labels:
      None
    • Environment:
      kvm-deb-bionic-aarch64

      Description

      galera.galera_nopk_unicode failed on BB 10.2 CS: Crash with SIGABRT.
      It seems to be a sporadic issue.

      stdio.log:

      10.2.37, cc2d6d1bb26cdcd649357ec23cd54dd28ddefaf3, kvm-deb-bionic-aarch64

      galera.galera_nopk_unicode 'innodb'      w1 [ fail ]  timeout after 900 seconds
              Test ended at 2021-01-25 06:41:47
       
      Test case timeout after 900 seconds
       
      == /dev/shm/var/1/log/galera_nopk_unicode.log == 
      f1 VARCHAR(255),
      KEY (f1)
      ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
      INSERT INTO t1 VALUES ('текст');
      connection node_2;
      SELECT f1 = 'текст' FROM t1;
      f1 = 'текст'
      1
      connection node_1;
      SET AUTOCOMMIT=OFF;
      START TRANSACTION;
      UPDATE t1 SET f1 = 'текст2';
      connection node_2;
      SET AUTOCOMMIT=OFF;
      START TRANSACTION;
      UPDATE t1 SET f1 = 'текст3';
      connection node_1;
      COMMIT;
      connection node_2;
      COMMIT;
       
       == /dev/shm/var/1/tmp/analyze-timeout-mysqld.2.err ==
      mysqltest: Could not open connection 'default': 2013 Lost connection to MySQL server at 'handshake: reading initial communication packet', system error: 104
       
      The server [mysqld.2 - pid: 9130, winpid: 9130, exit: 256] crashed while running 'analyze-timeout'
      Server log from this test:
      ----------SERVER LOG START-----------
      2021-01-25  6:30:19 281472334283072 [Warning] InnoDB: A long semaphore wait:
      --Thread 281472728461632 has waited at ha_innodb.cc line 19524 for 241.00 seconds the semaphore:
      Mutex at 0xaaab1c1f0180, Mutex LOCK_SYS created lock0lock.cc:469, lock var 2
       
      2021-01-25  6:30:19 281472334283072 [Note] InnoDB: A semaphore wait:
      --Thread 281472728461632 has waited at ha_innodb.cc line 19524 for 241.00 seconds the semaphore:
      Mutex at 0xaaab1c1f0180, Mutex LOCK_SYS created lock0lock.cc:469, lock var 2
       
      InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
      InnoDB: Pending reads 0, writes 0
       
      =====================================
      2021-01-25 06:30:27 0xffff617fa140 INNODB MONITOR OUTPUT
      =====================================
      Per second averages calculated from the last 20 seconds
      -----------------
      BACKGROUND THREAD
      -----------------
      srv_master_thread loops: 14 srv_active, 0 srv_shutdown, 265 srv_idle
      srv_master_thread log flush and writes: 279
      ----------
      SEMAPHORES
      ----------
      OS WAIT ARRAY INFO: reservation count 215
      --Thread 281472728461632 has waited at ha_innodb.cc line 19524 for 249.00 seconds the semaphore:
      Mutex at 0xaaab1c1f0180, Mutex LOCK_SYS created lock0lock.cc:469, lock var 2
       
      OS WAIT ARRAY INFO: signal count 137
      RW-shared spins 53, rounds 455, OS waits 11
      RW-excl spins 10, rounds 187, OS waits 6
      RW-sx spins 2, rounds 60, OS waits 2
      Spin rounds per wait: 8.58 RW-shared, 18.70 RW-excl, 30.00 RW-sx
      FAIL TO OBTAIN LOCK MUTEX, SKIP LOCK INFO PRINTING
      --------
      FILE I/O
      --------
      I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
      I/O thread 1 state: waiting for completed aio requests (log thread)
      I/O thread 2 state: waiting for completed aio requests (read thread)
      I/O thread 3 state: waiting for completed aio requests (read thread)
      I/O thread 4 state: waiting for completed aio requests (write thread)
      I/O thread 5 state: waiting for completed aio requests (write thread)
      Pending normal aio reads: [0, 0] , aio writes: [0, 0] ,
       ibuf aio reads:, log i/o's:, sync i/o's:
      Pending flushes (fsync) log: 0; buffer pool: 0
      333 OS file reads, 1162 OS file writes, 493 OS fsyncs
      0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
       
      [...]
       
      ----------------------------
      END OF INNODB MONITOR OUTPUT
      ============================
      InnoDB: ###### Diagnostic info printed to the standard error stream
      2021-01-25  6:41:10 281472334283072 [Warning] InnoDB: A long semaphore wait:
      --Thread 281472728461632 has waited at ha_innodb.cc line 19524 for 892.00 seconds the semaphore:
      Mutex at 0xaaab1c1f0180, Mutex LOCK_SYS created lock0lock.cc:469, lock var 2
       
      2021-01-25  6:41:10 281472334283072 [Note] InnoDB: A semaphore wait:
      --Thread 281472728461632 has waited at ha_innodb.cc line 19524 for 892.00 seconds the semaphore:
      Mutex at 0xaaab1c1f0180, Mutex LOCK_SYS created lock0lock.cc:469, lock var 2
       
      InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
      InnoDB: Pending reads 0, writes 0
      2021-01-25  6:41:21 281472972968256 [Note] WSREP: (e9cbaa36, 'tcp://0.0.0.0:16005') connection to peer e9910e27 with addr tcp://127.0.0.1:16002 timed out, no messages seen in PT3S, socket stats: rtt: 8444 rttvar: 14188 rto: 212000 lost: 0 last_data_recv: 3320 cwnd: 10 last_queued_since: 320401694 last_delivered_since: 3320480146 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
      2021-01-25  6:41:21 281472972968256 [Note] WSREP: (e9cbaa36, 'tcp://0.0.0.0:16005') turning message relay requesting on, nonlive peers: tcp://127.0.0.1:16002 
      2021-01-25  6:41:22 281472972968256 [Note] WSREP: (e9cbaa36, 'tcp://0.0.0.0:16005') reconnecting to e9910e27 (tcp://127.0.0.1:16002), attempt 0
      2021-01-25  6:41:25 281472972968256 [Note] WSREP: (e9cbaa36, 'tcp://0.0.0.0:16005') connection to peer 00000000 with addr tcp://127.0.0.1:16002 timed out, no messages seen in PT3S, socket stats: rtt: 41 rttvar: 20 rto: 212000 lost: 0 last_data_recv: 3004 cwnd: 10 last_queued_since: 3002020707 last_delivered_since: 3002020707 send_queue_length: 0 send_queue_bytes: 0
      210125  6:41:25 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.2.37-MariaDB-1:10.2.37+maria~bionic-log
      key_buffer_size=1048576
      read_buffer_size=131072
      max_used_connections=2
      max_threads=153
      thread_count=8
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63025 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x49000
      /usr/sbin/mysqld(my_print_stacktrace+0x2c)[0xaaaaddec6a3c]
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /dev/shm/var/1/mysqld.2/data
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        unlimited            unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             23605                23605                processes 
      Max open files            1024                 1024                 files     
      Max locked memory         16777216             16777216             bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       23605                23605                signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: |/usr/share/apport/apport %p %s %c %d %P
       
      ----------SERVER LOG END-------------
       
       - found 'core' (0/0)
       
      Trying 'dbx' to get a backtrace
       
      Trying 'gdb' to get a backtrace from coredump /dev/shm/var/1/log/galera.galera_nopk_unicode-innodb/mysqld.1/data/core
      Core generated by '/usr/sbin/mysqld'
      Output from gdb follows. The first stack trace is from the failing thread.
      The following stack traces are from all threads (so the failing one is
      duplicated).
      --------------------------
      [New LWP 7067]
      [New LWP 7069]
      [New LWP 7070]
      [New LWP 7071]
      [New LWP 7072]
      [New LWP 7073]
      [New LWP 7075]
      [New LWP 7088]
      [New LWP 7078]
      [New LWP 7093]
      [New LWP 7079]
      [New LWP 7100]
      [New LWP 7080]
      [New LWP 7106]
      [New LWP 7086]
      [New LWP 7076]
      [New LWP 7091]
      [New LWP 7068]
      [New LWP 7074]
      [New LWP 7101]
      [New LWP 7084]
      [New LWP 7077]
      [New LWP 7102]
      [New LWP 7085]
      [New LWP 7087]
      [New LWP 7127]
      [New LWP 7092]
      [New LWP 7103]
      [New LWP 7089]
      [New LWP 7090]
      [New LWP 7104]
      [New LWP 7105]
      [New LWP 7628]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
      Core was generated by `/usr/sbin/mysqld --defaults-group-suffix=.1 --defaults-file=/dev/shm/var/1/my.c'.
      Program terminated with signal SIGABRT, Aborted.
      #0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      57	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
      [Current thread is 1 (Thread 0xffff90e28440 (LWP 7067))]
      #0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      #1  0x0000aaaac27950a8 in handle_fatal_signal ()
      #2  <signal handler called>
      #3  0x0000ffff9103ae2c in __GI___poll (fds=0xffffc1e00ca0, nfds=187650381789728, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
      #4  0x0000aaaac2592508 in handle_connections_sockets() ()
      #5  0x0000aaaac2593a20 in mysqld_main(int, char**) ()
      #6  0x0000ffff90f94720 in __libc_start_main (main=0x0, argc=0, argv=0x0, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:310
      #7  0x0000aaaac25868a8 in _start ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)
       
      Thread 33 (Thread 0xffff77338140 (LWP 7628)):
      #0  0x0000ffff9132664c in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0xffff77337808, expected=0, futex_word=0xaaaac3346ff8 <COND_thread_cache+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0xffff77337808, mutex=0xaaaac3344a78 <LOCK_thread_cache>, cond=0xaaaac3346fd0 <COND_thread_cache>) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0xaaaac3346fd0 <COND_thread_cache>, mutex=0xaaaac3344a78 <LOCK_thread_cache>, abstime=0xffff77337808) at pthread_cond_wait.c:667
      #3  0x0000aaaac258a6f0 in one_thread_per_connection_end(THD*, bool) ()
      #4  0x0000aaaac26de330 in do_handle_one_connection(CONNECT*) ()
      #5  0x0000aaaac26de5cc in handle_one_connection ()
      #6  0x0000aaaac2c24fa0 in ?? ()
      #7  0x0000ffff91320088 in start_thread (arg=0xffffc1e008bf) at pthread_create.c:463
      #8  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 32 (Thread 0xffff8407d140 (LWP 7105)):
      #0  0x0000ffff90fa72ec in __GI___sigtimedwait (set=<optimized out>, set@entry=0xffff8407c870, info=info@entry=0xffff8407c738, timeout=0x1, timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:42
      #1  0x0000ffff9132b024 in __sigwait (set=0xffff8407c870, sig=0xffff8407c820) at ../sysdeps/unix/sysv/linux/sigwait.c:28
      #2  0x0000aaaac2589260 in signal_hand ()
      #3  0x0000aaaac2c24fa0 in ?? ()
      #4  0x0000ffff91320088 in start_thread (arg=0xffffc1e00d3f) at pthread_create.c:463
      #5  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 31 (Thread 0xffff5b7fe140 (LWP 7104)):
      #0  0x0000ffff9132664c in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0xffff5b7fd818, expected=0, futex_word=0xaaaac3be96a8) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0xffff5b7fd818, mutex=0xaaaac3be9648, cond=0xaaaac3be9680) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0xaaaac3be9680, mutex=0xaaaac3be9648, abstime=0xffff5b7fd818) at pthread_cond_wait.c:667
      #3  0x0000aaaac2b67968 in ?? ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)
       
      Thread 30 (Thread 0xffff67fff140 (LWP 7090)):
      #0  0x0000ffff913262a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xaaaae07e4b98) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaae07e4b40, cond=0xaaaae07e4b70) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0xaaaae07e4b70, mutex=0xaaaae07e4b40) at pthread_cond_wait.c:655
      #3  0x0000aaaac29621e4 in ?? ()
      #4  0x0000aaaac29e13f0 in ?? ()
      #5  0x0000ffff91320088 in start_thread (arg=0xffffc1df8daf) at pthread_create.c:463
      #6  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 29 (Thread 0xffff6cdf4140 (LWP 7089)):
      #0  0x0000ffff9132664c in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0xffff6cdf35e0, expected=0, futex_word=0xaaaae095d6e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0xffff6cdf35e0, mutex=0xaaaae095d690, cond=0xaaaae095d6c0) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0xaaaae095d6c0, mutex=0xaaaae095d690, abstime=0xffff6cdf35e0) at pthread_cond_wait.c:667
      #3  0x0000aaaac2961a84 in ?? ()
      #4  0x0000aaaac29620c0 in ?? ()
      #5  0x0000aaaac2a2d738 in ?? ()
      #6  0x0000aaaac2afd29c in ?? ()
      #7  0x0000ffff91320088 in start_thread (arg=0xffffc1df8cdf) at pthread_create.c:463
      #8  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 28 (Thread 0xffff5bfff140 (LWP 7103)):
      #0  0x0000ffff913262a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffff5bffe8f8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffff5bffe898, cond=0xffff5bffe8d0) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0xffff5bffe8d0, mutex=0xffff5bffe898) at pthread_cond_wait.c:655
      #3  0x0000aaaac28f1098 in ?? ()
      #4  0x0000aaaac2c24fa0 in ?? ()
      #5  0x0000ffff91320088 in start_thread (arg=0xffffc1e0015f) at pthread_create.c:463
      #6  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 27 (Thread 0xffff66ffd140 (LWP 7092)):
      #0  0x0000ffff913262a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xaaaae07e4c18) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaae07e4bc0, cond=0xaaaae07e4bf0) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0xaaaae07e4bf0, mutex=0xaaaae07e4bc0) at pthread_cond_wait.c:655
      #3  0x0000aaaac29621e4 in ?? ()
      #4  0x0000aaaac29dfd78 in ?? ()
      #5  0x0000ffff91320088 in start_thread (arg=0xffffc1df8daf) at pthread_create.c:463
      #6  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 26 (Thread 0xffff77383140 (LWP 7127)):
      #0  0x0000ffff9132664c in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0xffff77382808, expected=0, futex_word=0xaaaac3346ff8 <COND_thread_cache+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0xffff77382808, mutex=0xaaaac3344a78 <LOCK_thread_cache>, cond=0xaaaac3346fd0 <COND_thread_cache>) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0xaaaac3346fd0 <COND_thread_cache>, mutex=0xaaaac3344a78 <LOCK_thread_cache>, abstime=0xffff77382808) at pthread_cond_wait.c:667
      #3  0x0000aaaac258a6f0 in one_thread_per_connection_end(THD*, bool) ()
      #4  0x0000aaaac26de330 in do_handle_one_connection(CONNECT*) ()
      #5  0x0000aaaac26de5cc in handle_one_connection ()
      #6  0x0000aaaac2c24fa0 in ?? ()
      #7  0x0000ffff91320088 in start_thread (arg=0xffffc1e008bf) at pthread_create.c:463
      #8  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 25 (Thread 0xffff6f5f9140 (LWP 7087)):
      #0  0x0000ffff9132a5fc in __GI___nanosleep (requested_time=0xffff6f5f8850, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
      #1  0x0000aaaac296246c in ?? ()
      #2  0x0000aaaac29df274 in ?? ()
      #3  0x0000ffff91320088 in start_thread (arg=0xffffc1df8daf) at pthread_create.c:463
      #4  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 24 (Thread 0xffff6e5f7140 (LWP 7085)):
      #0  0x0000ffff9132664c in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0xffff6e5f6730, expected=0, futex_word=0xaaaae07e4d98) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0xffff6e5f6730, mutex=0xaaaae07e4d40, cond=0xaaaae07e4d70) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0xaaaae07e4d70, mutex=0xaaaae07e4d40, abstime=0xffff6e5f6730) at pthread_cond_wait.c:667
      #3  0x0000aaaac2961a84 in ?? ()
      #4  0x0000aaaac29620c0 in ?? ()
      #5  0x0000aaaac29de534 in ?? ()
      #6  0x0000ffff91320088 in start_thread (arg=0xffffc1df8daf) at pthread_create.c:463
      #7  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 23 (Thread 0xffff64ff9140 (LWP 7102)):
      #0  0x0000ffff913262a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xaaaae07e4f98) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaae07e4f40, cond=0xaaaae07e4f70) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0xaaaae07e4f70, mutex=0xaaaae07e4f40) at pthread_cond_wait.c:655
      #3  0x0000aaaac29621e4 in ?? ()
      #4  0x0000aaaac2a68a8c in ?? ()
      #5  0x0000ffff91320088 in start_thread (arg=0xffffc1df8daf) at pthread_create.c:463
      #6  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 22 (Thread 0xffff759ca140 (LWP 7077)):
      #0  0x0000ffff917926c0 in ?? () from /lib/aarch64-linux-gnu/libaio.so.1
      #1  0x0000ffff759c96d8 in ?? ()
      Backtrace stopped: previous frame inner to this frame (corrupt stack?)
       
      Thread 21 (Thread 0xffff6edf8140 (LWP 7084)):
      #0  0x0000ffff9132664c in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0xffff6edf77a0, expected=0, futex_word=0xaaaae0900e48) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0xffff6edf77a0, mutex=0xaaaae0900df0, cond=0xaaaae0900e20) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0xaaaae0900e20, mutex=0xaaaae0900df0, abstime=0xffff6edf77a0) at pthread_cond_wait.c:667
      #3  0x0000aaaac2961a84 in ?? ()
      #4  0x0000aaaac29620c0 in ?? ()
      #5  0x0000aaaac293d408 in ?? ()
      #6  0x0000ffff91320088 in start_thread (arg=0xffffc1df8daf) at pthread_create.c:463
      #7  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 20 (Thread 0xffff657fa140 (LWP 7101)):
      #0  0x0000ffff9132a5fc in __GI___nanosleep (requested_time=0xffff657f9210, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
      #1  0x0000aaaac296246c in ?? ()
      #2  0x0000aaaac2a5a9d0 in ?? ()
      #3  0x0000ffff91320088 in start_thread (arg=0xffffc1df8daf) at pthread_create.c:463
      #4  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 19 (Thread 0xffff771cd140 (LWP 7074)):
      #0  0x0000ffff917926c0 in ?? () from /lib/aarch64-linux-gnu/libaio.so.1
      #1  0x0000ffff771cc6d8 in ?? ()
      Backtrace stopped: previous frame inner to this frame (corrupt stack?)
       
      Thread 18 (Thread 0xffff8caa8140 (LWP 7068)):
      #0  0x0000ffff9132664c in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0xffff8caa78c8, expected=0, futex_word=0xaaaac3bf9608) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0xffff8caa78c8, mutex=0xaaaac3bf9570, cond=0xaaaac3bf95e0) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0xaaaac3bf95e0, mutex=0xaaaac3bf9570, abstime=0xffff8caa78c8) at pthread_cond_wait.c:667
      #3  0x0000aaaac2c72e4c in ?? ()
      #4  0x0000aaaac2c72ca8 in ?? ()
      Backtrace stopped: previous frame inner to this frame (corrupt stack?)
       
      Thread 17 (Thread 0xffff677fe140 (LWP 7091)):
      #0  0x0000ffff913262a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xaaaae07e4c9c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaae07e4c40, cond=0xaaaae07e4c70) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0xaaaae07e4c70, mutex=0xaaaae07e4c40) at pthread_cond_wait.c:655
      #3  0x0000aaaac29621e4 in ?? ()
      #4  0x0000aaaac29dfd78 in ?? ()
      #5  0x0000ffff91320088 in start_thread (arg=0xffffc1df8daf) at pthread_create.c:463
      #6  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 16 (Thread 0xffff761cb140 (LWP 7076)):
      #0  0x0000ffff917926c0 in ?? () from /lib/aarch64-linux-gnu/libaio.so.1
      #1  0x0000ffff761ca6d8 in ?? ()
      Backtrace stopped: previous frame inner to this frame (corrupt stack?)
       
      Thread 15 (Thread 0xffff6ddf6140 (LWP 7086)):
      #0  0x0000ffff9132664c in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0xffff6ddf5840, expected=0, futex_word=0xaaaae07e4e18) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0xffff6ddf5840, mutex=0xaaaae07e4dc0, cond=0xaaaae07e4df0) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0xaaaae07e4df0, mutex=0xaaaae07e4dc0, abstime=0xffff6ddf5840) at pthread_cond_wait.c:667
      #3  0x0000aaaac2961a84 in ?? ()
      #4  0x0000aaaac29620c0 in ?? ()
      #5  0x0000aaaac29e1d5c in ?? ()
      #6  0x0000ffff91320088 in start_thread (arg=0xffffc1df8daf) at pthread_create.c:463
      #7  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 14 (Thread 0xffff773ce140 (LWP 7106)):
      #0  0x0000ffff913262a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xaaaac33487a4 <COND_manager+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaac33487b0 <LOCK_manager>, cond=0xaaaac3348778 <COND_manager>) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0xaaaac3348778 <COND_manager>, mutex=0xaaaac33487b0 <LOCK_manager>) at pthread_cond_wait.c:655
      #3  0x0000aaaac2619364 in handle_manager ()
      #4  0x0000aaaac2c24fa0 in ?? ()
      #5  0x0000ffff91320088 in start_thread (arg=0xffffc1e00d0f) at pthread_create.c:463
      #6  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 13 (Thread 0xffff6ffff140 (LWP 7080)):
      #0  0x0000ffff9132664c in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0xffff6fffe640, expected=0, futex_word=0xaaaae07e4f1c) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0xffff6fffe640, mutex=0xaaaae07e4ec0, cond=0xaaaae07e4ef0) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0xaaaae07e4ef0, mutex=0xaaaae07e4ec0, abstime=0xffff6fffe640) at pthread_cond_wait.c:667
      #3  0x0000aaaac2961a84 in ?? ()
      #4  0x0000aaaac29620c0 in ?? ()
      #5  0x0000aaaac2a7a57c in ?? ()
      #6  0x0000ffff91320088 in start_thread (arg=0xffffc1df8daf) at pthread_create.c:463
      #7  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 12 (Thread 0xffff65ffb140 (LWP 7100)):
      #0  0x0000ffff913262a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xaaaae07e4e98) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaae07e4e40, cond=0xaaaae07e4e70) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0xaaaae07e4e70, mutex=0xaaaae07e4e40) at pthread_cond_wait.c:655
      #3  0x0000aaaac29621e4 in ?? ()
      #4  0x0000aaaac2a6f854 in ?? ()
      #5  0x0000ffff91320088 in start_thread (arg=0xffffc1df8daf) at pthread_create.c:463
      #6  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 11 (Thread 0xffff749c8140 (LWP 7079)):
      #0  0x0000ffff917926c0 in ?? () from /lib/aarch64-linux-gnu/libaio.so.1
      #1  0x0000ffff749c76d8 in ?? ()
      Backtrace stopped: previous frame inner to this frame (corrupt stack?)
       
      Thread 10 (Thread 0xffff667fc140 (LWP 7093)):
      #0  0x0000ffff913262a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xaaaae07e4d1c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaae07e4cc0, cond=0xaaaae07e4cf0) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0xaaaae07e4cf0, mutex=0xaaaae07e4cc0) at pthread_cond_wait.c:655
      #3  0x0000aaaac29621e4 in ?? ()
      #4  0x0000aaaac29dfd78 in ?? ()
      #5  0x0000ffff91320088 in start_thread (arg=0xffffc1df8daf) at pthread_create.c:463
      #6  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 9 (Thread 0xffff751c9140 (LWP 7078)):
      #0  0x0000ffff917926c0 in ?? () from /lib/aarch64-linux-gnu/libaio.so.1
      #1  0x0000ffff751c86d8 in ?? ()
      Backtrace stopped: previous frame inner to this frame (corrupt stack?)
       
      Thread 8 (Thread 0xffff6d5f5140 (LWP 7088)):
      #0  0x0000ffff9132664c in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0xffff6d5f47c0, expected=0, futex_word=0xaaaae09125e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  __pthread_cond_wait_common (abstime=0xffff6d5f47c0, mutex=0xaaaae0912590, cond=0xaaaae09125c0) at pthread_cond_wait.c:539
      #2  __pthread_cond_timedwait (cond=0xaaaae09125c0, mutex=0xaaaae0912590, abstime=0xffff6d5f47c0) at pthread_cond_wait.c:667
      #3  0x0000aaaac2961a84 in ?? ()
      #4  0x0000aaaac29620c0 in ?? ()
      #5  0x0000aaaac2aba35c in ?? ()
      #6  0x0000ffff91320088 in start_thread (arg=0xffffc1df8daf) at pthread_create.c:463
      #7  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 7 (Thread 0xffff769cc140 (LWP 7075)):
      #0  0x0000ffff917926c0 in ?? () from /lib/aarch64-linux-gnu/libaio.so.1
      #1  0x0000ffff769cb6d8 in ?? ()
      Backtrace stopped: previous frame inner to this frame (corrupt stack?)
       
      Thread 6 (Thread 0xffff844f7140 (LWP 7073)):
      #0  0x0000ffff913262a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffff8c0660ec) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffff8c066090, cond=0xffff8c0660c0) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=cond@entry=0xffff8c0660c0, mutex=mutex@entry=0xffff8c066090) at pthread_cond_wait.c:655
      #3  0x0000ffff8c22a0b8 in fifo_lock_get (q=0xffff8c066010) at ./galerautils/src/gu_fifo.c:262
      #4  gu_fifo_get_head (q=0xffff8c066010, err=0xffff844f632c, err@entry=0xffff844f637c) at ./galerautils/src/gu_fifo.c:330
      #5  0x0000ffff8c11d9b4 in gcs_recv (conn=0xaaaae0728240, action=action@entry=0xffff844f6390) at ./gcs/src/gcs.cpp:1881
      #6  0x0000ffff8c0f7d28 in galera::Gcs::recv (act=..., this=<optimized out>) at ./galera/src/galera_gcs.hpp:119
      #7  galera::GcsActionSource::process (this=0xaaaae071f738, recv_ctx=0xffff70000c08, exit_loop=@0xffff844f642f: false) at ./galera/src/gcs_action_source.cpp:174
      #8  0x0000ffff8c0cb358 in galera::ReplicatorSMM::async_recv (this=0xaaaae071efb0, recv_ctx=0xffff70000c08) at ./galera/src/replicator_smm.cpp:362
      #9  0x0000ffff8c0a661c in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at ./galera/src/wsrep_provider.cpp:244
      #10 0x0000aaaac2744f14 in ?? ()
      #11 0x0000aaaac2736d34 in start_wsrep_THD ()
      #12 0x0000aaaac2c24fa0 in ?? ()
      #13 0x0000ffff91320088 in start_thread (arg=0xffffc1e0090f) at pthread_create.c:463
      #14 0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 5 (Thread 0xffff8c065140 (LWP 7072)):
      #0  0x0000ffff913262a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xaaaac3369614 <COND_wsrep_rollback+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaac3369620 <LOCK_wsrep_rollback>, cond=0xaaaac33695e8 <COND_wsrep_rollback>) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0xaaaac33695e8 <COND_wsrep_rollback>, mutex=0xaaaac3369620 <LOCK_wsrep_rollback>) at pthread_cond_wait.c:655
      #3  0x0000aaaac2745670 in ?? ()
      #4  0x0000aaaac2736d34 in start_wsrep_THD ()
      #5  0x0000aaaac2c24fa0 in ?? ()
      #6  0x0000ffff91320088 in start_thread (arg=0xffffc1e0093f) at pthread_create.c:463
      #7  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 4 (Thread 0xffff84cf8140 (LWP 7071)):
      #0  0x0000ffff913262a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xaaaae0779a8c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaae0779a30, cond=0xaaaae0779a60) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=cond@entry=0xaaaae0779a60, mutex=0xaaaae0779a30) at pthread_cond_wait.c:655
      #3  0x0000ffff8c12c6c4 in gu::Lock::wait (this=0xffff84cf6990, cond=...) at ./galerautils/src/gu_lock.hpp:59
      #4  RecvBuf::front (timeout=..., this=0xffff84cf6ca0) at ./gcs/src/gcs_gcomm.cpp:114
      #5  gcomm_recv (backend=<optimized out>, msg=0xaaaae0728498, timeout=9223372035999999999) at ./gcs/src/gcs_gcomm.cpp:608
      #6  0x0000ffff8c123a8c in core_msg_recv (timeout=9223372035999999999, recv_msg=0xaaaae0728498, backend=0xaaaae07285c0) at ./gcs/src/gcs_core.cpp:469
      #7  gcs_core_recv (conn=0xaaaae0728430, recv_act=recv_act@entry=0xffff84cf78d8, timeout=9223372035999999999) at ./gcs/src/gcs_core.cpp:1102
      #8  0x0000ffff8c11e2b0 in gcs_recv_thread (arg=0xaaaae0728240) at ./gcs/src/gcs.cpp:1277
      #9  0x0000ffff91320088 in start_thread (arg=0xffffc1e004ff) at pthread_create.c:463
      #10 0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 3 (Thread 0xffff854f9140 (LWP 7070)):
      #0  0x0000ffff91044110 in __GI_epoll_pwait (epfd=<optimized out>, events=events@entry=0xffff854f7d68, maxevents=maxevents@entry=128, timeout=-2058385832, set=set@entry=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:42
      #1  0x0000ffff91044260 in epoll_wait (epfd=<optimized out>, events=events@entry=0xffff854f7d68, maxevents=maxevents@entry=128, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/epoll_wait.c:32
      #2  0x0000ffff8c101450 in asio::detail::epoll_reactor::run (this=0xaaaae0778c70, block=block@entry=true, ops=...) at ./asio/asio/detail/impl/epoll_reactor.ipp:391
      #3  0x0000ffff8c149728 in asio::detail::task_io_service::do_run_one (ec=..., this_thread=..., lock=..., this=0xaaaae0776a30) at ./asio/asio/detail/impl/task_io_service.ipp:355
      #4  asio::detail::task_io_service::run (ec=..., this=0xaaaae0776a30) at ./asio/asio/detail/impl/task_io_service.ipp:148
      #5  asio::io_service::run (this=<optimized out>) at ./asio/asio/impl/io_service.ipp:58
      #6  gcomm::AsioProtonet::event_loop (this=<optimized out>, period=...) at ./gcomm/src/asio_protonet.cpp:116
      #7  0x0000ffff8c12ed78 in GCommConn::run (this=0xaaaae0779830) at ./gcs/src/gcs_gcomm.cpp:433
      #8  0x0000ffff8c13747c in GCommConn::run_fn (arg=<optimized out>) at ./gcs/src/gcs_gcomm.cpp:174
      #9  0x0000ffff91320088 in start_thread (arg=0xffffc1dffd3f) at pthread_create.c:463
      #10 0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 2 (Thread 0xffff874fd140 (LWP 7069)):
      #0  0x0000ffff913262a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xaaaae071f664) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaae071f608, cond=0xaaaae071f638) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=cond@entry=0xaaaae071f638, mutex=0xaaaae071f608) at pthread_cond_wait.c:655
      #3  0x0000ffff8c0ef7ec in gu::Lock::wait (this=0xffff874fc820, cond=...) at ./galerautils/src/gu_lock.hpp:59
      #4  galera::ServiceThd::thd_func (arg=0xaaaae071f5f0) at ./galera/src/galera_service_thd.cpp:30
      #5  0x0000ffff91320088 in start_thread (arg=0xffffc1dfff5f) at pthread_create.c:463
      #6  0x0000ffff91043ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
       
      Thread 1 (Thread 0xffff90e28440 (LWP 7067)):
      #0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
      #1  0x0000aaaac27950a8 in handle_fatal_signal ()
      #2  <signal handler called>
      #3  0x0000ffff9103ae2c in __GI___poll (fds=0xffffc1e00ca0, nfds=187650381789728, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
      #4  0x0000aaaac2592508 in handle_connections_sockets() ()
      #5  0x0000aaaac2593a20 in mysqld_main(int, char**) ()
      #6  0x0000ffff90f94720 in __libc_start_main (main=0x0, argc=0, argv=0x0, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:310
      #7  0x0000aaaac25868a8 in _start ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)
       
       - deleting it, already saved 0
       - saving '/dev/shm/var/1/log/galera.galera_nopk_unicode-innodb/' to '/dev/shm/var/log/galera.galera_nopk_unicode-innodb/'
       
      Retrying test galera.galera_nopk_unicode, attempt(2/3)...
       
      worker[1] > Restart  - not started
      worker[1] > Restart  - not started
      ***Warnings generated in error logs during shutdown after running tests: galera.galera_lock_table galera.galera_nopk_large_varchar galera.galera_myisam_autocommit galera.galera_mdev_10812 galera.galera_kill_nochanges galera.galera_nopk_bit galera.galera_myisam_transactions galera.galera_kill_ddl galera.galera_many_indexes galera.galera_nopk_unicode galera.galera_kill_smallchanges galera.galera_multi_database galera.galera_nopk_blob
       
      Attempting backtrace. You can use the following information to find out
      2021-01-25  6:30:19 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:30:50 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:31:21 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:31:52 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:32:23 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:32:54 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:33:25 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:33:56 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:34:27 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:34:58 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:35:29 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:36:00 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:36:31 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:37:02 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:37:33 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:38:04 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:38:35 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:39:06 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:39:37 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:40:08 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:40:39 281472334283072 [Warning] InnoDB: A long semaphore wait:
      2021-01-25  6:41:10 281472334283072 [Warning] InnoDB: A long semaphore wait:
      Attempting backtrace. You can use the following information to find out
       
      galera.galera_nopk_unicode 'innodb'      w1 [ retry-pass ]   2072
       
      Retrying test galera.galera_nopk_unicode, attempt(3/3)...
       
      galera.galera_nopk_unicode 'innodb'      w1 [ retry-pass ]   2038
      

      Server logs.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jplindst Jan Lindström
              Reporter:
              stepan.patryshev Stepan Patryshev (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:

                  Git Integration