Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-23578

galera_3nodes.galera_ist_gcache_rollover MTR failed: WSREP: exception caused by message: {v=0,t=1,ut=255,o=4,s=0,sr=0,as=1,f=6,src=50524cfe,srcvid=view_id(REG,50524cfe,4),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=75,nl=(}

    XMLWordPrintable

    Details

      Description

      galera_3nodes.galera_ist_gcache_rollover failed on BB10.2: WSREP: exception caused by message:

      {v=0,t=1,ut=255,o=4,s=0,sr=0,as=1,f=6,src=50524cfe,srcvid=view_id(REG,50524cfe,4),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=75,nl=(}

      stdio.log:

      10.2.34 8cf8ad86d4b6f3479d80f3d8e8c2bcf463966924

      galera_3nodes.galera_ist_gcache_rollover 'innodb' w1 [ fail ]
              Test ended at 2020-08-25 11:48:06
       
      CURRENT_TEST: galera_3nodes.galera_ipv6_xtrabackup-v2
      CURRENT_TEST: galera_3nodes.galera_ipv6_xtrabackup-v2
      CURRENT_TEST: galera_3nodes.galera_ist_gcache_rollover
       
       
      Failed to start mysqld.3
       
       
       - skipping '/dev/shm/var/1/log/galera_3nodes.galera_ist_gcache_rollover-innodb/'
       
      Retrying test galera_3nodes.galera_ist_gcache_rollover, attempt(2/3)...
       
      ***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ist_gcache_rollover
       
      2020-08-25 11:48:04 3054057280 [ERROR] WSREP: caught exception in PC, state dump to stderr follows:
      2020-08-25 11:48:04 3054057280 [ERROR] WSREP: exception caused by message: {v=0,t=1,ut=255,o=4,s=0,sr=0,as=1,f=6,src=50524cfe,srcvid=view_id(REG,50524cfe,4),insvid=view_id(UNKNOWN,00000000,0),ru=00000000,r=[-1,-1],fs=75,nl=(
       }2020-08-25 11:48:04 3054057280 [ERROR] WSREP: failed to open gcomm backend connection: 131: 505b83f7 last prims not consistent (FATAL)
      2020-08-25 11:48:04 3054057280 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():209: Failed to open backend connection: -131 (State not recoverable)
      2020-08-25 11:48:04 3054057280 [ERROR] WSREP: gcs connect failed: State not recoverable
      2020-08-25 11:48:04 3054057280 [ERROR] Aborting
      

      On BB 10.3, BB 10.4 and BB 10.5 it failed with another output:

      stdio.log:

      10.3.25 95831888e8a89a4e141e76d51dbfc0701552c824

      galera_3nodes.galera_ist_gcache_rollover 'innodb' w2 [ fail ]
              Test ended at 2020-08-25 17:51:09
       
      CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
      CURRENT_TEST: galera_3nodes.galera_ipv6_rsync_section
      CURRENT_TEST: galera_3nodes.galera_ist_gcache_rollover
       
       
      Failed to start mysqld.3
       
       
      worker[2] > Restart  - using different config file
      worker[2] > Restart  - using different config file
      worker[2] > Restart  - using different config file
       - skipping '/dev/shm/var/2/log/galera_3nodes.galera_ist_gcache_rollover-innodb/'
       
      Retrying test galera_3nodes.galera_ist_gcache_rollover, attempt(2/3)...
       
      ***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ist_gcache_rollover
       
      WSREP_SST: [ERROR] rsync daemon port '16031' has been taken (20200825 17:50:25.449)
      2020-08-25 17:50:25 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '32484' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
      2020-08-25 17:50:25 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '32484' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
      2020-08-25 17:50:25 2 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
      2020-08-25 17:50:25 2 [ERROR] Aborting
      Warning: Memory not freed: 1381136
       
      worker[2] mysql-test-run: WARNING: Waited 60 seconds for /dev/shm/var/2/run/mysqld.3.pid to be created, still waiting for 120 seconds...
      worker[2] mysql-test-run: WARNING: Waited 120 seconds for /dev/shm/var/2/run/mysqld.3.pid to be created, still waiting for 60 seconds...
      worker[2] mysql-test-run: WARNING: Timeout after mysql-test-run waited 180 seconds for the process [mysqld.3 - pid: 979, winpid: 979] to create a pid file.
      galera_3nodes.galera_ipv6_rsync_section 'innodb' w2 [ retry-fail ]
              Test ended at 2020-08-25 17:54:14
       
      CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
      CURRENT_TEST: galera_3nodes.galera_ipv6_rsync_section
      CURRENT_TEST: galera_3nodes.galera_ist_gcache_rollover
      CURRENT_TEST: galera_3nodes.galera_ipv6_rsync_section
       
       
      Failed to start mysqld.3
       
       
      worker[2] > Restart  - using different config file
      worker[2] > Restart  - using different config file
      worker[2] > Restart  - using different config file
       - skipping '/dev/shm/var/2/log/galera_3nodes.galera_ipv6_rsync_section-innodb/'
       
      Test galera_3nodes.galera_ipv6_rsync_section has failed 2 times, no more retries!
       
      ***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ipv6_rsync_section
       
      WSREP_SST: [ERROR] rsync daemon port '16031' has been taken (20200825 17:51:11.334)
      2020-08-25 17:51:11 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '980' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --bind-address=:: --core-file --loose-debug-sync-timeout=300
      2020-08-25 17:51:11 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '980' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --bind-address=:: --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
      2020-08-25 17:51:11 1 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
      2020-08-25 17:51:11 1 [ERROR] Aborting
       
      wsrep.mysql_tzinfo_to_sql_symlink 'innodb' w2 [ pass ]     34
      worker[2] > Restart [mysqld.1 - pid: 1307, winpid: 1307] - using different config file
      galera_3nodes.galera_ist_gcache_rollover 'innodb' w2 [ skipped ]  debug_sync is not available
       
      Retrying test galera_3nodes.galera_ist_gcache_rollover, attempt(3/3)...
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jplindst Jan Lindström
              Reporter:
              stepan.patryshev Stepan Patryshev (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:

                  Git Integration