[MDEV-11208] galera_sst_xtrabackup-v2: assertion failure: global_status_var.global_memory_used == 0 - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Cannot Reproduce
Affects Version/s: 10.2.2
Fix Version/s: N/A
Component/s: Galera, Galera SST, Tests, wsrep
Labels:
- need_feedback

Sprint:
10.2.7-1

Description

$ perl mtr galera_sst_xtrabackup-v2

...

#5  0x00007f59c885d01a in __GI_abort () at abort.c:89

#6  0x00007f59c8853bd7 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x556ca6844940 "global_status_var.global_memory_used == 0", file=file@entry=0x556ca6844670 "/home/nirbhay/project/git-repo/mariadb/10.2/sql/mysqld.cc", line=line@entry=2152, function=function@entry=0x556ca684c6c0 <mysqld_exit(int)::__PRETTY_FUNCTION__> "void mysqld_exit(int)") at assert.c:92

#7  0x00007f59c8853c82 in __GI___assert_fail (assertion=0x556ca6844940 "global_status_var.global_memory_used == 0", file=0x556ca6844670 "/home/nirbhay/project/git-repo/mariadb/10.2/sql/mysqld.cc", line=2152, function=0x556ca684c6c0 <mysqld_exit(int)::__PRETTY_FUNCTION__> "void mysqld_exit(int)") at assert.c:101

#8  0x0000556ca5ccd450 in mysqld_exit (exit_code=1) at /home/nirbhay/project/git-repo/mariadb/10.2/sql/mysqld.cc:2152

#9  0x0000556ca5ccd359 in unireg_abort (exit_code=1) at /home/nirbhay/project/git-repo/mariadb/10.2/sql/mysqld.cc:2120

#10 0x0000556ca5f71aeb in wsrep_init_startup (first=true) at /home/nirbhay/project/git-repo/mariadb/10.2/sql/wsrep_mysqld.cc:830

#11 0x0000556ca5cd37a2 in init_server_components () at /home/nirbhay/project/git-repo/mariadb/10.2/sql/mysqld.cc:5226

#12 0x0000556ca5cd4ac7 in mysqld_main (argc=151, argv=0x7f59c80720c0) at /home/nirbhay/project/git-repo/mariadb/10.2/sql/mysqld.cc:5882

#13 0x0000556ca5cc9ae0 in main (argc=18, argv=0x7ffdc0cb6108) at /home/nirbhay/project/git-repo/mariadb/10.2/sql/main.cc:25

Attachments

Issue Links

relates to

MDEV-12445 Rocksdb does not shutdown worker threads and aborts in memleak check on server shutdown

Open

Activity

Ascending order - Click to sort in descending order

Nirbhay Choubey (Inactive) created issue - 2016-11-01 21:55

Sergei Golubchik made changes - 2017-02-25 21:34

Field	Original Value	New Value
Assignee	Nirbhay Choubey [ nirbhay_c ]

Sergei Golubchik made changes - 2017-04-04 13:57

Labels

galera

Sachin Setiya (Inactive) made changes - 2017-04-18 10:16

Assignee

Sachin Setiya [ sachin.setiya.007 ]

Sergei Golubchik made changes - 2017-04-18 10:29

Fix Version/s

10.2 [ 14601 ]

Rasmus Johansson (Inactive) made changes - 2017-06-22 05:46

Sprint

10.2.7-1 [ 171 ]

Sachin Setiya (Inactive) made changes - 2017-06-24 06:50

Status

Open [ 1 ]

In Progress [ 3 ]

Sachin Setiya (Inactive) made changes - 2017-07-04 13:20

Rank

Ranked lower

Sachin Setiya (Inactive) made changes - 2017-07-04 13:20

Rank

Ranked lower

Sachin Setiya (Inactive) made changes - 2017-07-04 13:20

Rank

Ranked lower

Sachin Setiya (Inactive) made changes - 2017-07-04 13:20

Rank

Ranked lower

Sachin Setiya (Inactive) made changes - 2017-07-04 13:20

Rank

Ranked lower

Marko Mäkelä made changes - 2017-08-18 08:05

Link

This issue relates to MDEV-12445 [ MDEV-12445 ]

Elena Stepanova made changes - 2018-06-19 14:21

Assignee

Sachin Setiya [ sachin.setiya.007 ]

Elena Stepanova [ elenst ]

Elena Stepanova made changes - 2018-07-17 00:27

Status

In Progress [ 3 ]

Stalled [ 10000 ]

Elena Stepanova added a comment - 2018-07-17 12:10

There is no assertion failure on the current 10.2, possibly because the test fails earlier on other reasons, but there is some still problem with memory:

10.2 1cc1d0429 (slave)
2018-07-17 12:23:35 140149644458880 [Note] WSREP: Waiting for SST to complete. waited 11205470 secs.
2018-07-17 12:23:35 140149404985088 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '127.0.0.1:16007' -
-datadir '/data/bld/10.2/mysql-test/var/mysqld.2/data/' --defaults-file '/data/bld/10.2/mysql-test/var/my.cnf' --defaults-group-suffix '.2' --parent
'20786' '' : 2 (No such file or directory)
2018-07-17 12:23:35 140149644458880 [Note] WSREP: Waiting for SST to complete. waited 11205480 secs.
2018-07-17 12:23:35 140149404985088 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
2018-07-17 12:23:35 140149644458880 [ERROR] WSREP: SST failed: 2 (No such file or directory)
2018-07-17 12:23:35 140149644458880 [ERROR] Aborting

Error in my_thread_global_end(): 1 threads didn't exit
Warning: 56 bytes lost at 0x7f77080171c0, allocated by T@0 at 0x564519db4581, 0x5645192f5236, 0x564519308c02, 0x5645193936e3, 0x5645192fa58a, 0x5645
195b8158, nptl/pthread_create.c:333, x86_64/clone.S:99
Warning: 2048 bytes lost at 0x56451c2f5b00, allocated by T@0 at mysys/array.c:70, mysys/hash.c:98, mysys/thr_mutex.c:176, mysys/thr_mutex.c:318, psi/m
ysql_thread.h:666, sql/sql_class.h:3683, sql/wsrep_mysqld.cc:2095, sql/wsrep_mysqld.cc:2239
Warning: 2048 bytes lost at 0x56451c2f5280, allocated by T@0 at mysys/array.c:70, mysys/hash.c:98, mysys/thr_mutex.c:171, mysys/thr_mutex.c:318, psi/m
ysql_thread.h:666, sql/sql_class.h:3683, sql/wsrep_mysqld.cc:2095, sql/wsrep_mysqld.cc:2239
Warning: 208 bytes lost at 0x56451c330930, allocated by T@0 at mysys/mulalloc.c:51, mysys/thr_mutex.c:158, mysys/thr_mutex.c:318, psi/mysql_thread.h:
666, sql/sql_class.h:3683, sql/wsrep_mysqld.cc:2095, sql/wsrep_mysqld.cc:2239, sql/mysqld.cc:2128
Warning: 2048 bytes lost at 0x56451c2f4a00, allocated by T@0 at mysys/array.c:70, mysys/hash.c:98, mysys/thr_mutex.c:176, mysys/thr_mutex.c:318, psi/m
ysql_thread.h:666, sql/wsrep_mysqld.cc:2100, sql/wsrep_mysqld.cc:2239, sql/mysqld.cc:2128
Warning: 2048 bytes lost at 0x56451c2f4180, allocated by T@0 at mysys/array.c:70, mysys/hash.c:98, mysys/thr_mutex.c:171, mysys/thr_mutex.c:318, psi/m
ysql_thread.h:666, sql/wsrep_mysqld.cc:2100, sql/wsrep_mysqld.cc:2239, sql/mysqld.cc:2128
Warning: 208 bytes lost at 0x56451c2d9300, allocated by T@0 at mysys/mulalloc.c:51, mysys/thr_mutex.c:158, mysys/thr_mutex.c:318, psi/mysql_thread.h:
666, sql/wsrep_mysqld.cc:2100, sql/wsrep_mysqld.cc:2239, sql/mysqld.cc:2128, sql/wsrep_mysqld.cc:840
...

Elena Stepanova added a comment - 2018-07-17 12:10 There is no assertion failure on the current 10.2, possibly because the test fails earlier on other reasons, but there is some still problem with memory: 10.2 1cc1d0429 (slave) 2018-07-17 12:23:35 140149644458880 [Note] WSREP: Waiting for SST to complete. waited 11205470 secs. 2018-07-17 12:23:35 140149404985088 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '127.0.0.1:16007' - -datadir '/data/bld/10.2/mysql-test/var/mysqld.2/data/' --defaults-file '/data/bld/10.2/mysql-test/var/my.cnf' --defaults-group-suffix '.2' --parent '20786' '' : 2 (No such file or directory) 2018-07-17 12:23:35 140149644458880 [Note] WSREP: Waiting for SST to complete. waited 11205480 secs. 2018-07-17 12:23:35 140149404985088 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script. 2018-07-17 12:23:35 140149644458880 [ERROR] WSREP: SST failed: 2 (No such file or directory) 2018-07-17 12:23:35 140149644458880 [ERROR] Aborting Error in my_thread_global_end(): 1 threads didn't exit Warning: 56 bytes lost at 0x7f77080171c0, allocated by T@0 at 0x564519db4581, 0x5645192f5236, 0x564519308c02, 0x5645193936e3, 0x5645192fa58a, 0x5645 195b8158, nptl/pthread_create.c:333, x86_64/clone.S:99 Warning: 2048 bytes lost at 0x56451c2f5b00, allocated by T@0 at mysys/array.c:70, mysys/hash.c:98, mysys/thr_mutex.c:176, mysys/thr_mutex.c:318, psi/m ysql_thread.h:666, sql/sql_class.h:3683, sql/wsrep_mysqld.cc:2095, sql/wsrep_mysqld.cc:2239 Warning: 2048 bytes lost at 0x56451c2f5280, allocated by T@0 at mysys/array.c:70, mysys/hash.c:98, mysys/thr_mutex.c:171, mysys/thr_mutex.c:318, psi/m ysql_thread.h:666, sql/sql_class.h:3683, sql/wsrep_mysqld.cc:2095, sql/wsrep_mysqld.cc:2239 Warning: 208 bytes lost at 0x56451c330930, allocated by T@0 at mysys/mulalloc.c:51, mysys/thr_mutex.c:158, mysys/thr_mutex.c:318, psi/mysql_thread.h: 666, sql/sql_class.h:3683, sql/wsrep_mysqld.cc:2095, sql/wsrep_mysqld.cc:2239, sql/mysqld.cc:2128 Warning: 2048 bytes lost at 0x56451c2f4a00, allocated by T@0 at mysys/array.c:70, mysys/hash.c:98, mysys/thr_mutex.c:176, mysys/thr_mutex.c:318, psi/m ysql_thread.h:666, sql/wsrep_mysqld.cc:2100, sql/wsrep_mysqld.cc:2239, sql/mysqld.cc:2128 Warning: 2048 bytes lost at 0x56451c2f4180, allocated by T@0 at mysys/array.c:70, mysys/hash.c:98, mysys/thr_mutex.c:171, mysys/thr_mutex.c:318, psi/m ysql_thread.h:666, sql/wsrep_mysqld.cc:2100, sql/wsrep_mysqld.cc:2239, sql/mysqld.cc:2128 Warning: 208 bytes lost at 0x56451c2d9300, allocated by T@0 at mysys/mulalloc.c:51, mysys/thr_mutex.c:158, mysys/thr_mutex.c:318, psi/mysql_thread.h: 666, sql/wsrep_mysqld.cc:2100, sql/wsrep_mysqld.cc:2239, sql/mysqld.cc:2128, sql/wsrep_mysqld.cc:840 ...

Elena Stepanova made changes - 2018-07-17 12:11

Component/s		Galera SST [ 10121 ]
Component/s		Tests [ 10800 ]

Elena Stepanova made changes - 2018-07-17 12:11

Labels

galera

Elena Stepanova made changes - 2018-07-17 12:11

Component/s

Galera [ 10124 ]

Elena Stepanova made changes - 2018-07-17 12:11

Assignee

Elena Stepanova [ elenst ]

Jan Lindström [ jplindst ]

Jan Lindström (Inactive) made changes - 2019-12-12 12:07

Assignee

Jan Lindström [ jplindst ]

Stepan Patryshev [ stepan.patryshev ]

Jan Lindström (Inactive) made changes - 2019-12-12 12:07

Labels

need_feedback

Stepan Patryshev (Inactive) made changes - 2019-12-18 16:17

Link

This issue blocks ~~MDEV-19937~~ [ ~~MDEV-19937~~ ]

Stepan Patryshev (Inactive) made changes - 2020-01-13 15:27

Summary

Galera: assertion failure: global_status_var.global_memory_used == 0

galera_sst_xtrabackup-v2: assertion failure: global_status_var.global_memory_used == 0

Stepan Patryshev (Inactive) added a comment - 2020-01-13 16:00

Closing as not reproduced. Last time it failed on bb (10.2) was on 01.04.2019.

Stepan Patryshev (Inactive) added a comment - 2020-01-13 16:00 Closing as not reproduced. Last time it failed on bb (10.2) was on 01.04.2019.

Stepan Patryshev (Inactive) made changes - 2020-01-13 16:00

Fix Version/s		N/A [ 14700 ]
Fix Version/s	10.2 [ 14601 ]
Resolution		Cannot Reproduce [ 5 ]
Status	Stalled [ 10000 ]	Closed [ 6 ]

Stepan Patryshev (Inactive) made changes - 2020-01-13 16:00

Link

This issue blocks ~~MDEV-19937~~ [ ~~MDEV-19937~~ ]

Sergei Golubchik made changes - 2021-12-06 21:43

Workflow

MariaDB v3 [ 78174 ]

MariaDB v4 [ 151168 ]

People

Assignee:: Stepan Patryshev (Inactive)

Reporter:: Nirbhay Choubey (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 2016-11-01 21:55

Updated:: 2020-01-13 16:00

Resolved:: 2020-01-13 16:00

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration