[MDEV-12232] wsrep_info.plugin crashes in buildbot on ppc64le Created: 2017-03-11  Updated: 2017-05-17  Resolved: 2017-05-16

Status: Closed
Project: MariaDB Server
Component/s: Tests, wsrep
Affects Version/s: 10.1, 10.2
Fix Version/s: 10.1.23, 10.2.6

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Daniel Bartholomew
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Duplicate
duplicates MDEV-12234 wsrep.pool_of_threads fails in buildb... Closed

 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-deb-trusty-ppc64le/builds/392

wsrep_info.plugin 'innodb_plugin'        w1 [ fail ]
        Test ended at 2017-03-10 19:16:39
 
CURRENT_TEST: wsrep_info.plugin
 
 
The server [mysqld.1 - pid: 10173, winpid: 10173, exit: 256] crashed while running 'check testcase before test'
Server log from this test:
----------SERVER LOG START-----------
2017-03-10 19:16:38 70366664889488 [Note] /usr/sbin/mysqld (mysqld 10.1.22-MariaDB-1~trusty) starting as process 10174 ...
2017-03-10 19:16:38 70366664889488 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4162)
2017-03-10 19:16:38 70366664889488 [Warning] WSREP: wsrep_sst_receive_address is set to '127.0.0.1:16003' which makes it impossible for another host to reach this one. Please set it to the address which this node can be connected at by other cluster members.
2017-03-10 19:16:38 70366664889488 [Note] WSREP: Read nil XID from storage engines, skipping position init
2017-03-10 19:16:38 70366664889488 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2017-03-10 19:16:38 70366664889488 [ERROR] WSREP: wsrep_load(): dlopen(): /usr/lib/powerpc64le-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /usr/lib/galera/libgalera_smm.so)
2017-03-10 19:16:38 70366664889488 [ERROR] WSREP: wsrep_load(/usr/lib/galera/libgalera_smm.so) failed: Invalid argument (22). Reverting to no provider.
2017-03-10 19:16:38 70366664889488 [Note] WSREP: Read nil XID from storage engines, skipping position init
2017-03-10 19:16:38 70366664889488 [Note] WSREP: wsrep_load(): loading provider library 'none'
2017-03-10 19:16:38 70366664889488 [Note] Plugin 'partition' is disabled.
2017-03-10 19:16:38 70366664889488 [Note] Plugin 'SEQUENCE' is disabled.
2017-03-10 19:16:38 3fff84109490 InnoDB: Warning: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead, see http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html.
2017-03-10 19:16:39 70366664889488 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2017-03-10 19:16:39 70366664889488 [Note] InnoDB: The InnoDB memory heap is disabled
2017-03-10 19:16:39 70366664889488 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-03-10 19:16:39 70366664889488 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-03-10 19:16:39 70366664889488 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-03-10 19:16:39 70366664889488 [Note] InnoDB: Using Linux native AIO
2017-03-10 19:16:39 70366664889488 [Note] InnoDB: Using POWER8 crc32 instructions
2017-03-10 19:16:39 70366664889488 [Note] InnoDB: Initializing buffer pool, size = 8.0M
2017-03-10 19:16:39 70366664889488 [Note] InnoDB: Completed initialization of buffer pool
2017-03-10 19:16:39 70366664889488 [Note] InnoDB: Highest supported file format is Barracuda.
2017-03-10 19:16:39 70366664889488 [Note] InnoDB: 128 rollback segment(s) are active.
2017-03-10 19:16:39 70366664889488 [Note] InnoDB: Waiting for purge to start
170310 19:16:39 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.1.22-MariaDB-1~trusty
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62897 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400
/usr/sbin/mysqld(my_print_stacktrace+0x64)[0x43c44ea4]
/usr/sbin/mysqld(handle_fatal_signal+0x430)[0x4362b5b0]
[0x3fff840a0478]
/lib64/ld64.so.2(+0x10c58)[0x3fff840d0c58]
2017-03-10 19:16:39 70366664889488 [Note] InnoDB: 5.6.35 started; log sequence number 1629796
2017-03-10 19:16:39 70366361809216 [Note] InnoDB: Dumping buffer pool(s) not yet started
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_CMP' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'XTRADB_READ_VIEW' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'XTRADB_RSEG' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'CHANGED_PAGE_BITMAPS' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'XTRADB_INTERNAL_HASH_TABLES' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_MUTEXES' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'FEEDBACK' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
2017-03-10 19:16:39 70366664889488 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
2017-03-10 19:16:39 70366664889488 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-startup-interval=20'
2017-03-10 19:16:39 70366664889488 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-first-interval=60'
2017-03-10 19:16:39 70366664889488 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-interval=60'
2017-03-10 19:16:39 70366664889488 [Warning] /usr/sbin/mysqld: unknown option '--loose-pam-debug'
2017-03-10 19:16:39 70366664889488 [Warning] /usr/sbin/mysqld: unknown option '--loose-skip-plugin-innodb-changed-pages'
2017-03-10 19:16:39 70366664889488 [Warning] /usr/sbin/mysqld: unknown variable 'loose-debug-sync-timeout=300'
/lib64/ld64.so.2(+0x10c88)[0x3fff840d0c88]
/lib64/ld64.so.2(+0x1d0c8)[0x3fff840dd0c8]
/lib64/ld64.so.2(+0x165c8)[0x3fff840d65c8]
2017-03-10 19:16:39 70366664889488 [Note] Server socket created on IP: '127.0.0.1'.
2017-03-10 19:16:39 70366664889488 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.1.22-MariaDB-1~trusty'  socket: '/run/shm/var/tmp/1/mysqld.1.sock'  port: 16000  mariadb.org binary distribution
/lib64/ld64.so.2(+0x1c42c)[0x3fff840dc42c]
/lib/powerpc64le-linux-gnu/libc.so.6(+0x1737f0)[0x3fff839d37f0]
/lib64/ld64.so.2(+0x165c8)[0x3fff840d65c8]
/lib/powerpc64le-linux-gnu/libc.so.6(__libc_dlopen_mode+0x68)[0x3fff839d3968]
/lib/powerpc64le-linux-gnu/libpthread.so.0(+0x15188)[0x3fff83ca5188]
/lib/powerpc64le-linux-gnu/libpthread.so.0(+0x153dc)[0x3fff83ca53dc]
/lib/powerpc64le-linux-gnu/libpthread.so.0(__pthread_unwind+0x40)[0x3fff83ca2280]
/lib/powerpc64le-linux-gnu/libpthread.so.0(pthread_exit+0x54)[0x3fff83c99f74]
/usr/lib/mysql/plugin/ha_innodb.so(+0x1cc2f8)[0x3fff7c33c2f8]
/usr/lib/mysql/plugin/ha_innodb.so(+0x271670)[0x3fff7c3e1670]
/lib/powerpc64le-linux-gnu/libpthread.so.0(+0x89a4)[0x3fff83c989a4]
/lib/powerpc64le-linux-gnu/libc.so.6(clone+0x98)[0x3fff8397f400]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file

I'm not sure whether it's supposed to work on ppc64le, but it surely shouldn't cause a server crash.



 Comments   
Comment by Daniel Black [ 2017-04-02 ]

Looks like libgalera_smm.so is just compiled against another C++ ABI than what is on this builldbot. So where did it come from? dbart?

Comment by Elena Stepanova [ 2017-05-08 ]

Recent occurrence on 10.2
http://buildbot.askmonty.org/buildbot/builders/kvm-deb-trusty-ppc64le/builds/924/steps/mtr/logs/stdio

Comment by Daniel Black [ 2017-05-11 ]

note: test currently masked mysql-test/unstable-tests - can be removed once this sysadmin issue is resolved

Comment by Daniel Bartholomew [ 2017-05-16 ]

This was fixed when we upgraded Galera to 25.3.20.

10.1 tests are now passing fine:
http://buildbot.askmonty.org/buildbot/builders/kvm-deb-trusty-ppc64le/builds/1146/steps/mtr/logs/stdio

Same with 10.2:
http://buildbot.askmonty.org/buildbot/builders/kvm-deb-trusty-ppc64le/builds/1156/steps/mtr/logs/stdio

Thanks.

Comment by Daniel Black [ 2017-05-16 ]

Thanks dbart.

elenst can this be removed from mysql-test/unstable-tests now?

Comment by Elena Stepanova [ 2017-05-17 ]

can this be removed from mysql-test/unstable-tests now?

It will be removed in due time if it doesn't show any signs of instability.

Generated at Thu Feb 08 07:56:07 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.