[MDEV-12315] Hitting Bug since the update from 10.1.20 to 10.1.22 Created: 2017-03-21  Updated: 2019-05-20  Resolved: 2019-05-20

Status: Closed
Project: MariaDB Server
Component/s: Galera, wsrep
Affects Version/s: 10.1.22
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Sven Barczyk Assignee: Jan Lindström (Inactive)
Resolution: Incomplete Votes: 0
Labels: galera, innodb, replication
Environment:

Cluster of 3 Nodes
4.4.0-67-generic
10.1.22+maria-1~xenial
64 GB RAM
2x 512 SSDs in Software-Raid1


Attachments: Text File mysql.20.03.17.log     Text File mysql.log    

 Description   

Hey Guys,
since the security-update has been published, i am hitting this bug on a regulary base.
I cant really tell what is triggering this Bug.

Mar 20 17:00:07 node1 mysqld[31089]: 170320 17:00:07 [ERROR] mysqld got signal 11 ;
Mar 20 17:00:07 node1 mysqld[31089]: This could be because you hit a bug. It is also possible that this binary
Mar 20 17:00:07 node1 mysqld[31089]: or one of the libraries it was linked against is corrupt, improperly built,
Mar 20 17:00:07 node1 mysqld[31089]: or misconfigured. This error can also be caused by malfunctioning hardware.
Mar 20 17:00:07 node1 mysqld[31089]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Mar 20 17:00:07 node1 mysqld[31089]: We will try our best to scrape up some info that will hopefully help
Mar 20 17:00:07 node1 mysqld[31089]: diagnose the problem, but since we have already crashed,
Mar 20 17:00:07 node1 mysqld[31089]: something is definitely wrong and this may fail.
Mar 20 17:00:07 node1 mysqld[31089]: Server version: 10.1.22-MariaDB-1~xenial
Mar 20 17:00:07 node1 mysqld[31089]: key_buffer_size=134217728
Mar 20 17:00:07 node1 mysqld[31089]: read_buffer_size=2097152
Mar 20 17:00:07 node1 mysqld[31089]: max_used_connections=54
Mar 20 17:00:07 node1 mysqld[31089]: max_threads=1026
Mar 20 17:00:07 node1 mysqld[31089]: thread_count=41
Mar 20 17:00:07 node1 mysqld[31089]: It is possible that mysqld could use up to
Mar 20 17:00:07 node1 mysqld[31089]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6455868 K  bytes of memory
Mar 20 17:00:07 node1 mysqld[31089]: Hope that's ok; if not, decrease some variables in the equation.
Mar 20 17:00:07 node1 mysqld[31089]: Thread pointer: 0x0
Mar 20 17:00:07 node1 mysqld[31089]: Attempting backtrace. You can use the following information to find out
Mar 20 17:00:07 node1 mysqld[31089]: where mysqld died. If you see no messages after this, something went
Mar 20 17:00:07 node1 mysqld[31089]: terribly wrong...
Mar 20 17:00:07 node1 mysqld[31089]: stack_bottom = 0x0 thread_stack 0x48400
Mar 20 17:00:07 node1 mysqld[31089]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5591b2d68d9e]
Mar 20 17:00:07 node1 mysqld[31089]: /usr/sbin/mysqld(handle_fatal_signal+0x2f5)[0x5591b28b62d5]
Mar 20 17:00:07 node1 mysqld[31089]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f75942d6390]
Mar 20 17:00:07 node1 mysqld[31089]: /lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x4)[0x7f75942ced44]
Mar 20 17:00:07 node1 mysqld[31089]: /usr/lib/x86_64-linux-gnu/libjemalloc.so.1(+0x2918a)[0x7f7594ff418a]
Mar 20 17:00:07 node1 mysqld[31089]: /usr/lib/x86_64-linux-gnu/libjemalloc.so.1(+0x295e5)[0x7f7594ff45e5]
Mar 20 17:00:07 node1 mysqld[31089]: /usr/lib/galera/libgalera_smm.so(_ZN5boost6detail17sp_counted_impl_pISt6vectorIhSaIhEEE7disposeEv+0x17)[0x7f759083f657]
Mar 20 17:00:07 node1 mysqld[31089]: /usr/lib/galera/libgalera_smm.so(+0x1846ea)[0x7f75908c96ea]
Mar 20 17:00:07 node1 mysqld[31089]: /usr/lib/galera/libgalera_smm.so(_ZN5gcomm13AsioTcpSocket12read_handlerERKN4asio10error_codeEm+0x705)[0x7f75908d1555]
Mar 20 17:00:07 node1 mysqld[31089]: /usr/lib/galera/libgalera_smm.so(_ZN4asio6detail7read_opINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS4_EEEEN5boost5arrayINS_14mutable_bufferELm1EEENS8_3_bi6bind_tImNS8_4_mfi3mf2ImN5gcomm13AsioTcpSocketERKNS_10error_codeEmEENSC_5list3INSC_5valueINS8_10shared_ptrISH_EEEEPFNS8_3argILi1EEEvEPFNSR_ILi2EEEvEEEEENSD_IvNSF_IvSH_SK_mEESY_EEEclESK_mi+0x4bb)[0x7f75908e289b]
Mar 20 17:00:07 node1 mysqld[31089]: /usr/lib/galera/libgalera_smm.so(_ZN4asio6detail23reactive_socket_recv_opINS0_17consuming_buffersINS_14mutable_bufferEN5boost5arrayIS3_Lm1EEEEENS0_7read_opINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceISB_EEEES6_NS4_3_bi6bind_tImNS4_4_mfi3mf2ImN5gcomm13AsioTcpSocketERKNS_10error_codeEmEENSF_5list3INSF_5valueINS4_10shared_ptrISK_EEEEPFNS4_3argILi1EEEvEPFNSU_ILi2EEEvEEEEENSG_IvNSI_IvSK_SN_mEES11_EEEEE11do_completeEPNS0_15task_io_serviceEPNS0_25task_io_service_operationESN_m+0x1b4)[0x7f75908e2ea4]
Mar 20 17:00:07 node1 mysqld[31089]: /usr/lib/galera/libgalera_smm.so(_ZN4asio6detail13epoll_reactor16descriptor_state11do_completeEPNS0_15task_io_serviceEPNS0_25task_io_service_operationERKNS_10error_codeEm+0x168)[0x7f75908d7b68]
Mar 20 17:00:07 node1 mysqld[31089]: /usr/lib/galera/libgalera_smm.so(_ZN5gcomm12AsioProtonet10event_loopERKN2gu8datetime6PeriodE+0x79a)[0x7f75908f471a]
Mar 20 17:00:07 node1 mysqld[31089]: /usr/lib/galera/libgalera_smm.so(_ZN9GCommConn3runEv+0xc4)[0x7f759090a174]
Mar 20 17:00:07 node1 mysqld[31089]: /usr/lib/galera/libgalera_smm.so(_ZN9GCommConn6run_fnEPv+0x9)[0x7f7590911e49]
Mar 20 17:00:07 node1 mysqld[31089]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f75942cc6ba]
Mar 20 17:00:07 node1 mysqld[31089]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f759397782d]
Mar 20 17:00:07 node1 mysqld[31089]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Mar 20 17:00:07 node1 mysqld[31089]: information that should help you find out what is causing the crash.



 Comments   
Comment by Jan Lindström (Inactive) [ 2017-03-21 ]

Could you provide us galera library version, full unedited error log and configuration files.

Comment by Sven Barczyk [ 2017-03-21 ]

galera-3 - 25.3.19-xenial
Sorry i have no dedicated mysql.log but in attach a syslog | grep mysql from the our second node.
Happend again at 10:49:49 but i trimmed systemd to restart on-failure after 30secs.

Comment by Jan Lindström (Inactive) [ 2017-03-21 ]

Thanks, you should definitely have dedicated mysql.log, assertions on these are different compared to one above, what is the latest assertion seen on syslog and is it same as above i.e. assertion on libjemalloc ?

Comment by Sven Barczyk [ 2017-03-21 ]

the assertion on 10.49:49 was the last one.
The First one (opening the Thread) was from yesterday, but it seems that the assertion was thrown after a startup.
Log from yesterday attached.

Generated at Thu Feb 08 07:56:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.