[#MXS-4124] Internal logic error, multiple crashs, different error messages

[MXS-4124] Internal logic error, multiple crashs, different error messages Created: 2022-05-09 Updated: 2022-07-06 Resolved: 2022-07-06
Status:	Closed
Project:	MariaDB MaxScale
Component/s:	readwritesplit
Affects Version/s:	6.3.0
Fix Version/s:	N/A

Type:

Bug

Priority:

Critical

Reporter:

acsfer

Assignee:

markus makela

Resolution:

Incomplete

Votes:

Labels:

None

Description

Maxscale crashed multiple times.
See below error messages (in red) and traces printed to logfile.

Internal logic error, disabling retain_last_statements.

alert  : MaxScale 6.3.0 received fatal signal 11. Commit ID: cdf2dd2884c6fdcaf2b42adebe28466836aa8d83 System name: Linux Release s

tring: Ubuntu 20.04.4 LTS

alert  : (readwritesplit); MaxScale 6.3.0 received fatal signal 11. Commit ID: cdf2dd2884c6fdcaf2b42adebe284

66836aa8d83 System name: Linux Release string: Ubuntu 20.04.4 LTS

alert  : (readwritesplit); Statement currently being classified: none/unknown

notice : (readwritesplit); For a more detailed stacktrace, install GDB and add 'debug=gdb-stacktrace' under

the [maxscale] section.

nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols

nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols

  /lib/x86_64-linux-gnu/libc.so.6(cfree+0x20): ??:0

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale7Backend9ack_writeEv+0x5c): /usr/include/c++/9/bits/d

eque.tcc:577

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale7Backend5closeENS0_10close_typeE+0x48): server/core/b

ackend.cc:51

  /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSessionD1Ev+0x89): server/modules/routing/readwritesplit/rws

plitsession.cc:82

  /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSessionD0Ev+0x16): server/modules/routing/readwritesplit/rws

plitsession.cc:91

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN15ServiceEndpoint5closeEv+0x8f): /usr/include/c++/9/bits/stl_i

terator.h:804

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN9ClientDCB8shutdownEv+0x25): /usr/include/c++/9/bits/unique_pt

r.h:154

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB7destroyEv+0x5d): server/core/dcb.cc:839

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale13RoutingWorker14delete_zombiesEv+0x61): /usr/include

/c++/9/bits/stl_vector.h:1005

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale13RoutingWorker10epoll_tickEv+0x21): /usr/include/c++

/9/bits/stl_iterator.h:804

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x2f0): maxutils/maxbase/sr

c/worker.cc:781

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/sr

c/worker.cc:558

  /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4): ??:?

  /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609): ??:?

  /lib/x86_64-linux-gnu/libc.so.6(clone+0x43): ??:0

alert  : Writing core dump.

Configuration :

[maxscale]

threads = auto

admin_gui = false

query_retries = 12

skip_name_resolve = true

[srv1]

type = server

address = 10.1.0.5

port = 3306

extra_port = 3307

protocol = MariaDBBackend

priority = 1

persistpoolmax = 128

persistmaxtime = 3600s

[srv2]

type = server

address = 10.1.0.6

port = 3306

extra_port = 3307

protocol = MariaDBBackend

priority = 2

persistpoolmax = 128

persistmaxtime = 3600s

[srv3]

type = server

address = 10.1.0.7

port = 3306

extra_port = 3307

protocol = MariaDBBackend

priority = 3

persistpoolmax = 128

persistmaxtime = 3600s

[monitor]

type = monitor

module = galeramon

servers = srv1,srv2,srv3

user = maxscale

password = 2B11DE20D111FCC055C

monitor_interval = 2000ms

use_priority = true

[readwritesplit]

type = service

router = readwritesplit

servers = srv1,srv2,srv3

user = maxscale

password = 2B11DE2A3E9F865D230564EA12FFA2A18B8E90D422C4A59C92E5FA08284D4862F8F2DA33C684D411B01C0D111FCC055C

delayed_retry = true

delayed_retry_timeout = 15s

retry_failed_reads = true

transaction_replay = true

transaction_replay_max_size = 10Mi

transaction_replay_attempts = 10

transaction_replay_retry_on_deadlock = true

[listener]

type = listener

service = readwritesplit

protocol = MariaDBClient

address = 127.0.0.1

port = 3306

Comments

Comment by acsfer [ 2022-05-11 ]

Another crash:

double free or corruption (out)

alert  : MaxScale 6.3.0 received fatal signal 6. Commit ID: cdf2dd2884c6fdcaf2b42adebe28466836aa8d83 System name: Linux Release string: Ubuntu     20.04.4 LTS

alert  : (readwritesplit); MaxScale 6.3.0 received fatal signal 6. Commit ID: cdf2dd2884c6fdcaf2b42adebe28466836aa8d83 S    ystem name: Linux Release string: Ubuntu 20.04.4 LTS

alert  : (readwritesplit); Statement currently being classified: none/unknown

notice : (readwritesplit); For a more detailed stacktrace, install GDB and add 'debug=gdb-stacktrace' under the [maxscal    e] section.

nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols

nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols

nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols

  /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb): ??:0

  /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b): ??:0

  /lib/x86_64-linux-gnu/libc.so.6(+0x8d29e): ??:?

  /lib/x86_64-linux-gnu/libc.so.6(+0x9532c): ??:?

  /lib/x86_64-linux-gnu/libc.so.6(+0x96fd0): ??:?

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale7Backend9ack_writeEv+0x5c): /usr/include/c++/9/bits/deque.tcc:577

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale7Backend5closeENS0_10close_typeE+0x48): server/core/backend.cc:51

  /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSessionD1Ev+0x89): server/modules/routing/readwritesplit/rwsplitsession.    cc:82

  /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSessionD0Ev+0x16): server/modules/routing/readwritesplit/rwsplitsession.    cc:91

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN15ServiceEndpoint5closeEv+0x8f): /usr/include/c++/9/bits/stl_iterator.h:80    4

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN9ClientDCB8shutdownEv+0x25): /usr/include/c++/9/bits/unique_ptr.h:154

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB7destroyEv+0x5d): server/core/dcb.cc:839

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale13RoutingWorker14delete_zombiesEv+0x61): /usr/include/c++/9/bits/    stl_vector.h:1005

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale13RoutingWorker10epoll_tickEv+0x21): /usr/include/c++/9/bits/stl_    iterator.h:804

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x2f0): maxutils/maxbase/src/worker.cc:    781

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/src/worker.cc:    558

  /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4): ??:?

  /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609): ??:?

  /lib/x86_64-linux-gnu/libc.so.6(clone+0x43): ??:0

alert  : Writing core dump.

Comment by acsfer [ 2022-05-12 ]

Another crash:

munmap_chunk(): invalid pointer

MaxScale 6.3.0 received fatal signal 6. Commit ID: cdf2dd2884c6fdcaf2b42adebe28466836aa8d83 System name: Linux Release string: Ubuntu 20.04.4 LTS

alert  : (readwritesplit); MaxScale 6.3.0 received fatal signal 6. Commit ID: cdf2dd2884c6fdcaf2b42adebe28466836aa8d83 System name: Linux Release string: Ubuntu 20.04.4 LTS

alert  : (readwritesplit); Statement currently being classified: none/unknown

notice : (readwritesplit); For a more detailed stacktrace, install GDB and add 'debug=gdb-stacktrace' under the [maxscale] section.

nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols

nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols

nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols

  /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb): ??:0

  /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b): ??:0

  /lib/x86_64-linux-gnu/libc.so.6(+0x8d29e): ??:?

  /lib/x86_64-linux-gnu/libc.so.6(+0x9532c): ??:?

  /lib/x86_64-linux-gnu/libc.so.6(+0x9557c): ??:?

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale7Backend9ack_writeEv+0x5c): /usr/include/c++/9/bits/deque.tcc:577

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale7Backend5closeENS0_10close_typeE+0x48): server/core/backend.cc:51

  /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSessionD1Ev+0x89): server/modules/routing/readwritesplit/rwsplitsession.cc:82

  /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSessionD0Ev+0x16): server/modules/routing/readwritesplit/rwsplitsession.cc:91

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN15ServiceEndpoint5closeEv+0x8f): /usr/include/c++/9/bits/stl_iterator.h:804

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN9ClientDCB8shutdownEv+0x25): /usr/include/c++/9/bits/unique_ptr.h:154

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB7destroyEv+0x5d): server/core/dcb.cc:839

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale13RoutingWorker14delete_zombiesEv+0x61): /usr/include/c++/9/bits/stl_vector.h:1005

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale13RoutingWorker10epoll_tickEv+0x21): /usr/include/c++/9/bits/stl_iterator.h:804

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x2f0): maxutils/maxbase/src/worker.cc:781

  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/src/worker.cc:558

  /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4): ??:?

  /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609): ??:?

  /lib/x86_64-linux-gnu/libc.so.6(clone+0x43): ??:0

alert  : Writing core dump.

Comment by markus makela [ 2022-06-07 ]

acs-ferreira Have you tried if this happens with MaxScale 6.3.1?

Comment by markus makela [ 2022-07-06 ]

Might be related to ~~MXS-4090~~ and as the 6.4.1 release will contain code that'll let us know, I'll close this for now. If this problem still persists with 6.4.1 when it has been release, please let us know and we'll reopen this bug.

Generated at Thu Feb 08 04:26:22 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.

[MXS-4124] Internal logic error, multiple crashs, different error messages Created: 2022-05-09 Updated: 2022-07-06 Resolved: 2022-07-06