[MXS-4929] Crash on repeated calls to "maxctrl show sessions" Created: 2024-01-11  Updated: 2024-01-22  Resolved: 2024-01-19

Status: Closed
Project: MariaDB MaxScale
Component/s: maxctrl
Affects Version/s: 23.08.2
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Phil Sumner Assignee: markus makela
Resolution: Duplicate Votes: 0
Labels: None

Issue Links:
Duplicate
duplicates MXS-4847 Crash on `maxctrl list sessions` Closed

 Description   

CentOS Linux release 7.9.2009 (Core)

[psumner@xxxxx ~]$ uname -a
Linux dblb02..... 3.10.0-1160.90.1.el7.x86_64 #1 SMP Thu May 4 15:21:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[psumner@xxxxx ~]$ maxctrl --version
23.08.2

Repeated calls - not necessarily done in quick succession - to:

maxctrl show sessions

causes Maxscale to restart with this in the error log:

2024-01-11 13:56:06   alert  : MaxScale 23.08.2 received fatal signal 11. Commit ID: be2d3b6e3373dcd6febbde93a5a78ed24ca4eda6, System name: Linux, Release string: CentOS Linux 7 (Core), Thread: Worker-11
2024-01-11 13:56:06   alert  : Statement currently being classified: none/unknown
2024-01-11 13:56:06   notice : For a more detailed stacktrace, install GDB and add 'debug=gdb-stacktrace' under the [maxscale] section.
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0 (MariaDBClientConnection::diagnostics() const): ??:?
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0 (Session::as_json_resource(char const*, bool) const): /opt/rh/devtoolset-9/root/usr/include/c++/9/bits/std_function.h:687
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0 (seslist_cb(DCB*, void*)): /opt/rh/devtoolset-9/root/usr/include/c++/9/bits/std_function.h:687
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0 (SerialDcbTask::execute(maxbase::Worker&)): server/core/dcb.cc:1199
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0 (maxbase::Worker::handle_message(maxbase::MessageQueue&, maxbase::MessageQueueMessage const&)): ??:?
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0 (maxbase::EventMessageQueue::handle_poll_events(maxbase::Worker*, unsigned int, maxbase::Pollable::Context)): maxutils/maxbase/src/messagequeue.cc:222
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0 (maxbase::Worker::deliver_events(maxbase::Pollable*, unsigned int, maxbase::Pollable::Context)): ??:?
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0 (maxbase::Worker::poll_waitevents()): ??:?
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0 (maxbase::Worker::run(maxbase::Semaphore*)): ??:?
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0 (execute_native_thread_routine): thread48.o:?
  /lib64/libpthread.so.0 (start_thread): pthread_create.c:?
  /lib64/libc.so.6 (__clone): ??:?
MaxScale 23.08.2 received fatal signal 11. Commit ID: be2d3b6e3373dcd6febbde93a5a78ed24ca4eda6, System name: Linux, Release string: CentOS Linux 7 (Core), Thread: Worker-11Writing core dump.

I can't easily install GDB on this machine, so am hoping this information is useful enough.



 Comments   
Comment by Phil Sumner [ 2024-01-11 ]

This is quite a busy Maxscale server, with between 600-1,000 sessions per backend at any time

[psumner@dblb02 ~]$ maxctrl list servers
┌─────────┬─────────────────┬──────┬─────────────┬─────────────────────────┬─────────────────────────────────────────────────────────────────┬───────────────┐
│ Server  │ Address         │ Port │ Connections │ State                   │ GTID                                                            │ Monitor       │
├─────────┼─────────────────┼──────┼─────────────┼─────────────────────────┼─────────────────────────────────────────────────────────────────┼───────────────┤
│ db01 │ 192.168.1.1 │ 3306 │ 656         │ Slave, Synced, Running  │ 0-10-71664821512,126-10-16,127-10-2,139-10-2,232-10-70885485545 │ GaleraMonitor │
├─────────┼─────────────────┼──────┼─────────────┼─────────────────────────┼─────────────────────────────────────────────────────────────────┼───────────────┤
│ db02 │ 192.168.1.2 │ 3306 │ 659         │ Master, Synced, Running │ 126-10-16,232-10-71664875813                                    │ GaleraMonitor │
├─────────┼─────────────────┼──────┼─────────────┼─────────────────────────┼─────────────────────────────────────────────────────────────────┼───────────────┤
│ db03 │ 192.168.1.3 │ 3306 │ 656         │ Slave, Synced, Running  │ 0-10-71664821523,126-10-16,139-10-2,232-10-70885485545          │ GaleraMonitor │
└─────────┴─────────────────┴──────┴─────────────┴─────────────────────────┴─────────────────────────────────────────────────────────────────┴───────────────┘

Comment by Phil Sumner [ 2024-01-11 ]

[psumner@dblb02 ~]$ maxctrl show service SS
┌─────────────────────┬────────────────────────────────────────────────────────┐
│ Service             │ SS                                                     │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ Source              │ /etc/maxscale.cnf                                      │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ Router              │ readwritesplit                                         │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ State               │ Started                                                │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ Started At          │ 1/11/2024, 13:56:08                                    │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ Users Loaded At     │ 1/11/2024, 13:56:08                                    │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ Current Connections │ 741                                                    │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ Total Connections   │ 520854                                                 │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ Max Connections     │ 1500                                                   │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ Cluster             │                                                        │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ Servers             │ db01                                                │
│                     │ db02                                                │
│                     │ db03                                                │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ Services            │                                                        │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ Filters             │ hintFilter                                             │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ Parameters          │ {                                                      │
│                     │     "auth_all_servers": false,                         │
│                     │     "causal_reads": "none",                            │
│                     │     "causal_reads_timeout": "10000ms",                 │
│                     │     "connection_keepalive": "300000ms",                │
│                     │     "delayed_retry": false,                            │
│                     │     "delayed_retry_timeout": "10000ms",                │
│                     │     "disable_sescmd_history": true,                    │
│                     │     "enable_root_user": false,                         │
│                     │     "force_connection_keepalive": false,               │
│                     │     "idle_session_pool_time": "-1ms",                  │
│                     │     "lazy_connect": false,                             │
│                     │     "localhost_match_wildcard_host": true,             │
│                     │     "log_auth_warnings": true,                         │
│                     │     "log_debug": false,                                │
│                     │     "log_info": false,                                 │
│                     │     "log_notice": false,                               │
│                     │     "log_warning": true,                               │
│                     │     "master_accept_reads": false,                      │
│                     │     "master_failure_mode": "fail_instantly",           │
│                     │     "master_reconnection": false,                      │
│                     │     "max_connections": 1500,                           │
│                     │     "max_replication_lag": "0ms",                      │
│                     │     "max_sescmd_history": 50,                          │
│                     │     "max_slave_connections": 255,                      │
│                     │     "multiplex_timeout": "60000ms",                    │
│                     │     "net_write_timeout": "0ms",                        │
│                     │     "optimistic_trx": false,                           │
│                     │     "password": "*****",                               │
│                     │     "prune_sescmd_history": true,                      │
│                     │     "rank": "primary",                                 │
│                     │     "retain_last_statements": -1,                      │
│                     │     "retry_failed_reads": true,                        │
│                     │     "reuse_prepared_statements": false,                │
│                     │     "router": "readwritesplit",                        │
│                     │     "session_trace": false,                            │
│                     │     "slave_connections": 255,                          │
│                     │     "slave_selection_criteria": "adaptive_routing",    │
│                     │     "strict_multi_stmt": true,                         │
│                     │     "strict_sp_calls": true,                           │
│                     │     "strict_tmp_tables": false,                        │
│                     │     "strip_db_esc": true,                              │
│                     │     "transaction_replay": false,                       │
│                     │     "transaction_replay_attempts": 5,                  │
│                     │     "transaction_replay_checksum": "full",             │
│                     │     "transaction_replay_max_size": 1048576,            │
│                     │     "transaction_replay_retry_on_deadlock": false,     │
│                     │     "transaction_replay_retry_on_mismatch": false,     │
│                     │     "transaction_replay_safe_commit": true,            │
│                     │     "transaction_replay_timeout": "0ms",               │
│                     │     "type": "service",                                 │
│                     │     "use_sql_variables_in": "master",                  │
│                     │     "user": "maxscale",                                │
│                     │     "user_accounts_file": null,                        │
│                     │     "user_accounts_file_usage": "add_when_load_ok",    │
│                     │     "version_string": null,                            │
│                     │     "wait_timeout": "0ms"                              │
│                     │ }                                                      │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ Router Diagnostics  │ {                                                      │
│                     │     "avg_sescmd_history_length": 0,                    │
│                     │     "max_sescmd_history_length": 0,                    │
│                     │     "queries": 37268874,                               │
│                     │     "replayed_transactions": 0,                        │
│                     │     "ro_transactions": 4843,                           │
│                     │     "route_all": 7589830,                              │
│                     │     "route_master": 4451032,                           │
│                     │     "route_slave": 25226096,                           │
│                     │     "rw_transactions": 0,                              │
│                     │     "server_query_statistics": [                       │
│                     │         {                                              │
│                     │             "avg_selects_per_session": 32,             │
│                     │             "avg_sess_active_pct": 1.4977732867335007, │
│                     │             "avg_sess_duration": "2.82303s",           │
│                     │             "id": "db01",                           │
│                     │             "read": 19661894,                          │
│                     │             "total": 19661901,                         │
│                     │             "write": 0                                 │
│                     │         },                                             │
│                     │         {                                              │
│                     │             "avg_selects_per_session": 18,             │
│                     │             "avg_sess_active_pct": 0,                  │
│                     │             "avg_sess_duration": "2.82303s",           │
│                     │             "id": "db02",                           │
│                     │             "read": 7591751,                           │
│                     │             "total": 12044686,                         │
│                     │             "write": 4451032                           │
│                     │         },                                             │
│                     │         {                                              │
│                     │             "avg_selects_per_session": 33,             │
│                     │             "avg_sess_active_pct": 1.7848739712971686, │
│                     │             "avg_sess_duration": "2.82303s",           │
│                     │             "id": "db03",                           │
│                     │             "read": 20705437,                          │
│                     │             "total": 20705443,                         │
│                     │             "write": 0                                 │
│                     │         }                                              │
│                     │     ],                                                 │
│                     │     "trx_max_size_exceeded": 0                         │
│                     │ }                                                      │
└─────────────────────┴────────────────────────────────────────────────────────┘

Comment by Phil Sumner [ 2024-01-11 ]

[psumner@dblb02 ~]$ maxctrl show maxscale
┌──────────────┬──────────────────────────────────────────────────────────────┐
│ Version      │ 23.08.2                                                      │
├──────────────┼──────────────────────────────────────────────────────────────┤
│ Commit       │ be2d3b6e3373dcd6febbde93a5a78ed24ca4eda6                     │
├──────────────┼──────────────────────────────────────────────────────────────┤
│ Started At   │ Thu, 11 Jan 2024 13:56:08 GMT                                │
├──────────────┼──────────────────────────────────────────────────────────────┤
│ Activated At │ Thu, 11 Jan 2024 13:56:08 GMT                                │
├──────────────┼──────────────────────────────────────────────────────────────┤
│ Uptime       │ 2727                                                         │
├──────────────┼──────────────────────────────────────────────────────────────┤
│ Config Sync  │ null                                                         │
├──────────────┼──────────────────────────────────────────────────────────────┤
│ Parameters   │ {                                                            │
│              │     "admin_audit": false,                                    │
│              │     "admin_audit_exclude_methods": [],                       │
│              │     "admin_audit_file": "/var/log/maxscale/admin_audit.csv", │
│              │     "admin_auth": true,                                      │
│              │     "admin_enabled": true,                                   │
│              │     "admin_gui": true,                                       │
│              │     "admin_host": "127.0.0.1",                               │
│              │     "admin_jwt_algorithm": "auto",                           │
│              │     "admin_jwt_issuer": "maxscale",                          │
│              │     "admin_jwt_key": null,                                   │
│              │     "admin_jwt_max_age": "86400000ms",                       │
│              │     "admin_log_auth_failures": true,                         │
│              │     "admin_oidc_url": null,                                  │
│              │     "admin_pam_readonly_service": null,                      │
│              │     "admin_pam_readwrite_service": null,                     │
│              │     "admin_port": 8989,                                      │
│              │     "admin_secure_gui": true,                                │
│              │     "admin_ssl_ca": null,                                    │
│              │     "admin_ssl_cert": null,                                  │
│              │     "admin_ssl_key": null,                                   │
│              │     "admin_ssl_version": "MAX",                              │
│              │     "admin_verify_url": null,                                │
│              │     "auth_connect_timeout": "10000ms",                       │
│              │     "auth_read_timeout": "10000ms",                          │
│              │     "auth_write_timeout": "10000ms",                         │
│              │     "auto_tune": [],                                         │
│              │     "cachedir": "/var/cache/maxscale",                       │
│              │     "config_sync_cluster": null,                             │
│              │     "config_sync_db": "mysql",                               │
│              │     "config_sync_interval": "5000ms",                        │
│              │     "config_sync_password": null,                            │
│              │     "config_sync_timeout": "10000ms",                        │
│              │     "config_sync_user": null,                                │
│              │     "connector_plugindir": "/usr/lib64/maxscale/plugin",     │
│              │     "datadir": "/var/lib/maxscale",                          │
│              │     "debug": null,                                           │
│              │     "dump_last_statements": "never",                         │
│              │     "execdir": "/usr/bin",                                   │
│              │     "key_manager": "none",                                   │
│              │     "language": "/var/lib/maxscale",                         │
│              │     "libdir": "/usr/lib64/maxscale",                         │
│              │     "load_persisted_configs": false,                         │
│              │     "local_address": null,                                   │
│              │     "log_debug": false,                                      │
│              │     "log_info": false,                                       │
│              │     "log_notice": true,                                      │
│              │     "log_throttling": {                                      │
│              │         "count": 10,                                         │
│              │         "suppress": 10000,                                   │
│              │         "window": 1000                                       │
│              │     },                                                       │
│              │     "log_warn_super_user": false,                            │
│              │     "log_warning": true,                                     │
│              │     "logdir": "/var/log/maxscale",                           │
│              │     "max_auth_errors_until_block": 10,                       │
│              │     "max_read_amount": 0,                                    │
│              │     "maxlog": true,                                          │
│              │     "module_configdir": "/etc/maxscale.modules.d",           │
│              │     "ms_timestamp": false,                                   │
│              │     "passive": false,                                        │
│              │     "persist_runtime_changes": true,                         │
│              │     "persistdir": "/var/lib/maxscale/maxscale.cnf.d",        │
│              │     "piddir": "/var/run/maxscale",                           │
│              │     "query_classifier_cache_size": 288838041,                │
│              │     "query_retries": 1,                                      │
│              │     "query_retry_timeout": "5000ms",                         │
│              │     "rebalance_period": "0ms",                               │
│              │     "rebalance_threshold": 20,                               │
│              │     "rebalance_window": 10,                                  │
│              │     "retain_last_statements": 0,                             │
│              │     "session_trace": 0,                                      │
│              │     "session_trace_match": null,                             │
│              │     "skip_name_resolve": false,                              │
│              │     "sql_mode": "default",                                   │
│              │     "syslog": false,                                         │
│              │     "threads": 12,                                           │
│              │     "threads_max": 256,                                      │
│              │     "users_refresh_interval": "0ms",                         │
│              │     "users_refresh_time": "30000ms",                         │
│              │     "writeq_high_water": 65536,                              │
│              │     "writeq_low_water": 1024                                 │
│              │ }                                                            │
├──────────────┼──────────────────────────────────────────────────────────────┤
│ System       │ {                                                            │
│              │     "machine": {                                             │
│              │         "cores_available": 12,                               │
│              │         "cores_physical": 12,                                │
│              │         "cores_virtual": 12,                                 │
│              │         "memory_available": 1925586944,                      │
│              │         "memory_physical": 1925586944                        │
│              │     },                                                       │
│              │     "maxscale": {                                            │
│              │         "query_classifier_cache_size": 288838041,            │
│              │         "threads": 12                                        │
│              │     },                                                       │
│              │     "os": {                                                  │
│              │         "machine": "x86_64",                                 │
│              │         "nodename": "dblb02.....",        │
│              │         "release": "3.10.0-1160.90.1.el7.x86_64",            │
│              │         "sysname": "Linux",                                  │
│              │         "version": "#1 SMP Thu May 4 15:21:22 UTC 2023"      │
│              │     }                                                        │
│              │ }                                                            │
└──────────────┴──────────────────────────────────────────────────────────────┘

Comment by markus makela [ 2024-01-11 ]

Most likely a duplicate of MXS-4847. Please try upgrading to MaxScale 23.08.4.

Comment by Phil Sumner [ 2024-01-22 ]

I upgraded today and it appears (so far) to have resolved the issue.

Comment by markus makela [ 2024-01-22 ]

OK, that's good. I think it confirms that this was indeed a duplicate of MXS-4847. However if you do see something, please let us know and we'll reopen the issue.

Generated at Thu Feb 08 04:32:10 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.