[MXS-21] bugzillaId-738: logging to file instead of shared memory breaks 'load_balancing' and 'crash_out_of_files' tests Created: 2015-01-25  Updated: 2019-12-19  Resolved: 2015-07-08

Status: Closed
Project: MariaDB MaxScale
Component/s: Core, Plugin
Affects Version/s: 1.0.6
Fix Version/s: 1.1.1

Type: Bug Priority: Minor
Reporter: Timofey Turenko Assignee: markus makela
Resolution: Not a Bug Votes: 0
Labels: None
Environment:

Linux


 Description   

This is imported from bugzilla. http://bugs.mariadb.com/show_bug.cgi?id=738

Timofey Turenko 2015-02-25 17:39:22 UTC
1. start Maxscale with logging to file (now it is default option)
2. run 'load_balancing' or 'crash_out_of_files' test

Expected result: test PASSED, not garbage in the logs

Actual result:

  • test FAILED
  • log is full with:

015-02-25 19:38:10   Backend hangup error handling.
 
2015-02-25 19:38:10   Backend hangup error handling.
 
2015-02-25 19:38:10   Backend hangup error handling.
 
2015-02-25 19:38:10   Error : Unable to write to backend due to authentication failure.
 
2015-02-25 19:38:10   Backend hangup -> closing session.
 
2015-02-25 19:38:21   Backend hangup error handling.
 
2015-02-25 19:38:22   Backend hangup error handling.
 
2015-02-25 19:38:22   Backend hangup error handling.
 
2015-02-25 19:38:22   Backend hangup error handling.
 
2015-02-25 19:38:22   Backend hangup -> closing session.
 
2015-02-25 19:38:24   Backend hangup error handling.
 
2015-02-25 19:38:24   Backend hangup -> closing session.
 
2015-02-25 19:38:24   Backend hangup error handling.
 
2015-02-25 19:38:24   Backend hangup -> closing session.
 
2015-02-25 19:38:34   Backend hangup error handling.
 
2015-02-25 19:38:34   Backend hangup error handling.
 
2015-02-25 19:38:35   Error : Routing the query failed. Session will be closed.
 
2015-02-25 19:38:35   Backend hangup error handling.
 
2015-02-25 19:38:35   Backend hangup error handling.
 
2015-02-25 19:38:35   Error : Routing the query failed. Session will be closed.
 
2015-02-25 19:38:35   Backend hangup error handling.
 
2015-02-25 19:38:36   Backend hangup error handling.
 
2015-02-25 19:38:36   Backend hangup error handling.
 
2015-02-25 19:38:36   Backend hangup error handling.
 
2015-02-25 19:38:37   Backend hangup error handling.
 
2015-02-25 19:38:37   Backend hangup error handling.
 
2015-02-25 19:38:37   Error : Routing the query failed. Session will be closed.
 
2015-02-25 19:38:37   Backend hangup error handling.
 
2015-02-25 19:38:37   Backend hangup error handling.
 
2015-02-25 19:38:38   Backend hangup error handling.
 
2015-02-25 19:38:38   Backend hangup -> closing session.
 
2015-02-25 19:38:39   Backend hangup error handling.
 
2015-02-25 19:38:39   Backend hangup error handling.
 
2015-02-25 19:38:40   Error: Can't route COM_QUERY:QUERY_TYPE_READ:"SELECT * FROM t1;" to backend server. Router is closed.
 
2015-02-25 19:38:40   Error : Routing the query failed. Session will be closed.
 
2015-02-25 19:38:40   Backend hangup -> closing session.
 
2015-02-25 19:38:40   Backend error event handling.
 
2015-02-25 19:38:40   Backend hangup -> closing session.
 
2015-02-25 19:38:41   Backend hangup error handling.
 
2015-02-25 19:38:41   Backend hangup -> closing session.
 
2015-02-25 19:38:41   Backend hangup error handling.
 
2015-02-25 19:38:41   Backend hangup error handling.
 
2015-02-25 19:38:41   Backend hangup error handling.
 
2015-02-25 19:38:42   Backend hangup error handling.
 
2015-02-25 19:38:42   Backend hangup error handling.
 
2015-02-25 19:38:42   Backend hangup error handling.
 
2015-02-25 19:38:42   Backend hangup error handling.
 
2015-02-25 19:38:43   Backend hangup error handling.
 
2015-02-25 19:38:43   Backend error event handling.
 
2015-02-25 19:38:43   Error : Routing the query failed. Session will be closed.
 
2015-02-25 19:38:43   Backend hangup error handling.
 
2015-02-25 19:38:44   Backend hangup error handling.
 
2015-02-25 19:38:44   Backend hangup error handling.
 
2015-02-25 19:38:44   Backend hangup error handling.
 
2015-02-25 19:38:44   Backend hangup error handling.
 
2015-02-25 19:38:45   Backend hangup error handling.
 
2015-02-25 19:38:45   Backend hangup -> closing session.
 
2015-02-25 19:38:45   Backend hangup -> closing session.
 
2015-02-25 19:38:45   Backend hangup -> closing session.
 
2015-02-25 19:38:45   Backend hangup -> closing session.
 
2015-02-25 19:38:46   Backend hangup error handling.
 
2015-02-25 19:38:46   Error : Routing the query failed. Session will be closed.
 
2015-02-25 19:38:46   Backend hangup error handling.
 
2015-02-25 19:38:47   Backend hangup error handling.
 
2015-02-25 19:38:47   Backend hangup error handling.
 
2015-02-25 19:38:48   Backend hangup error handling.
 
2015-02-25 19:38:48   Backend hangup error handling.
 
2015-02-25 19:38:48   Hangup in session that is not ready for routing, Error reported is 'Broken pipe'.
 
2015-02-25 19:38:48   Backend hangup error handling.
 
2015-02-25 19:38:48   Error : Routing the query failed. Session will be closed.
 
2015-02-25 19:38:48   Backend hangup error handling.
 
2015-02-25 19:38:48   Backend hangup -> closing session.
 
2015-02-25 19:38:49   Backend hangup error handling.
 
2015-02-25 19:38:49   Backend hangup error handling.
 
2015-02-25 19:38:49   Backend hangup error handling.
 
2015-02-25 19:38:50   Backend hangup error handling.
 
2015-02-25 19:38:50   Backend hangup error handling.
 
 
 
 

Starting MaxScale with -lshm option makes tests PASSED

 Comments   
Comment by Dipti Joshi (Inactive) [ 2015-03-09 ]

This is history of comments imported from bugzilla

Comment 1 Timofey Turenko 2015-02-25 22:20:06 UTC

'http://bugs.mariadb.com/show_bug.cgi?id=649' test is also affected by this issue.

Comment 2 Timofey Turenko 2015-02-25 22:44:15 UTC
'short_session' test also fails due to it. If logging is set to file test is extremely slow (less then 7000 'open session, do INSERT one row, close session' cycles per hour) and error log is :

2015-02-26 00:40:00   Backend hangup error handling.
 
2015-02-26 00:40:00   Error : Unable to write to backend due to authentication failure.
 
2015-02-26 00:40:03   Backend hangup error handling.
 
2015-02-26 00:40:03   Backend hangup error handling.
 
2015-02-26 00:40:03   Backend hangup -> closing session.
 
2015-02-26 00:40:03   Backend hangup -> closing session.
 
2015-02-26 00:40:04   Backend hangup error handling.
 
2015-02-26 00:40:04   Backend hangup error handling.
 
2015-02-26 00:40:04   Backend hangup -> closing session.
 
2015-02-26 00:40:06   Backend hangup error handling.
 
2015-02-26 00:40:06   Backend hangup -> closing session.
 
2015-02-26 00:40:07   Backend hangup error handling.
 
2015-02-26 00:40:08   Backend hangup error handling.
 
2015-02-26 00:40:08   Backend hangup -> closing session.
 
2015-02-26 00:40:15   Hangup in session that is not ready for routing, Error reported is 'Broken pipe'.
 
2015-02-26 00:40:18   Backend hangup error handling.
 
2015-02-26 00:40:18   Error : Unable to write to backend due to authentication failure.
 
2015-02-26 00:40:20   Backend hangup error handling.
 
2015-02-26 00:40:21   Error : Unable to write to backend due to authentication failure.
 
2015-02-26 00:40:24   Backend hangup error handling.
 
2015-02-26 00:40:24   Backend hangup -> closing session.
 
2015-02-26 00:40:25   Backend hangup error handling.
 
2015-02-26 00:40:25   Backend hangup -> closing session.
 
2015-02-26 00:40:27   Backend hangup error handling.
 
2015-02-26 00:40:27   Backend hangup -> closing session.
 
2015-02-26 00:40:27   Hangup in session that is not ready for routing, Error reported is 'Broken pipe'.
 
2015-02-26 00:40:28   Hangup in session that is not ready for routing, Error reported is 'Broken pipe'.
 
2015-02-26 00:40:28   Backend hangup error handling.
 
2015-02-26 00:40:28   Backend hangup error handling.
 
2015-02-26 00:40:28   Backend hangup -> closing session.
 
2015-02-26 00:40:29   Backend hangup error handling.
 
2015-02-26 00:40:30   Backend hangup error handling.
 
2015-02-26 00:40:30   Backend hangup error handling.
 
2015-02-26 00:40:30   Backend hangup -> closing session.
 
2015-02-26 00:40:30   Backend hangup -> closing session.
 
2015-02-26 00:40:34   Backend hangup error handling.
 
2015-02-26 00:40:34   Backend hangup error handling.
 
2015-02-26 00:40:34   Backend hangup -> closing session.
 
2015-02-26 00:40:35   Backend hangup error handling.
 
2015-02-26 00:40:35   Error : Unable to write to backend due to authentication failure.
 
2015-02-26 00:40:36   Backend hangup error handling.
 
2015-02-26 00:40:36   Backend hangup -> closing session.
 
2015-02-26 00:40:38   Backend hangup error handling.
 
2015-02-26 00:40:39   RW Split Router: Refresh rate limit exceeded for load of users' table.
 
2015-02-26 00:40:39   Error : Unable to write to backend due to authentication failure.
 
2015-02-26 00:40:43   Backend hangup error handling.
 
2015-02-26 00:40:43   RW Split Router: Refresh rate limit exceeded for load of users' table.
 
2015-02-26 00:40:43   Error : Unable to write to backend due to authentication failure.
 
2015-02-26 00:40:47   Hangup in session that is not ready for routing, Error reported is 'Broken pipe'.
 
2015-02-26 00:40:49   Backend hangup error handling.
 
2015-02-26 00:40:49   Backend hangup -> closing session.
 
2015-02-26 00:40:53   Backend hangup error handling.

Comment 3 Timofey Turenko 2015-02-26 16:29:38 UTC
lowering severity because default logging mode is set to shared memory now. trace and debug logs are enabled only for Debug build, so, no problem for RELEASE build

Comment by markus makela [ 2015-04-21 ]

The failures are partly due to the fact that debug builds of MaxScale generate very large debug logs. For example running the integrated test suite generates roughly 400MB of debug log messages.

Comment by markus makela [ 2015-05-02 ]

This is an example of the verbose nature of the debug and trace log. Executing a single statement causes 32k of data to be written to disk and under stressful testing it will slow MaxScale to a crawl.

[markus@localhost maxscale.15080]$ du -h *
0 debug2.log
0 trace2.log
[markus@localhost maxscale.15080]$ mysql -u maxuser -pmaxpwd -P 4006 -h 127.0.0.1 -e"select 1"
---

1

---

1

---
[markus@localhost maxscale.15080]$ du -h *
28K debug2.log
4.0K trace2.log

Comment by markus makela [ 2015-07-08 ]

Closing this as there is no way to reduce the amount of debug and trace log output without reducing information. The test failures are due to reduced performance and disabling trace and debug log will fix the failure.

Generated at Thu Feb 08 03:56:11 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.