Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-21

bugzillaId-738: logging to file instead of shared memory breaks 'load_balancing' and 'crash_out_of_files' tests

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Not a Bug
    • 1.0.6
    • 1.1.1
    • Core, Plugin
    • None
    • Linux

    Description

      This is imported from bugzilla. http://bugs.mariadb.com/show_bug.cgi?id=738

      Timofey Turenko 2015-02-25 17:39:22 UTC
      1. start Maxscale with logging to file (now it is default option)
      2. run 'load_balancing' or 'crash_out_of_files' test

      Expected result: test PASSED, not garbage in the logs

      Actual result:

      • test FAILED
      • log is full with:

      015-02-25 19:38:10   Backend hangup error handling.
      2015-02-25 19:38:10   Backend hangup error handling.
      2015-02-25 19:38:10   Backend hangup error handling.
      2015-02-25 19:38:10   Error : Unable to write to backend due to authentication failure.
      2015-02-25 19:38:10   Backend hangup -> closing session.
      2015-02-25 19:38:21   Backend hangup error handling.
      2015-02-25 19:38:22   Backend hangup error handling.
      2015-02-25 19:38:22   Backend hangup error handling.
      2015-02-25 19:38:22   Backend hangup error handling.
      2015-02-25 19:38:22   Backend hangup -> closing session.
      2015-02-25 19:38:24   Backend hangup error handling.
      2015-02-25 19:38:24   Backend hangup -> closing session.
      2015-02-25 19:38:24   Backend hangup error handling.
      2015-02-25 19:38:24   Backend hangup -> closing session.
      2015-02-25 19:38:34   Backend hangup error handling.
      2015-02-25 19:38:34   Backend hangup error handling.
      2015-02-25 19:38:35   Error : Routing the query failed. Session will be closed.
      2015-02-25 19:38:35   Backend hangup error handling.
      2015-02-25 19:38:35   Backend hangup error handling.
      2015-02-25 19:38:35   Error : Routing the query failed. Session will be closed.
      2015-02-25 19:38:35   Backend hangup error handling.
      2015-02-25 19:38:36   Backend hangup error handling.
      2015-02-25 19:38:36   Backend hangup error handling.
      2015-02-25 19:38:36   Backend hangup error handling.
      2015-02-25 19:38:37   Backend hangup error handling.
      2015-02-25 19:38:37   Backend hangup error handling.
      2015-02-25 19:38:37   Error : Routing the query failed. Session will be closed.
      2015-02-25 19:38:37   Backend hangup error handling.
      2015-02-25 19:38:37   Backend hangup error handling.
      2015-02-25 19:38:38   Backend hangup error handling.
      2015-02-25 19:38:38   Backend hangup -> closing session.
      2015-02-25 19:38:39   Backend hangup error handling.
      2015-02-25 19:38:39   Backend hangup error handling.
      2015-02-25 19:38:40   Error: Can't route COM_QUERY:QUERY_TYPE_READ:"SELECT * FROM t1;" to backend server. Router is closed.
      2015-02-25 19:38:40   Error : Routing the query failed. Session will be closed.
      2015-02-25 19:38:40   Backend hangup -> closing session.
      2015-02-25 19:38:40   Backend error event handling.
      2015-02-25 19:38:40   Backend hangup -> closing session.
      2015-02-25 19:38:41   Backend hangup error handling.
      2015-02-25 19:38:41   Backend hangup -> closing session.
      2015-02-25 19:38:41   Backend hangup error handling.
      2015-02-25 19:38:41   Backend hangup error handling.
      2015-02-25 19:38:41   Backend hangup error handling.
      2015-02-25 19:38:42   Backend hangup error handling.
      2015-02-25 19:38:42   Backend hangup error handling.
      2015-02-25 19:38:42   Backend hangup error handling.
      2015-02-25 19:38:42   Backend hangup error handling.
      2015-02-25 19:38:43   Backend hangup error handling.
      2015-02-25 19:38:43   Backend error event handling.
      2015-02-25 19:38:43   Error : Routing the query failed. Session will be closed.
      2015-02-25 19:38:43   Backend hangup error handling.
      2015-02-25 19:38:44   Backend hangup error handling.
      2015-02-25 19:38:44   Backend hangup error handling.
      2015-02-25 19:38:44   Backend hangup error handling.
      2015-02-25 19:38:44   Backend hangup error handling.
      2015-02-25 19:38:45   Backend hangup error handling.
      2015-02-25 19:38:45   Backend hangup -> closing session.
      2015-02-25 19:38:45   Backend hangup -> closing session.
      2015-02-25 19:38:45   Backend hangup -> closing session.
      2015-02-25 19:38:45   Backend hangup -> closing session.
      2015-02-25 19:38:46   Backend hangup error handling.
      2015-02-25 19:38:46   Error : Routing the query failed. Session will be closed.
      2015-02-25 19:38:46   Backend hangup error handling.
      2015-02-25 19:38:47   Backend hangup error handling.
      2015-02-25 19:38:47   Backend hangup error handling.
      2015-02-25 19:38:48   Backend hangup error handling.
      2015-02-25 19:38:48   Backend hangup error handling.
      2015-02-25 19:38:48   Hangup in session that is not ready for routing, Error reported is 'Broken pipe'.
      2015-02-25 19:38:48   Backend hangup error handling.
      2015-02-25 19:38:48   Error : Routing the query failed. Session will be closed.
      2015-02-25 19:38:48   Backend hangup error handling.
      2015-02-25 19:38:48   Backend hangup -> closing session.
      2015-02-25 19:38:49   Backend hangup error handling.
      2015-02-25 19:38:49   Backend hangup error handling.
      2015-02-25 19:38:49   Backend hangup error handling.
      2015-02-25 19:38:50   Backend hangup error handling.
      2015-02-25 19:38:50   Backend hangup error handling.
       
       

      Starting MaxScale with -lshm option makes tests PASSED

      Attachments

        Activity

          This is history of comments imported from bugzilla

          Comment 1 Timofey Turenko 2015-02-25 22:20:06 UTC

          'http://bugs.mariadb.com/show_bug.cgi?id=649' test is also affected by this issue.

          Comment 2 Timofey Turenko 2015-02-25 22:44:15 UTC
          'short_session' test also fails due to it. If logging is set to file test is extremely slow (less then 7000 'open session, do INSERT one row, close session' cycles per hour) and error log is :

          2015-02-26 00:40:00   Backend hangup error handling.
          2015-02-26 00:40:00   Error : Unable to write to backend due to authentication failure.
          2015-02-26 00:40:03   Backend hangup error handling.
          2015-02-26 00:40:03   Backend hangup error handling.
          2015-02-26 00:40:03   Backend hangup -> closing session.
          2015-02-26 00:40:03   Backend hangup -> closing session.
          2015-02-26 00:40:04   Backend hangup error handling.
          2015-02-26 00:40:04   Backend hangup error handling.
          2015-02-26 00:40:04   Backend hangup -> closing session.
          2015-02-26 00:40:06   Backend hangup error handling.
          2015-02-26 00:40:06   Backend hangup -> closing session.
          2015-02-26 00:40:07   Backend hangup error handling.
          2015-02-26 00:40:08   Backend hangup error handling.
          2015-02-26 00:40:08   Backend hangup -> closing session.
          2015-02-26 00:40:15   Hangup in session that is not ready for routing, Error reported is 'Broken pipe'.
          2015-02-26 00:40:18   Backend hangup error handling.
          2015-02-26 00:40:18   Error : Unable to write to backend due to authentication failure.
          2015-02-26 00:40:20   Backend hangup error handling.
          2015-02-26 00:40:21   Error : Unable to write to backend due to authentication failure.
          2015-02-26 00:40:24   Backend hangup error handling.
          2015-02-26 00:40:24   Backend hangup -> closing session.
          2015-02-26 00:40:25   Backend hangup error handling.
          2015-02-26 00:40:25   Backend hangup -> closing session.
          2015-02-26 00:40:27   Backend hangup error handling.
          2015-02-26 00:40:27   Backend hangup -> closing session.
          2015-02-26 00:40:27   Hangup in session that is not ready for routing, Error reported is 'Broken pipe'.
          2015-02-26 00:40:28   Hangup in session that is not ready for routing, Error reported is 'Broken pipe'.
          2015-02-26 00:40:28   Backend hangup error handling.
          2015-02-26 00:40:28   Backend hangup error handling.
          2015-02-26 00:40:28   Backend hangup -> closing session.
          2015-02-26 00:40:29   Backend hangup error handling.
          2015-02-26 00:40:30   Backend hangup error handling.
          2015-02-26 00:40:30   Backend hangup error handling.
          2015-02-26 00:40:30   Backend hangup -> closing session.
          2015-02-26 00:40:30   Backend hangup -> closing session.
          2015-02-26 00:40:34   Backend hangup error handling.
          2015-02-26 00:40:34   Backend hangup error handling.
          2015-02-26 00:40:34   Backend hangup -> closing session.
          2015-02-26 00:40:35   Backend hangup error handling.
          2015-02-26 00:40:35   Error : Unable to write to backend due to authentication failure.
          2015-02-26 00:40:36   Backend hangup error handling.
          2015-02-26 00:40:36   Backend hangup -> closing session.
          2015-02-26 00:40:38   Backend hangup error handling.
          2015-02-26 00:40:39   RW Split Router: Refresh rate limit exceeded for load of users' table.
          2015-02-26 00:40:39   Error : Unable to write to backend due to authentication failure.
          2015-02-26 00:40:43   Backend hangup error handling.
          2015-02-26 00:40:43   RW Split Router: Refresh rate limit exceeded for load of users' table.
          2015-02-26 00:40:43   Error : Unable to write to backend due to authentication failure.
          2015-02-26 00:40:47   Hangup in session that is not ready for routing, Error reported is 'Broken pipe'.
          2015-02-26 00:40:49   Backend hangup error handling.
          2015-02-26 00:40:49   Backend hangup -> closing session.
          2015-02-26 00:40:53   Backend hangup error handling.

          Comment 3 Timofey Turenko 2015-02-26 16:29:38 UTC
          lowering severity because default logging mode is set to shared memory now. trace and debug logs are enabled only for Debug build, so, no problem for RELEASE build

          dshjoshi Dipti Joshi (Inactive) added a comment - This is history of comments imported from bugzilla Comment 1 Timofey Turenko 2015-02-25 22:20:06 UTC 'http://bugs.mariadb.com/show_bug.cgi?id=649' test is also affected by this issue. Comment 2 Timofey Turenko 2015-02-25 22:44:15 UTC 'short_session' test also fails due to it. If logging is set to file test is extremely slow (less then 7000 'open session, do INSERT one row, close session' cycles per hour) and error log is : 2015-02-26 00:40:00 Backend hangup error handling. 2015-02-26 00:40:00 Error : Unable to write to backend due to authentication failure. 2015-02-26 00:40:03 Backend hangup error handling. 2015-02-26 00:40:03 Backend hangup error handling. 2015-02-26 00:40:03 Backend hangup -> closing session. 2015-02-26 00:40:03 Backend hangup -> closing session. 2015-02-26 00:40:04 Backend hangup error handling. 2015-02-26 00:40:04 Backend hangup error handling. 2015-02-26 00:40:04 Backend hangup -> closing session. 2015-02-26 00:40:06 Backend hangup error handling. 2015-02-26 00:40:06 Backend hangup -> closing session. 2015-02-26 00:40:07 Backend hangup error handling. 2015-02-26 00:40:08 Backend hangup error handling. 2015-02-26 00:40:08 Backend hangup -> closing session. 2015-02-26 00:40:15 Hangup in session that is not ready for routing, Error reported is 'Broken pipe'. 2015-02-26 00:40:18 Backend hangup error handling. 2015-02-26 00:40:18 Error : Unable to write to backend due to authentication failure. 2015-02-26 00:40:20 Backend hangup error handling. 2015-02-26 00:40:21 Error : Unable to write to backend due to authentication failure. 2015-02-26 00:40:24 Backend hangup error handling. 2015-02-26 00:40:24 Backend hangup -> closing session. 2015-02-26 00:40:25 Backend hangup error handling. 2015-02-26 00:40:25 Backend hangup -> closing session. 2015-02-26 00:40:27 Backend hangup error handling. 2015-02-26 00:40:27 Backend hangup -> closing session. 2015-02-26 00:40:27 Hangup in session that is not ready for routing, Error reported is 'Broken pipe'. 2015-02-26 00:40:28 Hangup in session that is not ready for routing, Error reported is 'Broken pipe'. 2015-02-26 00:40:28 Backend hangup error handling. 2015-02-26 00:40:28 Backend hangup error handling. 2015-02-26 00:40:28 Backend hangup -> closing session. 2015-02-26 00:40:29 Backend hangup error handling. 2015-02-26 00:40:30 Backend hangup error handling. 2015-02-26 00:40:30 Backend hangup error handling. 2015-02-26 00:40:30 Backend hangup -> closing session. 2015-02-26 00:40:30 Backend hangup -> closing session. 2015-02-26 00:40:34 Backend hangup error handling. 2015-02-26 00:40:34 Backend hangup error handling. 2015-02-26 00:40:34 Backend hangup -> closing session. 2015-02-26 00:40:35 Backend hangup error handling. 2015-02-26 00:40:35 Error : Unable to write to backend due to authentication failure. 2015-02-26 00:40:36 Backend hangup error handling. 2015-02-26 00:40:36 Backend hangup -> closing session. 2015-02-26 00:40:38 Backend hangup error handling. 2015-02-26 00:40:39 RW Split Router: Refresh rate limit exceeded for load of users' table. 2015-02-26 00:40:39 Error : Unable to write to backend due to authentication failure. 2015-02-26 00:40:43 Backend hangup error handling. 2015-02-26 00:40:43 RW Split Router: Refresh rate limit exceeded for load of users' table. 2015-02-26 00:40:43 Error : Unable to write to backend due to authentication failure. 2015-02-26 00:40:47 Hangup in session that is not ready for routing, Error reported is 'Broken pipe'. 2015-02-26 00:40:49 Backend hangup error handling. 2015-02-26 00:40:49 Backend hangup -> closing session. 2015-02-26 00:40:53 Backend hangup error handling. Comment 3 Timofey Turenko 2015-02-26 16:29:38 UTC lowering severity because default logging mode is set to shared memory now. trace and debug logs are enabled only for Debug build, so, no problem for RELEASE build
          markus makela markus makela added a comment -

          The failures are partly due to the fact that debug builds of MaxScale generate very large debug logs. For example running the integrated test suite generates roughly 400MB of debug log messages.

          markus makela markus makela added a comment - The failures are partly due to the fact that debug builds of MaxScale generate very large debug logs. For example running the integrated test suite generates roughly 400MB of debug log messages.
          markus makela markus makela added a comment - - edited

          This is an example of the verbose nature of the debug and trace log. Executing a single statement causes 32k of data to be written to disk and under stressful testing it will slow MaxScale to a crawl.

          [markus@localhost maxscale.15080]$ du -h *
          0 debug2.log
          0 trace2.log
          [markus@localhost maxscale.15080]$ mysql -u maxuser -pmaxpwd -P 4006 -h 127.0.0.1 -e"select 1"
          ---

          1

          ---

          1

          ---
          [markus@localhost maxscale.15080]$ du -h *
          28K debug2.log
          4.0K trace2.log

          markus makela markus makela added a comment - - edited This is an example of the verbose nature of the debug and trace log. Executing a single statement causes 32k of data to be written to disk and under stressful testing it will slow MaxScale to a crawl. [markus@localhost maxscale.15080] $ du -h * 0 debug2.log 0 trace2.log [markus@localhost maxscale.15080] $ mysql -u maxuser -pmaxpwd -P 4006 -h 127.0.0.1 -e"select 1" --- 1 --- 1 --- [markus@localhost maxscale.15080] $ du -h * 28K debug2.log 4.0K trace2.log
          markus makela markus makela added a comment -

          Closing this as there is no way to reduce the amount of debug and trace log output without reducing information. The test failures are due to reduced performance and disabling trace and debug log will fix the failure.

          markus makela markus makela added a comment - Closing this as there is no way to reduce the amount of debug and trace log output without reducing information. The test failures are due to reduced performance and disabling trace and debug log will fix the failure.

          People

            markus makela markus makela
            tturenko Timofey Turenko
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.