[MXS-564] Loading database dump through readwritesplit fails Created: 2016-02-01  Updated: 2016-02-11  Resolved: 2016-02-11

Status: Closed
Project: MariaDB MaxScale
Component/s: readwritesplit
Affects Version/s: 1.3.0
Fix Version/s: 1.3.0

Type: Bug Priority: Major
Reporter: markus makela Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MXS-570 Loading database dump through readwri... Closed
relates to MXS-559 Crash due to debug assertion in readw... Closed

 Description   

This is a continuation on MXS-559.

Originally reported by engel75:
OK - config looks like:

[galera_5_monitor]
 
type=monitor
 
module=galeramon
 
servers=dbf1,dbf2,dbf3,dbf4,dbf5
 
user=maxscalemon
 
passwd=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 
monitor_interval=2000
 
disable_master_failback=1
 
available_when_donor=1
 
disable_master_role_setting=0
 
backend_connect_timeout=10
 
backend_read_timeout=10

Same result:

 time mysql -h10.0.248.202  -uadmin -pxxxxxxxxxxxxxxxxxx  ops_meta_db < ops_meta_db_2.sql 
ERROR 2003 (HY000) at line 5317: Lost connection to backend server.
 
2016-01-28 09:55:47.849   error  : Backend hangup error handling.
 
Jan 28 09:55:47 wsccms-db05f-prod mysqld: 160128  9:55:47 [Warning] Aborted connection 945714 to db: 'ops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)

Maxscale did not crash, so no core dump.

 Comments   
Comment by Florian Engelmann [ 2016-02-03 ]

Dear markus makela any update from your side? How to move on? Anything I could test for you?

Comment by markus makela [ 2016-02-03 ]

One thing that could be tested is to use only one node in the monitor and in the service. I believe this will give more verbose output because there's no chance of replacing the node.

It would also be useful if you could provide a way to duplicate this so we could investigate it ourselves. Any special characteristics of the database would be useful to know e.g. BLOB or TEXT columns.

Comment by Florian Engelmann [ 2016-02-03 ]

OK is did split the 4.9GB dump into table dumps to figure out which table causes trouble. One of those table dumps is 2.8GB big. That's the one causing trouble. I did try to import the dump twi times now and it failed on different lines:

ERROR 2003 (HY000) at line 2186: Lost connection to backend server.
 
[...]
 
ERROR 2003 (HY000) at line 2152: Lost connection to backend server.
 
[...]

What I am testing right now is your suggestion to enable only one Galera Node for the monitor and the RW split router service. The import is still ongoing...

Comment by Florian Engelmann [ 2016-02-04 ]

After several imports without any error it looks like the RW split router and monitor configured with only one Galera node does not cause any error. I will test with the even bigger dump (4.9GB) now to be sure.

Comment by Florian Engelmann [ 2016-02-04 ]

Same result using the 4.9GB dump - no error.

@markus makela I uploaded the "troublemaker" dump to our filedrop server. You should have received an email by now including all download details. Please let me know if you got any problem downloading the file.

Comment by markus makela [ 2016-02-05 ]

I've downloaded the file successfully and was able to reproduce the failure. It seems to happen even with a regular master-slave cluster when I shut down a slave while the dump is ongoing.

Comment by markus makela [ 2016-02-05 ]

I've found out that the issue relates to the session state modification commands done at the start of a database dump. These commands are stored in MaxScale so that the slaves can execute them when necessary if they are either lagging behind or being replaced. The slaves have a state which signals if a reply from them has been received. Due to the changes made to how the replies are processed, this state was not cleared properly.

The fix is quite simple and I'll provide a package for Ubuntu 14.04 for confirmation ASAP.

Comment by markus makela [ 2016-02-05 ]

Here is a version with the fix implemented: http://maxscale-jenkins.mariadb.com/ci-repository/release-1.3.0-sescmd-fix/mariadb-maxscale/

Comment by Florian Engelmann [ 2016-02-05 ]

sounds awesome! Just installed the fixed package... import is running...

I'll loop it over the weekend:

while true ; do date; time ( cat spider_logs.sql |  mysql -h10.0.248.202  -uadmin -pxxxxxxxxxxxxxxxxxxxxxxxx  ops_meta_db ); done

I will report on Monday - keeping my fingers xed! Thank you for your quick help!

Comment by Florian Engelmann [ 2016-02-05 ]

Dear markus makela great success! The import finished successful! I still get those errors:

2016-02-05 15:15:51.782   error  : Backend hangup error handling.
 
2016-02-05 15:15:51.783   error  : Backend hangup error handling.
 
2016-02-05 15:15:51.784   error  : Backend hangup error handling.
 
2016-02-05 15:15:51.784   error  : Backend hangup error handling.
 
2016-02-05 15:29:23.315   error  : Backend hangup error handling.
 
2016-02-05 15:29:23.315   error  : Backend hangup error handling.
 
2016-02-05 15:29:23.316   error  : Backend hangup error handling.
 
2016-02-05 15:29:23.319   error  : Backend hangup error handling.
 
2016-02-05 15:42:57.576   error  : Backend hangup error handling.
 
2016-02-05 15:42:57.578   error  : Backend hangup error handling.
 
2016-02-05 15:42:57.579   error  : Backend hangup error handling.
 
2016-02-05 15:42:57.579   error  : Backend hangup error handling.
 
2016-02-05 15:56:30.138   error  : Backend hangup error handling.
 
2016-02-05 15:56:30.138   error  : Backend hangup error handling.
 
2016-02-05 15:56:30.139   error  : Backend hangup error handling.
 
2016-02-05 15:56:30.140   error  : Backend hangup error handling.
 
2016-02-05 16:10:06.993   error  : Backend hangup error handling.
 
2016-02-05 16:10:06.993   error  : Backend hangup error handling.
 
2016-02-05 16:10:06.993   error  : Backend hangup error handling.
 
2016-02-05 16:10:07.014   error  : Backend hangup error handling.
 
2016-02-05 16:23:40.733   error  : Backend hangup error handling.
 
2016-02-05 16:23:40.733   error  : Backend hangup error handling.
 
2016-02-05 16:23:40.735   error  : Backend hangup error handling.
 
2016-02-05 16:23:40.736   error  : Backend hangup error handling.
 
2016-02-05 16:37:23.695   error  : Backend hangup error handling.
 
2016-02-05 16:37:23.695   error  : Backend hangup error handling.
 
2016-02-05 16:37:23.697   error  : Backend hangup error handling.
 
2016-02-05 16:37:23.699   error  : Backend hangup error handling.

The syslog of those Galera nodes looks like:

Feb  5 15:15:51 wsccms-db01f-prod mysqld: 160205 15:15:51 [Warning] Aborted connection 2539773 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:29:23 wsccms-db01f-prod mysqld: 160205 15:29:23 [Warning] Aborted connection 2541432 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:42:57 wsccms-db01f-prod mysqld: 160205 15:42:57 [Warning] Aborted connection 2543106 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:56:30 wsccms-db01f-prod mysqld: 160205 15:56:30 [Warning] Aborted connection 2544780 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 16:10:06 wsccms-db01f-prod mysqld: 160205 16:10:06 [Warning] Aborted connection 2546459 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 16:23:40 wsccms-db01f-prod mysqld: 160205 16:23:40 [Warning] Aborted connection 2548143 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 16:37:23 wsccms-db01f-prod mysqld: 160205 16:37:23 [Warning] Aborted connection 2549837 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:15:51 wsccms-db03f-prod mysqld: 160205 15:15:51 [Warning] Aborted connection 1695844 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:29:23 wsccms-db03f-prod mysqld: 160205 15:29:23 [Warning] Aborted connection 1696702 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:42:57 wsccms-db03f-prod mysqld: 160205 15:42:57 [Warning] Aborted connection 1697566 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:56:30 wsccms-db03f-prod mysqld: 160205 15:56:30 [Warning] Aborted connection 1698429 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 16:10:06 wsccms-db03f-prod mysqld: 160205 16:10:06 [Warning] Aborted connection 1699297 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 16:23:40 wsccms-db03f-prod mysqld: 160205 16:23:40 [Warning] Aborted connection 1700171 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 16:37:23 wsccms-db03f-prod mysqld: 160205 16:37:23 [Warning] Aborted connection 1701044 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 12:03:02 wsccms-db04f-prod mysqld: 160205 12:03:02 [Warning] Aborted connection 1683684 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 12:23:14 wsccms-db04f-prod mysqld: 160205 12:23:14 [Warning] Aborted connection 1684983 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 13:58:12 wsccms-db04f-prod mysqld: 160205 13:58:12 [Warning] Aborted connection 1691049 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 14:28:30 wsccms-db04f-prod mysqld: 160205 14:28:30 [Warning] Aborted connection 1692992 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 14:38:36 wsccms-db04f-prod mysqld: 160205 14:38:36 [Warning] Aborted connection 1693635 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 14:48:41 wsccms-db04f-prod mysqld: 160205 14:48:41 [Warning] Aborted connection 1694279 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:05:09 wsccms-db04f-prod mysqld: 160205 15:05:09 [Warning] Aborted connection 1681093 to db: 'unconnected' user: 'maxscalemon' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:15:51 wsccms-db04f-prod mysqld: 160205 15:15:51 [Warning] Aborted connection 1696031 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:29:23 wsccms-db04f-prod mysqld: 160205 15:29:23 [Warning] Aborted connection 1696889 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:42:57 wsccms-db04f-prod mysqld: 160205 15:42:57 [Warning] Aborted connection 1697753 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:56:30 wsccms-db04f-prod mysqld: 160205 15:56:30 [Warning] Aborted connection 1698618 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 16:10:06 wsccms-db04f-prod mysqld: 160205 16:10:06 [Warning] Aborted connection 1699486 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 16:23:40 wsccms-db04f-prod mysqld: 160205 16:23:40 [Warning] Aborted connection 1700360 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 16:37:23 wsccms-db04f-prod mysqld: 160205 16:37:23 [Warning] Aborted connection 1701235 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 11:12:25 wsccms-db05f-prod mysqld: 160205 11:12:25 [Warning] Aborted connection 3727084 to db: 'unconnected' user: 'maxscalemon' host: '10.0.249.211' (Unknown error)
 
Feb  5 11:22:38 wsccms-db05f-prod mysqld: 160205 11:22:38 [Warning] Aborted connection 3798430 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 11:32:43 wsccms-db05f-prod mysqld: 160205 11:32:43 [Warning] Aborted connection 3799067 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 11:42:49 wsccms-db05f-prod mysqld: 160205 11:42:49 [Warning] Aborted connection 3799712 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 11:52:55 wsccms-db05f-prod mysqld: 160205 11:52:55 [Warning] Aborted connection 3800362 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 12:03:01 wsccms-db05f-prod mysqld: 160205 12:03:01 [Warning] Aborted connection 3801007 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 12:13:07 wsccms-db05f-prod mysqld: 160205 12:13:07 [Warning] Aborted connection 3801662 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 12:23:13 wsccms-db05f-prod mysqld: 160205 12:23:13 [Warning] Aborted connection 3802306 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 12:33:19 wsccms-db05f-prod mysqld: 160205 12:33:19 [Warning] Aborted connection 3802949 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 12:43:24 wsccms-db05f-prod mysqld: 160205 12:43:24 [Warning] Aborted connection 3803591 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 12:57:37 wsccms-db05f-prod mysqld: 160205 12:57:37 [Warning] Aborted connection 3804497 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 13:07:43 wsccms-db05f-prod mysqld: 160205 13:07:43 [Warning] Aborted connection 3805143 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 13:17:48 wsccms-db05f-prod mysqld: 160205 13:17:48 [Warning] Aborted connection 3805797 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 13:27:54 wsccms-db05f-prod mysqld: 160205 13:27:54 [Warning] Aborted connection 3806439 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 13:37:59 wsccms-db05f-prod mysqld: 160205 13:37:59 [Warning] Aborted connection 3807082 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 13:48:05 wsccms-db05f-prod mysqld: 160205 13:48:05 [Warning] Aborted connection 3807725 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 13:58:11 wsccms-db05f-prod mysqld: 160205 13:58:11 [Warning] Aborted connection 3808370 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 14:08:17 wsccms-db05f-prod mysqld: 160205 14:08:17 [Warning] Aborted connection 3809011 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 14:18:23 wsccms-db05f-prod mysqld: 160205 14:18:23 [Warning] Aborted connection 3809666 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 14:28:29 wsccms-db05f-prod mysqld: 160205 14:28:29 [Warning] Aborted connection 3810311 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 14:38:34 wsccms-db05f-prod mysqld: 160205 14:38:34 [Warning] Aborted connection 3810954 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 14:48:40 wsccms-db05f-prod mysqld: 160205 14:48:40 [Warning] Aborted connection 3811598 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 14:58:47 wsccms-db05f-prod mysqld: 160205 14:58:47 [Warning] Aborted connection 3812242 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:05:09 wsccms-db05f-prod mysqld: 160205 15:05:09 [Warning] Aborted connection 3798418 to db: 'unconnected' user: 'maxscalemon' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:15:51 wsccms-db05f-prod mysqld: 160205 15:15:51 [Warning] Aborted connection 3813354 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:29:23 wsccms-db05f-prod mysqld: 160205 15:29:23 [Warning] Aborted connection 3814210 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:42:57 wsccms-db05f-prod mysqld: 160205 15:42:57 [Warning] Aborted connection 3815074 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 15:56:30 wsccms-db05f-prod mysqld: 160205 15:56:30 [Warning] Aborted connection 3815939 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 16:10:06 wsccms-db05f-prod mysqld: 160205 16:10:06 [Warning] Aborted connection 3816807 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 16:23:40 wsccms-db05f-prod mysqld: 160205 16:23:40 [Warning] Aborted connection 3817681 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
 
Feb  5 16:37:23 wsccms-db05f-prod mysqld: 160205 16:37:23 [Warning] Aborted connection 3818556 to db: 'xoops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)

It looks like each import causes 4 of those errors. Any idea what that could be?

Comment by Timofey Turenko [ 2016-02-05 ]

I added test 'mxs564_big_dump' to our test suite. Test is executing session command and then sending INSERTs while another thread blocks one or two slaves.
Passed with fix http://max-tst-01.mariadb.com:8089/job/run_test/670/consoleFull and failed with 1.3.0 http://max-tst-01.mariadb.com:8089/job/run_test/671/consoleFull

Interesting thing:
it fails only if only RWSplit router is involved. Second part of the test uses two routers at the same time:

  • connect to RWSplit
  • connect to Read connection router in master mode
  • sends INSERTs in the loop: one to RWSplit, one to ReadConn
    No failure in this case.
Comment by Florian Engelmann [ 2016-02-09 ]

Dear all - any update on this one? Anything we could test within our environment?

Comment by markus makela [ 2016-02-09 ]

I think this particular bug is fixed and the error messages warrant a new bug report. I'll create a new bug report with the details you provided.

The report is here: https://mariadb.atlassian.net/browse/MXS-570

Generated at Thu Feb 08 04:00:16 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.