Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-2079

Closing slave connection due to inconsistent session state

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 2.2.15
    • 2.3.1
    • readwritesplit
    • None

    Description

      Folks,

      Since older versions of Maxscale, we have been getting the below log entries:

      2018-10-04 07:00:38.382   warning: (452953) [readwritesplit] (discard_if_response_differs): 
      Slave server 'server-gal-a': response (0xff) differs from master's response (0x00) to 
      COM_QUERY: `SELECT DISTINCT ti.hdr_id,IF(ti.is_void AND void_bus_date<='2018-10-
      03',void_hdr_new_status_id,hdr_new_status_id) AS hdr_new_status_id, ti.hdr_new_sub_status_id As 
      hdr_new_sub_status_id INTO OUTFILE '/tmp/iecljfsjpe_6084_hdr.csv' FIELDS TERMINATED BY ',' 
      OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n' FROM trans_item AS ti INNER JOIN 
      _ti_ids_iecljfsjpe_6084 AS tmp ON ti.id=tmp.ti_id INNER JOIN trans_hdr AS th ON ti.hdr_id=th.id 
      HAVING hdr_new_status_id IN (1,4,5,18,19,20,21,27) `. Closing slave connection due to 
      inconsistent session state.
      

      All occurrences in the log file currently have only queries with SELECT INTO OUTFILE.

      Could you please have a look at that?

      Thanks!

      Attachments

        Issue Links

          Activity

            We think this bug might be the cause of this ticket:

            https://support.mariadb.com/view.php?id=23932

            ivenn Isaac Venn (Inactive) added a comment - We think this bug might be the cause of this ticket: https://support.mariadb.com/view.php?id=23932

            We think this is related to the support ticket because of the following errors found in the log:

            Nov 16 03:00:44 916567-MXS-D01 maxscale[64511]: (2173989) [readwritesplit] Slave server 'db02': response (0xff) differs from master's response (0x00) to COM_QUERY: `SELECT DISTINCT ti.hdr_id,IF(ti.is_void AND void_bus_date<='2018-11-16',void_hdr_new_status_id,hdr_new_status_id) AS hdr_new_status_id, ti.hdr_new_sub_status_id As hdr_new_sub_status_id INTO OUTFILE '/tmp/flppssmqjf_8197540_hdr.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n' FROM trans_item AS ti INNER JOIN _ti_ids_flppssmqjf_8197540 AS tmp ON ti.id=tmp.ti_id INNER JOIN trans_hdr AS th ON ti.hdr_id=th.id HAVING hdr_new_status_id IN (1) `. Closing slave connection due to inconsistent session state.
            Nov 16 03:00:44 916567-MXS-D01 maxscale[64511]: (2173989) [readwritesplit] Slave server 'db03': response (0xff) differs from master's response (0x00) to COM_QUERY: `SELECT DISTINCT ti.hdr_id,IF(ti.is_void AND void_bus_date<='2018-11-16',void_hdr_new_status_id,hdr_new_status_id) AS hdr_new_status_id, ti.hdr_new_sub_status_id As hdr_new_sub_status_id INTO OUTFILE '/tmp/flppssmqjf_8197540_hdr.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n' FROM trans_item AS ti INNER JOIN _ti_ids_flppssmqjf_8197540 AS tmp ON ti.id=tmp.ti_id INNER JOIN trans_hdr AS th ON ti.hdr_id=th.id HAVING hdr_new_status_id IN (1) `. Closing slave connection due to inconsistent session state.
            Nov 16 03:00:44 916567-MXS-D01 maxscale[64511]: (2173989) [readwritesplit] Slave server 'db05': response (0xff) differs from master's response (0x00) to COM_QUERY: `SELECT DISTINCT ti.hdr_id,IF(ti.is_void AND void_bus_date<='2018-11-16',void_hdr_new_status_id,hdr_new_status_id) AS hdr_new_status_id, ti.hdr_new_sub_status_id As hdr_new_sub_status_id INTO OUTFILE '/tmp/flppssmqjf_8197540_hdr.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n' FROM trans_item AS ti INNER JOIN _ti_ids_flppssmqjf_8197540 AS tmp ON ti.id=tmp.ti_id INNER JOIN trans_hdr AS th ON ti.hdr_id=th.id HAVING hdr_new_status_id IN (1) `. Closing slave connection due to inconsistent session state.
            Nov 16 03:00:44 916567-MXS-D01 maxscale[64511]: (2173989) [readwritesplit] Slave server 'db06': response (0xff) differs from master's response (0x00) to COM_QUERY: `SELECT DISTINCT ti.hdr_id,IF(ti.is_void AND void_bus_date<='2018-11-16',void_hdr_new_status_id,hdr_new_status_id) AS hdr_new_status_id, ti.hdr_new_sub_status_id As hdr_new_sub_status_id INTO OUTFILE '/tmp/flppssmqjf_8197540_hdr.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n' FROM trans_item AS ti INNER JOIN _ti_ids_flppssmqjf_8197540 AS tmp ON ti.id=tmp.ti_id INNER JOIN trans_hdr AS th ON ti.hdr_id=th.id HAVING hdr_new_status_id IN (1) `. Closing slave connection due to inconsistent session state.

            Following this error, we show a much higher percentage of queries going to the master host that would normally be handled by slave servers. We think this is because the connections to the slaves are listed as blocked (temporarily?). Client is concerned because of the very uneven load and because the have seen the master go offline because of load.

            Example query going to master that should be routed to slave:

            Nov 16 12:39:37 916567-MXS-D01 maxscale[64511]: (2216337) [readwritesplit] > Autocommit: [enabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 2245, type: QUERY_TYPE_READ, stmt: SELECT tv.id,tv.company_id,tv.store_id,tv.customer_id,tv.prop_type,tv.vehicle_type,tv.coowner_id,tv.created_dt,tv.created_user_id,tv.updated_dt,#015#012 tv.updated_user_id,tv.owner_name,tv.v_bodystyle_id,tv.v_year,tv.v_make,tv.v_model,tv.v_color,tv.v_vin,IFNULL(tv.v_titlenum,' ') AS v_titlenum,#015#012 IFNULL(tv.v_tagnum,'') AS v_tagnum,tv.v_tag_state_id,tv.v_tag_county,tv.v_tag_expire_date,tv.ins_type,tv.ins_company,tv.ins_policy_num,#015#012 tv.serial_num, tv.bb_series, tv.bb_style, tv.est_value_src, tv.est_value, #015#012 tv.cvr_control_num,tv.cvr_request_num,tv.cvr_stateapp_num,tv.cvr_last_update,tv.max_loan_amt,tv.v_condition AS condition_text,#015#012 CAST(v_condition As UNSIGNED) As v_condition, tv.mileage, tv.description, tv.fee_percent, tv.fee_type, tv.have_key, tv.lien_placed, tv.has_writing, tv.sold_ttl_id, #015#012 IF(tv.prop_type=2,tv.v_vin,tv.serial_num) AS vin_serial,CASE prop_type WHEN 1 THEN 'PP' WHEN 2 THEN 'MP' END AS prop_type_txt,#015#012
            Nov 16 12:39:37 916567-MXS-D01 maxscale[64511]: (2216337) [readwritesplit] Route query to master #011[172.24.100.81]:3306 <

            We have verified these queries are not being wrapped in transactions.

            ivenn Isaac Venn (Inactive) added a comment - We think this is related to the support ticket because of the following errors found in the log: Nov 16 03:00:44 916567-MXS-D01 maxscale [64511] : (2173989) [readwritesplit] Slave server 'db02': response (0xff) differs from master's response (0x00) to COM_QUERY: `SELECT DISTINCT ti.hdr_id,IF(ti.is_void AND void_bus_date<='2018-11-16',void_hdr_new_status_id,hdr_new_status_id) AS hdr_new_status_id, ti.hdr_new_sub_status_id As hdr_new_sub_status_id INTO OUTFILE '/tmp/flppssmqjf_8197540_hdr.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n' FROM trans_item AS ti INNER JOIN _ti_ids_flppssmqjf_8197540 AS tmp ON ti.id=tmp.ti_id INNER JOIN trans_hdr AS th ON ti.hdr_id=th.id HAVING hdr_new_status_id IN (1) `. Closing slave connection due to inconsistent session state. Nov 16 03:00:44 916567-MXS-D01 maxscale [64511] : (2173989) [readwritesplit] Slave server 'db03': response (0xff) differs from master's response (0x00) to COM_QUERY: `SELECT DISTINCT ti.hdr_id,IF(ti.is_void AND void_bus_date<='2018-11-16',void_hdr_new_status_id,hdr_new_status_id) AS hdr_new_status_id, ti.hdr_new_sub_status_id As hdr_new_sub_status_id INTO OUTFILE '/tmp/flppssmqjf_8197540_hdr.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n' FROM trans_item AS ti INNER JOIN _ti_ids_flppssmqjf_8197540 AS tmp ON ti.id=tmp.ti_id INNER JOIN trans_hdr AS th ON ti.hdr_id=th.id HAVING hdr_new_status_id IN (1) `. Closing slave connection due to inconsistent session state. Nov 16 03:00:44 916567-MXS-D01 maxscale [64511] : (2173989) [readwritesplit] Slave server 'db05': response (0xff) differs from master's response (0x00) to COM_QUERY: `SELECT DISTINCT ti.hdr_id,IF(ti.is_void AND void_bus_date<='2018-11-16',void_hdr_new_status_id,hdr_new_status_id) AS hdr_new_status_id, ti.hdr_new_sub_status_id As hdr_new_sub_status_id INTO OUTFILE '/tmp/flppssmqjf_8197540_hdr.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n' FROM trans_item AS ti INNER JOIN _ti_ids_flppssmqjf_8197540 AS tmp ON ti.id=tmp.ti_id INNER JOIN trans_hdr AS th ON ti.hdr_id=th.id HAVING hdr_new_status_id IN (1) `. Closing slave connection due to inconsistent session state. Nov 16 03:00:44 916567-MXS-D01 maxscale [64511] : (2173989) [readwritesplit] Slave server 'db06': response (0xff) differs from master's response (0x00) to COM_QUERY: `SELECT DISTINCT ti.hdr_id,IF(ti.is_void AND void_bus_date<='2018-11-16',void_hdr_new_status_id,hdr_new_status_id) AS hdr_new_status_id, ti.hdr_new_sub_status_id As hdr_new_sub_status_id INTO OUTFILE '/tmp/flppssmqjf_8197540_hdr.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n' FROM trans_item AS ti INNER JOIN _ti_ids_flppssmqjf_8197540 AS tmp ON ti.id=tmp.ti_id INNER JOIN trans_hdr AS th ON ti.hdr_id=th.id HAVING hdr_new_status_id IN (1) `. Closing slave connection due to inconsistent session state. Following this error, we show a much higher percentage of queries going to the master host that would normally be handled by slave servers. We think this is because the connections to the slaves are listed as blocked (temporarily?). Client is concerned because of the very uneven load and because the have seen the master go offline because of load. Example query going to master that should be routed to slave: Nov 16 12:39:37 916567-MXS-D01 maxscale [64511] : (2216337) [readwritesplit] > Autocommit: [enabled] , trx is [not open] , cmd: (0x03) COM_QUERY, plen: 2245, type: QUERY_TYPE_READ, stmt: SELECT tv.id,tv.company_id,tv.store_id,tv.customer_id,tv.prop_type,tv.vehicle_type,tv.coowner_id,tv.created_dt,tv.created_user_id,tv.updated_dt,#015#012 tv.updated_user_id,tv.owner_name,tv.v_bodystyle_id,tv.v_year,tv.v_make,tv.v_model,tv.v_color,tv.v_vin,IFNULL(tv.v_titlenum,' ') AS v_titlenum,#015#012 IFNULL(tv.v_tagnum,'') AS v_tagnum,tv.v_tag_state_id,tv.v_tag_county,tv.v_tag_expire_date,tv.ins_type,tv.ins_company,tv.ins_policy_num,#015#012 tv.serial_num, tv.bb_series, tv.bb_style, tv.est_value_src, tv.est_value, #015#012 tv.cvr_control_num,tv.cvr_request_num,tv.cvr_stateapp_num,tv.cvr_last_update,tv.max_loan_amt,tv.v_condition AS condition_text,#015#012 CAST(v_condition As UNSIGNED) As v_condition, tv.mileage, tv.description, tv.fee_percent, tv.fee_type, tv.have_key, tv.lien_placed, tv.has_writing, tv.sold_ttl_id, #015#012 IF(tv.prop_type=2,tv.v_vin,tv.serial_num) AS vin_serial,CASE prop_type WHEN 1 THEN 'PP' WHEN 2 THEN 'MP' END AS prop_type_txt,#015#012 Nov 16 12:39:37 916567-MXS-D01 maxscale [64511] : (2216337) [readwritesplit] Route query to master #011 [172.24.100.81] :3306 < We have verified these queries are not being wrapped in transactions.
            markus makela markus makela added a comment -

            MXS-2096 should fix this.

            markus makela markus makela added a comment - MXS-2096 should fix this.

            People

              markus makela markus makela
              wagnerbianchi Wagner Bianchi (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.