[MCOL-400] On a 2PM combo stack with query stats enabled, one of the PMs returned an error when queries is executed Created: 2016-11-10  Updated: 2020-08-25  Resolved: 2017-08-16

Status: Closed
Project: MariaDB ColumnStore
Component/s: ExeMgr
Affects Version/s: 1.0.4
Fix Version/s: 1.0.11, 1.1.0

Type: Bug Priority: Minor
Reporter: Daniel Lee (Inactive) Assignee: David Hall (Inactive)
Resolution: Fixed Votes: 1
Labels: None

Sprint: 2016-22, 2016-23, 2016-24, 2016-25, 2017-16

 Description   

Build tested: 1.0.4-1 beta

I setup a combo stack with 2 PMs, then I changed parameters in Columnstore.xml to enable query stats:
<CrossEngineSupport>
<Host>localhost</Host>
<Port>3306</Port>
<User>root</User>
<Password/>
</CrossEngineSupport>
<QueryStats>
<Enabled>Y</Enabled>
</QueryStats>

After the above change, PM2 worked as expected and queries are being logged in infinidb_querystats.querystats table, but PM2 would return an warning. The executed query was not log in either of the PMs.
MariaDB [mytest]> select count from orders;
----------

count

----------

1500000

----------
1 row in set, 1 warning (0.05 sec)

MariaDB [mytest]> show warnings;
-----------------------------------------------------------------------------------------------------------

Level Code Message

-----------------------------------------------------------------------------------------------------------

Note 1051 Unknown table 'infinidb_vtable.$vtable_16'
Warning 9999 Columnstore Query Stats - IDB-8002: Error: fatal error executing query in querystats lib (17).

-----------------------------------------------------------------------------------------------------------
2 rows in set (0.00 sec)

MariaDB [mytest]> select * from infinidb_querystats.querystats;
Empty set (0.00 sec)

MariaDB [mytest]> quit



 Comments   
Comment by David Thompson (Inactive) [ 2016-11-29 ]

Can still reproduce this, the description is unclear but in my case pm1 would show the warning and have no stats and pm2 is fine (which i think is what Daniel meant). Nothing obvious in logs.

Comment by David Thompson (Inactive) [ 2017-01-04 ]

This is tied up with cross engine join visibility. If you change the cross engine join config to use the pm1 hostname then pm1 works but now pm2 no longer works.

Further there seems to be an error condition when the insert into querystats is performed from a remote host, e.g. pm2 exemgr:

> insert delayed into querystats values (0, 123, "centos2", "root", "1", "ABC", "select 1 from foo","2017-01-01", "2017-01-01",1,123, 0, 0, 1,2,1,2,3,4,5,6);
ERROR 1616 (HY000): DELAYED option not supported for table 'querystats'

One question is why does a query executed on um/pm 1 always want to use the exemgr on pm2? It would seem more efficient to use the local one and then a localhost cross engine join config would work.

Comment by David Thompson (Inactive) [ 2017-01-04 ]

The system does round robin between exemgrs and this is also broken in infinidb. This will require some architectural changes so is more than just a small maintenance release bug fix. Essentially QueryStats will only work fully in an environment with one UM. It will partially record on a multi um deployment.

Comment by David Thompson (Inactive) [ 2017-08-08 ]

The actual root cause is due to a porting issue to mariadb. The querystats table was updated to use InnoDB but the querystats insert statement uses the insert delayed syntax which is not supported by InnoDB. Thus the fix for this is to either revert to using a MyISAM table or update the insert statement to remove the delayed keyword.

A workaround for now is to migrate the querystats table to use MyISAM after installation:

 alter table infinidb_querystats.querystats engine=MyISAM;

Comment by David Hall (Inactive) [ 2017-08-15 ]

The change for develop was mistakenly directly cherry-picked in, so there's no pull request for that, only for develop-1.0

Comment by Daniel Lee (Inactive) [ 2017-08-16 ]

Builds tested: 1.0.1-1 and 1.1.0-1 (GitHub source build)

1.0.1-1
On a combo stack (PM1 and PM2), DML sync is one way, from PM1 to PM2. With querystats enabled, when executing a query on PM1, querystats on both PM1 and PM2 gets a new row. This is the expected behavior. When executing a query on PM2, only querystats on PM2 gets the new row. With the current implementation, this is to be expected. But I don't think this is the EXPECTED behavior.

1.1.0

When querystats is enabled, mysqld crashes on any user query.

MariaDB [mytest]> select count from orders;
ERROR 2013 (HY000): Lost connection to MySQL server during query

No entries recorded in err.log and crit.log. mysqld gets a new PID. This issue occurs on both PMs.

Comment by Daniel Lee (Inactive) [ 2017-08-16 ]

reopen per my last comment

Comment by David Thompson (Inactive) [ 2017-08-16 ]

Please open a new bug for 1.1.0 and then we can remove the fixed in here.

For 1.0.x this is the correct behavior based on having the default setup of replication on and using localhost as the cross engine join host.

Comment by Daniel Lee (Inactive) [ 2017-08-16 ]

opened ticket MCOL-879 to track the mysqld crashing issue. Closing the ticket for 1.0.11-1

Generated at Thu Feb 08 02:20:47 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.