Hi,
After doing some close inspection on the core dump and experimenting with mtr, I managed to come up with a reproducer test case:
--source include/galera_cluster.inc
|
--source include/have_innodb.inc
|
--source include/have_debug_sync.inc
|
--source include/galera_have_debug_sync.inc
|
|
|
# Setup
|
CREATE TABLE t1 (f1 VARCHAR(32) NOT NULL) ENGINE=InnoDB; |
INSERT INTO t1 (f1) VALUES ('0e66c5227a8a'); |
INSERT INTO t1 (f1) VALUES ('c6c112992c9'); |
|
|
--connection node_2
|
--let $wait_condition = SELECT COUNT(*)=2 FROM t1;
|
--source include/wait_condition.inc
|
|
|
# Ensure that we have enough applier threads to process transactions in parallel |
SET GLOBAL wsrep_slave_threads = 2; |
|
|
# Set up a synchronization point to catch the first transaction |
--let $galera_sync_point = apply_monitor_slave_enter_sync
|
--source include/galera_set_sync_point.inc
|
|
|
--connection node_1
|
# Invoke the first transaction |
START TRANSACTION; |
UPDATE t1 SET f1='5ffceebfada' WHERE t1.f1 = 'c6c112992c9'; |
COMMIT; |
|
|
--connection node_2
|
# Wait for the first transaction to appear |
SET SESSION wsrep_sync_wait = 0; |
--let $galera_sync_point = apply_monitor_slave_enter_sync
|
--source include/galera_wait_sync_point.inc
|
|
|
# Clear the previous syncronizaton point and setup a different one for the |
# second transaction |
--source include/galera_clear_sync_point.inc
|
SET GLOBAL DEBUG_DBUG = "d,sync.wsrep_apply_cb"; |
|
|
--connection node_1
|
# Invoke the second transaction |
START TRANSACTION; |
UPDATE t1 SET f1='4ffceebfcdc' WHERE t1.f1 = '0e66c5227a8a'; |
COMMIT; |
|
|
--connection node_2
|
# Wait for the second transaction to appear |
SET SESSION DEBUG_SYNC = "now WAIT_FOR sync.wsrep_apply_cb_reached"; |
|
|
# Remove syncronization point
|
SET GLOBAL DEBUG_DBUG = ""; |
|
|
# Setup an additional syncronization point for the second transaction |
# in commit stage |
--let $galera_sync_point = commit_monitor_slave_enter_sync
|
--source include/galera_set_sync_point.inc
|
|
|
# Let the second transaction to proceed |
SET DEBUG_SYNC = "now SIGNAL signal.wsrep_apply_cb"; |
|
|
# Wait for the second transaction to obtain lock |
--let $galera_sync_point = apply_monitor_slave_enter_sync commit_monitor_slave_enter_sync
|
--source include/galera_wait_sync_point.inc
|
|
|
# Now, the second transaction is ahead of the first one |
|
|
# Clear the syncronization point
|
--source include/galera_clear_sync_point.inc
|
|
|
# Let the first transaction to proceed |
--let $galera_sync_point = apply_monitor_slave_enter_sync
|
--source include/galera_signal_sync_point.inc
|
|
|
# Crash occurs at this point |
# int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed. |
|
|
# Teardown [incomplete...]
|
--connection node_1
|
SET GLOBAL wsrep_slave_threads = DEFAULT; |
|
|
DROP TABLE t1; |
I'll try to fiddle with the code, but I would really appreciate if someone could take a look on this one.
Best regards,
GOro
Hi,
Executed the above mentioned test case on the 10.4 branch head (0cc811c633d1fe5290b10fa49fad0a4b889383fa) and it crashes at a different point:
mysqld: /var/lib/src/mariadb/sql/service_wsrep.cc:271: my_bool wsrep_thd_is_aborting(const THD*): Assertion `(&(&thd->LOCK_thd_data)->m_mutex)->count > 0 && pthread_equal(pthread_self(), (&(&thd->LOCK_thd_data)->m_mutex)->thread)' failed.
|
Best regards,
GOro
@jplindst, sorry for the mention. I saw that you did quite a lot of changes on this area recently that also shifted the actual assertion failure to a different place. Could you please take a look on this one and comment?
Thanks for your help in advance!
Best regards,
GOro
Hi,
Just to summarize my findings, I think there are basically two scenarios that can lead to this kind of crashes:
- The transaction with a lower sequence number holding the lock, while the other transaction with a higher sequence number tries to acquire lock: This is a transient BF-BF lock conflict as the transactions are in the right order and the lock holder will eventually release it after a while. Thus, the transaction that is requesting the lock just have to wait for a bit. This issue is present in 10.4.17 and a fix for that will be released in 10.4.19.
- The transaction with a higher sequence number holding the lock, while the other transaction with a lower sequence number tries to acquire lock: That is what my test case tries to extort. For me, this seems to be an out of order committing case, and to my understanding that should not happen as Galera uses strict total order committing by default. I tried to set it explicitly, but it does not help. The only workaround that has been proven to be effective is to reduce the number of applier thread (wsrep_slave_threads) count to 1. However, I haven't done any measurement as of yet, so the performance impact isn't known.
Considering the fact that Galera gives option to enable OOOC, my expectation then is that MariaDB should be capable to tackle with such situation, otherwise this needs to be listed as a limitation.
Best regards,
GOro
Hi Gabor TheGOro , thanks for the mtr test, nice work there, much appreciated.
I can reproduce the issue with this mtr test. The cause for the issue may be related to the fact that the test table does not have primary key. When I a change the test so that the test table has separate primary key, or if I declare the f1 as primary key, then the crash does not happen anymore, e.g. by this change in the test:
CREATE TABLE t1 (f1 VARCHAR(32) PRIMARY KEY NOT NULL) ENGINE=InnoDB;
From the error log sample of this issue description, it looks probable the the affected table (neutron.alembic_version) also misses primary key. Is this the correct, and is this the case with all previous crashes as well?
The test executes with wsrep_certify_nopk = ON, but it does not help here, because the certification key hashes calculated over the two replicated update rows do not conflict.
-seppo
Hi Seppo,
Thanks for your response! Yes, the absence of the PK on the test table was intentional as Alembic (migration tool for the SQLAlchemy ORM library) does not use it either. I did some research on the topic, and it turned out that this was a known issue/deficiency in earlier versions of Alembic and it got fixed:
I'll do some testing with the modified table declaration to see that this affects only just the second (out of order committing) case or both of them, to decide whether we can stay on 10.4.17 (with the aforementioned patches) for a while or we have to ask our vendor for MDEV-24923 backport.
Best regards,
GOro
Hi Seppo,
We have concluded our testing and verification using the updated Alembic package and also with our separate reproducer suite that got aligned. The results are good and we haven't observed any problems.
I guess the test case that I wrote makes little sense, as this scenario couldn't be considered as a supported one. Thus, I'm not trying to prep it up and push it as a contribution. Anyways, thanks for your help and guidance!
Feel free to close this ticket.
Best regards,
GOro
Hi,
FYI, we got this error on 2 differents node from 2 differents cluster:
Error :
mysqld: /home/buildbot/buildbot/build/mariadb-10.4.19/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
|
210519 10:12:07 [ERROR] mysqld got signal 6 ;
|
OS:
Debian buster 10.8
Version mariadb/galera-4 :
mariadb-server => 1:10.4.19+maria~buster / galera-4 => 26.4.8-buster
I don't know if I should create a new issue but this error looks very similar.
Someone else experiencing this error ?
The original problem was diagnosed to be a result of using a table with no primary key. As this is not recommended use case, the priority of this issue was lowered. However, I have a fix for supporting also tables with no primary key (in this scenario), and it will go to code review soon.
Please, take a look if your use case also is using table(s) without primary key.
MDEV-21181 and later MDEV-24001 have been filed for automatically creating a hidden primary key in the SQL layer if the user provided none. That would prevent InnoDB from creating a hidden primary key on its own. The InnoDB GEN_CLUST_INDEX(DB_ROW_ID) is not visible in the SQL layer except as a 48-bit key. The InnoDB internal column DB_ROW_ID is only materialized if no PRIMARY KEY and no UNIQUE NOT NULL index is available.
As I noted in MDEV-24001, row-based replication of an UPDATE of all n rows of a table that lacks a primary key should explode to n table scans, visiting n² rows in total.
In my opinion, ultimately this problem needs to be solved in the SQL layer.
Colleagues, good afternoon!
This error is present in the configuration 10.5.19-MariaDB galera-4-26.4.14-1.el8.x86_64.
wsrep_slave_threads = 10.
I experienced the same errors in version 10.4.22. Unfortunately, the option with wsrep_slave_threads = 1 is not suitable, because the cluster performance is reduced to 0.
Setting cert.optimistic_pa=no solves the problem, but does not solve the performance issue.
All tables have primary keys.