[MCOL-1959] mxs_adapter assertion crash on multi table stream Created: 2018-11-23  Updated: 2023-10-26  Resolved: 2018-12-20

Status: Closed
Project: MariaDB ColumnStore
Component/s: ?
Affects Version/s: 1.2.1
Fix Version/s: 1.2.2

Type: Bug Priority: Major
Reporter: Dipti Joshi (Inactive) Assignee: Zdravelina Sokolovska (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Environment:

RHEL 7

Attachments: File maxscale.cnf    
Issue Links:
Problem/Incident
is caused by MXS-2184 event_number is not incremented for u... Closed

 Description   

Getting crash when trying to ingest two tables using multi-streaming option of mxs_adapter

[ec2-user@ip-172-31-38-194 ~]$ mxs_adapter -c /etc/Columnstore.xml -u cdcuser -p cdc -h 127.0.0.1 -P 4001 -r 2 -d -n -z -f stream-table-list
 
2018-11-23 17:41:17 [main] Started thread 0x10ea2c0
 
2018-11-23 17:41:17 [main] Started thread 0x10ea710
 
2018-11-23 17:41:17 [main] Started 2 threads
 
2018-11-23 17:41:17 [test.t3] Requesting data for table: test.t3
 
2018-11-23 17:41:17 [test.t2] Requesting data for table: test.t2
 
2018-11-23 17:41:17 [test.t2] Continuing from GTID: 0-1-16:4
 
2018-11-23 17:41:17 [test.t2] Skipped 7 rows
 
mxs_adapter: /src/maxscale-cdc-adapter/src/mxs_adapter.cpp:430: void transformRow(UContext&, CDC::SRow&): Assertion `ctx->update_before' failed.
 
2018-11-23 17:41:17 [test.t2] Received fatal signal 6
 
mxs_adapter[0x42c5f5]
 
/lib64/libpthread.so.0(+0xf680)[0x7fc08fb14680]
 
/lib64/libc.so.6(gsignal+0x37)[0x7fc08e463207]
 
/lib64/libc.so.6(abort+0x148)[0x7fc08e4648f8]
 
/lib64/libc.so.6(+0x2f026)[0x7fc08e45c026]
 
/lib64/libc.so.6(+0x2f0d2)[0x7fc08e45c0d2]
 
mxs_adapter(_Z12transformRowRSt10unique_ptrI7ContextSt14default_deleteIS0_EERNSt3tr110shared_ptrIN3CDC3RowEEE+0x3a9)[0x41cbb9]
 
mxs_adapter(_Z12processTableRSt10unique_ptrI7ContextSt14default_deleteIS0_EE+0xfd)[0x41d37d]
 
mxs_adapter(_Z11streamTableSsSs+0x20c)[0x41d9ec]
 
mxs_adapter(_ZNSt6thread5_ImplISt12_Bind_simpleIFPFvSsSsESsSsEEE6_M_runEv+0x37)[0x41f0b7]
 
/lib64/libstdc++.so.6(+0xb5070)[0x7fc08edc7070]
 
/lib64/libpthread.so.0(+0x7dd5)[0x7fc08fb0cdd5]
 
/lib64/libc.so.6(clone+0x6d)[0x7fc08e52bb3d]
 
Aborted

Table definition on MariaDB Server

MariaDB [test]> show create table t2
 
    -> ;
 
+-------+-------------------------------------------------------------------------------------------------------------------+
 
| Table | Create Table                                                                                                      |
 
+-------+-------------------------------------------------------------------------------------------------------------------+
 
| t2    | CREATE TABLE `t2` (
 
  `a` int(11) DEFAULT NULL,
 
  `b` int(11) DEFAULT NULL
 
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
 
+-------+-------------------------------------------------------------------------------------------------------------------+
 
1 row in set (0.000 sec)
 
MariaDB [test]> select * from t2;
 
+------+------+
 
| a    | b    |
 
+------+------+
 
|    1 |    1 |
 
|    1 |    2 |
 
|    1 |    3 |
 
|    1 |    4 |
 
|    1 |    5 |
 
|    1 |    6 |
 
|    1 |    7 |
 
|    1 |    8 |
 
|    1 |    9 |
 
|    1 |   10 |
 
|    2 |   11 |
 
|    2 |   12 |
 
|    2 |   13 |
 
|    2 |   14 |
 
+------+------+
 
14 rows in set (0.000 sec)
 
 
 
MariaDB [test]> show create table t3;
 
+-------+-------------------------------------------------------------------------------------------------------------------+
 
| Table | Create Table                                                                                                      |
 
+-------+-------------------------------------------------------------------------------------------------------------------+
 
| t3    | CREATE TABLE `t3` (
 
  `b` int(11) DEFAULT NULL,
 
  `c` int(11) DEFAULT NULL
 
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
 
+-------+-------------------------------------------------------------------------------------------------------------------+
 
1 row in set (0.000 sec)
 
MariaDB [test]> select * from t3;
 
Empty set (0.000 sec)

Table definition on MariaDB ColumnStore

MariaDB [test]> show create table t2;
 
+-------+------------------------------------------------------------------------------------------------------------------------+
 
| Table | Create Table                                                                                                           |
 
+-------+------------------------------------------------------------------------------------------------------------------------+
 
| t2    | CREATE TABLE `t2` (
 
  `a` int(11) DEFAULT NULL,
 
  `b` int(11) DEFAULT NULL
 
) ENGINE=Columnstore DEFAULT CHARSET=latin1 |
 
+-------+------------------------------------------------------------------------------------------------------------------------+
 
1 row in set (0.000 sec)
 
 
 
MariaDB [test]> show create table t3;
 
+-------+------------------------------------------------------------------------------------------------------------------------+
 
| Table | Create Table                                                                                                           |
 
+-------+------------------------------------------------------------------------------------------------------------------------+
 
| t3    | CREATE TABLE `t3` (
 
  `b` int(11) DEFAULT NULL,
 
  `c` int(11) DEFAULT NULL
 
) ENGINE=Columnstore DEFAULT CHARSET=latin1 |
 
+-------+------------------------------------------------------------------------------------------------------------------------+
 
1 row in set (0.000 sec)



 Comments   
Comment by Dipti Joshi (Inactive) [ 2018-11-23 ]

Please note that before doing this multi-stream option on mxs_adapter, I had previously streamed table t2 directly using command line option like this:

mxs_adapter -c /etc/Columnstore.xml -u cdcuser -p cdc -h 127.0.0.1 -P 4001 -r 2 -d -n -z test t2

Comment by Zdravelina Sokolovska (Inactive) [ 2018-11-26 ]

checked with mxs_adapter and mcsapi v1.2.2-1 installed on max-scale server with v2.3.1-1 ,
crash was not recreated but it's received

[root@localhost ~]# mxs_adapter -c Columnstore.xml -u root -p 1 -h 127.0.0.1 -P 4001 -r 2 -d -n -z -f tbls
2018-11-26 14:47:15 [main] Started thread 0x17e7220
2018-11-26 14:47:15 [main] Started thread 0x17e70a0
2018-11-26 14:47:15 [main] Started 2 threads
2018-11-26 14:47:15 [AAA.bbb] Requesting data for table: AAA.bbb
2018-11-26 14:47:15 [AAA.bbb] MaxScale connection could not be created: Failed to connect: Connection refused
2018-11-26 14:47:15 [AAA.aaa] Requesting data for table: AAA.aaa
2018-11-26 14:47:15 [AAA.aaa] MaxScale connection could not be created: Failed to connect: Connection refused

MariaDB-TX part ------- max-scale server && mxs_adapter && mcsapi --------- MariaDB-AX part

the crash was not recreated but

MariaDB-TX part seen from MAX-Scale

[root@localhost ~]# maxctrl list servers
┌─────────┬────────────┬──────┬─────────────┬─────────────────────────┬──────┐
│ Server │ Address │ Port │ Connections │ State │ GTID │
├─────────┼────────────┼──────┼─────────────┼─────────────────────────┼──────┤
│ server1 │ 172.20.3.4 │ 3306 │ 0 │ Master, Synced, Running │ │
├─────────┼────────────┼──────┼─────────────┼─────────────────────────┼──────┤
│ server2 │ 172.20.3.5 │ 3306 │ 0 │ Slave, Synced, Running │ │
└─────────┴────────────┴──────┴─────────────┴─────────────────────────┴──────┘

MariaDB-AX part
[root@pm1 ~]# mcsadmin getsystems
getsystemstatus Mon Nov 26 16:24:22 2018

System columnstore-1

System and Module statuses

Component Status Last Status Change
------------ -------------------------- ------------------------
System ACTIVE Mon Nov 26 16:23:33 2018

Module um1 ACTIVE Mon Nov 26 16:23:31 2018
Module pm1 ACTIVE Mon Nov 26 16:23:14 2018
Module pm2 ACTIVE Mon Nov 26 16:23:21 2018

Comment by Dipti Joshi (Inactive) [ 2018-12-01 ]

This indicates that either

  • CDC user that you are using is not defined in MaxScale and source database
  • OR
  • The port 4001 on MaxScale server is not accepting connection.

Please provide maxscale.cnf from your MaxScale winstone

Comment by Zdravelina Sokolovska (Inactive) [ 2018-12-10 ]

dshjoshi
CDC user was added and the source database is accessible by the CDC user . but now it fails with Error "MaxScale responded with an error: ERR NO-FILE File 'AAA.aaa.000001.avro' not found.' / attached the maxscale.cnf 

 maxctrl call command cdc add_user avro-router cdcuser cdcpassword
 
OK
 
 
 
 
 
 mysql -h 127.0.0.1 -u cdcuser -pcdcpassword  -P 3307
 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
 
Your MariaDB connection id is 7
 
Server version: 10.3.10-MariaDB
 
 
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
 
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
 
 
 
 
source DB is accessible:
 
 
 
MariaDB [(none)]> connect AAA ;
 
Connection id:    8
 
Current database: AAA
 



 
 
  mxs_adapter -c /usr/local/mariadb/columnstore/etc/Columnstore.xml -u cdcuser -p cdcpassword -h 127.0.0.1  -P 4001 -r 2 -d -n -z AAA aaa
 
2018-12-10 18:33:00 [main] Started thread 0xea6450
 
2018-12-10 18:33:00 [main] Started 1 threads
 
2018-12-10 18:33:00 [AAA.aaa] Requesting data for table: AAA.aaa
 
2018-12-10 18:33:00 [AAA.aaa] MaxScale connection could not be created: MaxScale responded with an error: ERR NO-FILE File 'AAA.aaa.000001.avro' not found.
 
 

Comment by Dipti Joshi (Inactive) [ 2018-12-10 ]

markus makela Please help Zdravelina Sokolovska in troubleshooting her setup

Comment by Zdravelina Sokolovska (Inactive) [ 2018-12-20 ]

note: if second table was empty at begging the subsequent inserts into that table on the source MariaDB-TX are not streamed to MariaDB-AX 

[root@localhost /]# mxs_adapter -c /etc/Columnstore.xml -u cdcuser -p cdcpassword -h 127.0.0.1 -P 4001 -r 2  -f stream-table-list
 
2018-12-20 19:30:35 [testA.t4] Continuing from GTID: 0-1-209:1
 
2018-12-20 19:30:35 [testA.t5] MaxScale connection could not be created: MaxScale responded with an error: ERR NO-FILE File 'testA.t5.000001.avro' not found.
 
 
 
2018-12-20 19:30:44 [testA.t4] 1 rows, 0 transactions inserted over 0.267465 seconds. GTID = 0-1-210:1
 
2018-12-20 19:30:44 [testA.t4] 2 rows, 1 transactions inserted over 0.222182 seconds. GTID = 0-1-212:1
 
2018-12-20 19:30:55 [testA.t4] 1 rows, 0 transactions inserted over 10.2337 seconds. GTID = 0-1-213:1
 
 
 
2018-12-20 19:32:50 [testA.t4] 1 rows, 0 transactions inserted over 0.272257 seconds. GTID = 0-1-214:1
 
2018-12-20 19:32:51 [testA.t4] 2 rows, 1 transactions inserted over 0.220511 seconds. GTID = 0-1-216:1
 
2018-12-20 19:32:52 [testA.t4] 2 rows, 1 transactions inserted over 0.21406 seconds. GTID = 0-1-218:1
 
2018-12-20 19:32:53 [testA.t4] 2 rows, 1 transactions inserted over 0.242956 seconds. GTID = 0-1-220:1
 
2018-12-20 19:32:54 [testA.t4] 2 rows, 1 transactions inserted over 0.222823 seconds. GTID = 0-1-222:1
 
2018-12-20 19:33:05 [testA.t4] 1 rows, 0 transactions inserted over 10.2513 seconds. GTID = 0-1-223:1
 
 

2. both tables initially are not empty 

[root@localhost /]# mxs_adapter -c /etc/Columnstore.xml -u cdcuser -p cdcpassword -h 127.0.0.1 -P 4001 -r 2 -d -n  -f stream-table-list
 
2018-12-20 19:41:50 [main] Started thread 0x17872a0
 
2018-12-20 19:41:50 [main] Started thread 0x17870c0
 
2018-12-20 19:41:50 [main] Started 2 threads
 
2018-12-20 19:41:50 [testA.t7] Requesting data for table: testA.t7
 
2018-12-20 19:41:50 [testA.t6] Requesting data for table: testA.t6
 
2018-12-20 19:42:01 [testA.t6] a: 1
 
2018-12-20 19:42:01 [testA.t6] b: 1
 
2018-12-20 19:42:01 [testA.t6] 1 rows, 0 transactions inserted over 0.162819 seconds. GTID = 0-1-326:1
 
2018-12-20 19:42:01 [testA.t6] a: 1
 
2018-12-20 19:42:01 [testA.t6] b: 2
 
2018-12-20 19:42:01 [testA.t6] a: 1
 
2018-12-20 19:42:01 [testA.t6] b: 3
 
2018-12-20 19:42:01 [testA.t6] GTID 0-1-328
 
2018-12-20 19:42:01 [testA.t6] 2 rows, 1 transactions inserted over 0.146391 seconds. GTID = 0-1-328:1
 
2018-12-20 19:42:02 [testA.t7] b: 2
 
2018-12-20 19:42:02 [testA.t7] c: 1
 
2018-12-20 19:42:02 [testA.t7] 1 rows, 0 transactions inserted over 0.150914 seconds. GTID = 0-1-337:1
 
2018-12-20 19:42:02 [testA.t6] a: 1
 
2018-12-20 19:42:02 [testA.t6] b: 4
 
2018-12-20 19:42:02 [testA.t6] a: 1
 
2018-12-20 19:42:02 [testA.t6] b: 5
 
2018-12-20 19:42:02 [testA.t6] GTID 0-1-330
 
2018-12-20 19:42:02 [testA.t6] 2 rows, 1 transactions inserted over 0.151085 seconds. GTID = 0-1-330:1
 
2018-12-20 19:42:03 [testA.t7] b: 2
 
2018-12-20 19:42:03 [testA.t7] c: 2
 
2018-12-20 19:42:03 [testA.t7] b: 2
 
2018-12-20 19:42:03 [testA.t7] c: 3
 
2018-12-20 19:42:03 [testA.t7] GTID 0-1-339
 
2018-12-20 19:42:03 [testA.t7] 2 rows, 1 transactions inserted over 0.144482 seconds. GTID = 0-1-339:1
 
2018-12-20 19:42:04 [testA.t6] a: 1
 
2018-12-20 19:42:04 [testA.t6] b: 6
 
2018-12-20 19:42:04 [testA.t6] a: 1
 
2018-12-20 19:42:04 [testA.t6] b: 7
 
2018-12-20 19:42:04 [testA.t6] GTID 0-1-332
 
2018-12-20 19:42:04 [testA.t6] 2 rows, 1 transactions inserted over 0.16718 seconds. GTID = 0-1-332:1
 
2018-12-20 19:42:05 [testA.t7] b: 2
 
2018-12-20 19:42:05 [testA.t7] c: 4
 
2018-12-20 19:42:05 [testA.t7] b: 2
 
2018-12-20 19:42:05 [testA.t7] c: 5
 
2018-12-20 19:42:05 [testA.t7] GTID 0-1-341
 
2018-12-20 19:42:05 [testA.t7] 2 rows, 1 transactions inserted over 0.148735 seconds. GTID = 0-1-341:1
 
2018-12-20 19:42:06 [testA.t6] a: 1
 
2018-12-20 19:42:06 [testA.t6] b: 8
 
2018-12-20 19:42:06 [testA.t6] a: 1
 
2018-12-20 19:42:06 [testA.t6] b: 9
 
2018-12-20 19:42:06 [testA.t6] GTID 0-1-334
 
2018-12-20 19:42:06 [testA.t6] 2 rows, 1 transactions inserted over 0.146702 seconds. GTID = 0-1-334:1
 
2018-12-20 19:42:07 [testA.t7] b: 2
 
2018-12-20 19:42:07 [testA.t7] c: 6
 
2018-12-20 19:42:07 [testA.t7] b: 2
 
2018-12-20 19:42:07 [testA.t7] c: 7
 
2018-12-20 19:42:07 [testA.t7] GTID 0-1-343
 
2018-12-20 19:42:07 [testA.t7] 2 rows, 1 transactions inserted over 0.145437 seconds. GTID = 0-1-343:1
 
2018-12-20 19:42:08 [testA.t6] a: 1
 
2018-12-20 19:42:08 [testA.t6] b: 10
 
2018-12-20 19:42:08 [testA.t6] a: 1
 
2018-12-20 19:42:08 [testA.t6] b: 11
 
2018-12-20 19:42:08 [testA.t6] GTID 0-1-336
 
2018-12-20 19:42:08 [testA.t6] 2 rows, 1 transactions inserted over 0.138285 seconds. GTID = 0-1-336:1
 
2018-12-20 19:42:09 [testA.t7] b: 2
 
2018-12-20 19:42:09 [testA.t7] c: 8
 
2018-12-20 19:42:09 [testA.t7] b: 2
 
2018-12-20 19:42:09 [testA.t7] c: 9
 
2018-12-20 19:42:09 [testA.t7] GTID 0-1-345
 
2018-12-20 19:42:09 [testA.t7] 2 rows, 1 transactions inserted over 0.151025 seconds. GTID = 0-1-345:1
 
2018-12-20 19:42:11 [testA.t7] b: 2
 
2018-12-20 19:42:11 [testA.t7] c: 10
 
2018-12-20 19:42:11 [testA.t7] b: 2
 
2018-12-20 19:42:11 [testA.t7] c: 11
 
2018-12-20 19:42:11 [testA.t7] GTID 0-1-347
 
2018-12-20 19:42:11 [testA.t7] 2 rows, 1 transactions inserted over 0.150856 seconds. GTID = 0-1-347:1

Generated at Thu Feb 08 02:32:38 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.