[MXS-4785] KafkaCDC JSON conversion is taking most of the CPU time Created: 2023-09-28  Updated: 2023-10-20  Resolved: 2023-10-20

Status: Closed
Project: MariaDB MaxScale
Component/s: kafkacdc
Affects Version/s: 6.4.10
Fix Version/s: 24.02.0

Type: Bug Priority: Major
Reporter: markus makela Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None

Attachments: File perf-replicator.svg    
Issue Links:
Relates
relates to MXS-4775 KafkaCDC: current_gtid.txt is moving ... Closed
Sprint: MXS-SPRINT-192

 Description   

In MXS-4775 it was found that there are inefficiencies in the KafkaCDC event-to-json conversion pipeline. Here's the visualization of the profiling from MXS-4775.

perf-replicator.svg

Here's a quick list of improvements that can be done based on some quick profiling of the code I did locally.

  • The gtid_pos_t::to_string() function uses a std::stringstream to create the GTID string. Using mxb::cat() and std::to_string() might prove to be better. sprintf() could also behave better as it is known that the maximum length of a single GTID value is fixed.
  • The UTF8 checks done by the JSON library show up in profiling. This could be avoided by using json_string_nocheck() but the strings need to be validated to be UTF for this to be reliable. If the strings are known to be UTF8, the checks can be avoided.
  • Most of the time is spent in KafkaEventHandler::commit() where the JSON object is converted into a string. If the string form of the JSON payload is constructed in-place, the overall cost of the JSON conversion would be lower.


 Comments   
Comment by markus makela [ 2023-10-13 ]

The JSON conversion overhead could be reduced by switching to a faster JSON library in KafkaCDC. Initial benchmarks show that RapidJson would be the best alternative and with the limited scope of where it would be used, it might make sense to use it in KafkaCDC but not elsewhere in MaxScale.

Comment by markus makela [ 2023-10-20 ]

Switched the kafkacdc JSON processing to use RapidJSON. This seems to have significantly improved the overall performance of the module.

Generated at Thu Feb 08 04:31:08 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.