Michael Widenius added a comment - 2024-02-22 07:19 Changes done as part of increasing performance for tables with a lot of partitions: Optimize bitmap code to make it notable faster Bitmaps underlying storage changed from 32 to 64 bit Using dedicated cpu instructions, instead of loops, to find next set / next reset bit. Simplify storage of last bitmap to simply loops over all bits handler_stats_disable() is now when handler stats are already disabled (common case) Partition engine will not anymore call handler::extra(HA_EXTRA..CHILDREN) if myisamrg is not used by the partition engine (common case)

Michael Widenius added a comment - 2024-02-22 07:38 Serg, please review the bitmap code 367c87a34f89e11d57bc7cd231037dc4dc098b6d

Sergei Petrunia added a comment - 2024-02-28 13:58 Commits: commit 31463f1164530464e97f30f6dad27a21842f3b19 (HEAD -> 10.6, origin/bb-10.6-mdev-33502-part4, origin/10.6) Author: Sergei Petrunia <sergey@mariadb.com> Date: Wed Feb 28 16:01:48 2024 +0300   MDEV-33502: part#4: Dont make redundant extra(HA_EXTRA_[NO]_KEYREAD) calls In most cases, ha_partition forwards calls to extra() to all locked_partitions. It doesn't make sense to forward some calls for partitions that were pruned away. This patch introduces ha_partition::loop_read_partitions and makes these calls use it: - ha_partition::extra_opt(HA_EXTRA_KEYREAD) - ha_partition::extra(HA_EXTRA_KEYREAD) - ha_partition::extra(HA_EXTRA_NO_KEYREAD) Reviewed-by: Monty commit b5d65fc105b6efa07cbbafcf053f6ce46f02dbc0 Author: Monty <monty@mariadb.org> Date: Sun Feb 18 17:30:01 2024 +0200   Optimize performance of my_bitmap MDEV-33502 Slowdown when running nested statement with many partitions This change was triggered to help some MariaDB users with close to 10000 bits in their bitmaps. - Change underlaying storage to be 64 bit instead of 32bit. - This reduses number of loops to scan bitmaps. - This can cause some bitmaps to be 4 byte large. - Ensure that all not used top-bits are always 0 (simplifes code as the last 64 bit storage is not a special case anymore). - Use my_find_first_bit() to find the first set bit which is much faster than scanning trough things byte by byte and then bit by bit. Other things: - Added a bool to remember if my_bitmap_init() did allocate the bitmap array. my_bitmap_free() will only free arrays it did allocate. This allowed me to remove setting 'bitmap=0' before calling my_bitmap_free() for cases where the bitmap's where allocated externally. - my_bitmap_init() sets bitmap to 0 in case of failure. - Added 'universal' asserts to most bitmap functions. - Change all remaining calls to bitmap_init() to my_bitmap_init(). - To finish the change from 2014. - Changed all usage of uint32 in my_bitmap.h to my_bitmap_map. - Updated bitmap_copy() to handle bitmaps of different size. - Removed const from bitmap_exists_intersection() as this caused casts on all usage. - Removed not used function bitmap_set_above(). - Renamed create_last_word_mask() to create_last_bit_mask() (to match name changes in my_bitmap.cc) - Extended bitmap-t with test for more bitmap functions. commit d4e1731fbc5f8fb05648a8b3e9ecd8bef4cafd36 Author: Monty <monty@mariadb.org> Date: Sun Feb 18 16:05:42 2024 +0200   Optimize handler_stats_disable() when handler_stats are already disabled MDEV-33502 Slowdown when running nested statement with many partitions commit a8f6b86c9053a929e455b8339e2068651faf5084 Author: Monty <monty@mariadb.org> Date: Sun Feb 18 16:02:40 2024 +0200   Have ha_partition ignore HA_EXTRA..CHILDREN extra() calls if no myisamrg MDEV-33502 Slowdown when running nested statement with many partitions Optimization for tables with a lot of partitions

Sergei Petrunia added a comment - 2024-03-04 12:16 Notes for the changelog: A statement over partitioned table incurs some O(#partitions-in-table) overhead even if it only accesses a small subset of partitions. If the query only accesses a few rows and the table has thousands of partitions, the overhead is visible. It got larger in MariaDB 10.6, this fix eliminates the increase in overhead.