[MDEV-30964] MAX_SEL_ARGS memory exhaustion is not visible in the optimizer trace Created: 2023-03-29  Updated: 2023-09-07  Resolved: 2023-06-08

Status: Closed
Project: MariaDB Server
Component/s: Optimizer
Affects Version/s: 10.4
Fix Version/s: 10.8.8, 10.4.31, 10.5.22, 10.6.15, 10.9.8, 10.10.6, 10.11.5, 11.0.3, 11.1.2

Type: Bug Priority: Major
Reporter: Sergei Petrunia Assignee: Sergei Petrunia
Resolution: Fixed Votes: 1
Labels: None


 Description   

It is possible to construct queries that hit the MAX_SEL_ARG=16000 limit in the range optimizer. Unfortunately, this limit is only visible in debugger.

It is trivial print hitting this limit into the optimizer trace.



 Comments   
Comment by Sergei Petrunia [ 2023-03-29 ]

Note: starting from 10.5 and fix for MDEV-9750, there other limits one can hit in the range optimizer. The optimizer makes a note in the trace when hitting them

examples:

                  "setup_range_conditions": [
                    {
                      "enforce_sel_arg_weight_limit": {
                        "index": "PRIMARY",
                        "old_weight": 105600,
                        "new_weight": 1560
                      }
                    }
                  ],

                      "sel_arg_weight_heuristic": {
                        "key1_field": "year_id",
                        "key2_field": "location_id",
                        "key1_weight": 30,
                        "key2_weight": 3519
                      }

Comment by Sergei Petrunia [ 2023-03-29 ]

Johnston could you review this?

Comment by Sergei Petrunia [ 2023-08-22 ]

MAX_SEL_ARGS is "the maximum number of SEL_ARG objects that may be allocated during range analysis" (range analysis is performed on a per-table basis) .
MAX_SEL_ARGS is a #define in opt_range.cc, it is not visible to the user.

max_sel_arg_weight is a user-visible variable. It is "the maximum weight of a SEL_ARG graph that the optimizer may construct". The definitions of SEL_ARG graph and its weight are rather convoluted.

If the optimizer allocated N SEL_ARG objects, the SEL_ARG graph it constructed may have a weight that's less, equal, or greater than N.

Comment by Sergei Petrunia [ 2023-08-22 ]

Note:

Optimizer trace will get these keywords for hitting the weight limit:

  • sel_arg_weight_heuristic
  • enforce_sel_arg_weight_limit

The patch for this MDEV added:

  • sel_arg_alloc_limit_hit
Comment by Sergei Petrunia [ 2023-08-22 ]

Summary for documentation:

The optimizer may cause high memory usage or long processing time when analyzing complex WHERE clauses.
It has multiple limits which tell it to short-cut the analysis if it gets too time and/or memory intensive.
One of these limit is MAX_SEL_ARGS constant which is a #define in the source code. After this bugfix, the optimizer will write a sel_arg_alloc_limit_hit record in the Optimizer Trace when it hits the MAX_SEL_ARGS limitation.

Generated at Thu Feb 08 10:20:13 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.