[MDEV-32524] Use named constants for ids used in spider mem alloc functions to avoid collision Created: 2023-10-19  Updated: 2024-02-01  Resolved: 2023-11-19

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Spider
Fix Version/s: 10.4.33, 10.5.24, 10.6.17, 10.11.7, 11.0.5, 11.1.4

Type: Task Priority: Critical
Reporter: Yuchen Pei Assignee: Yuchen Pei
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-32486 Assertion `!trx->alloc_line_no[id] ||... Closed

 Description   

To avoid more issues like in MDEV-32486...



 Comments   
Comment by Yuchen Pei [ 2023-10-25 ]

Hi holyfoot, ptal thanks (based on 11.0)

upstream/bb-11.0-mdev-32524 8ce2924e6be5361d1f2e73f54aa3bc53b98f30e5
MDEV-32524 Use named constants for ids passed to spider_bulk_malloc()
 
This will avoid issues like MDEV-32486

For a 10.4 version, see

4ff721cffa6 upstream/bb-10.4-mdev-32524 MDEV-32524 Use named constants for ids passed to spider_bulk_malloc()

Comment by Alexey Botchkov [ 2023-10-30 ]

Great that you fixed this.
Still i have two comments there
https://github.com/MariaDB/server/commit/8ce2924e6be5361d1f2e73f54aa3bc53b98f30e5

Comment by Yuchen Pei [ 2023-10-31 ]

Hi holyfoot, thanks for the comment, and ptal thanks

b9d74012c02 bb-11.0-mdev-32524 MDEV-32524 Fix long lines
9b79247b8c4 MDEV-32524 Use enums for ids passed to spider mem alloc functions

Comment by Alexey Botchkov [ 2023-10-31 ]

ok to push.

Comment by Yuchen Pei [ 2023-11-17 ]

I have prepared commits generated by a script for all versions up to
11.0, with the 10.4 almost ready to push. Each version of 10.4-10.11
seems to be quite different. Will push on Monday, after

1. Fixing the commits due to a minor issue in the script (not bounding
regex search by right brace)
2. Waiting for any comments on slack regarding potential merging
troubles

Comment by Yuchen Pei [ 2023-11-19 ]

Pushed 0b36694ff8293ee3e6ae5bffd0c254cced0a4ff0 to 10.4.

Conflicts and solutions:

  • 10.4->10.5: 9f7da106c2c5dd9f48f8fcf40823a9d4fd655298
  • 10.6->10.11: 6a110e64b908e9d574e50c0f403b02769cfc4052
  • 10.11->11.0: 4d8167563e79f2657454a9f410e9cf2a2673bf76

Because of the nature of the change, the best way to resolve merge
conflicts is:

1. merge until the parent commit of this commit
2. git merge this commit; presented with conflicts
3. disregard the changes, i.e. choose "ours" for all files
4. run the script (see below)
5. complete the merge of this commit
6. merge the descendants of this commit

I can do 2-5 if needed

elisp script to generate the change for this ticket:

(defvar my-mdev-32524-ids nil "List of ids used.")
 
(defun my-mdev-32524-make-id ()
  "Return an id on the current position."
  (let* ((base
          (replace-regexp-in-string
           "::" "_"
           (replace-regexp-in-string
            "^\\(SPIDER_\\)?" "SPD_MID_"
            (upcase (which-function)))))
         (last-id (car my-mdev-32524-ids))
         (counter (cdr last-id))
         (result))
    (if (equal (car last-id) base)
        (progn
          (setq result (format "%s_%d" base (1+ counter)))
          (push (cons base (1+ counter)) my-mdev-32524-ids))
      (setq result (format "%s_1" base))
      (push (cons base 1) my-mdev-32524-ids))
    result))
 
(defun my-mdev-32524-replace-ids ()
  "Replace magic id numbers with enum symbols."
  (setq my-mdev-32524-ids nil)
  (dolist (file (directory-files "./storage/spider" t ".\\(cc\\|h\\)$"))
    ;; Don't do this with spd_malloc.{cc,h}
    (unless (string-match "spd_malloc" file)
      (with-current-buffer (find-file-noselect file)
        (goto-char (point-min))
        (while (re-search-forward
                (concat
                 "\\(spider_alloc_calc_mem_init\\|init_calc_mem\\|spider_malloc\\|"
                 "spider_bulk_alloc_mem\\|spider_bulk_malloc\\)"
                 "\\(([^0-9)]*\\)\\([0-9]+\\)")
                nil t)
          (let ((beg (match-beginning 3))
                (end (match-end 3)))
            (replace-region-contents beg end #'my-mdev-32524-make-id)))
        (save-buffer)))))
 
(defun my-mdev-32524-insert-enum-def ()
  "Insert enum def at point according to `my-mdev-32524-ids'."
  (with-current-buffer (find-file-noselect "./storage/spider/spd_include.h")
    (goto-char (point-min))
    (re-search-forward "#define.* SPIDER_CONN_META_BUF_LEN")
    (beginning-of-line 2)
    (let ((pre
           "
/*
  IDs for spider mem alloc functions, including
  - spider_alloc_calc_mem_init()
  - spider_string::init_calc_mem()
  - spider_malloc()
  - spider_bulk_alloc_mem()
  - spider_bulk_malloc()
  In the format of
  SPD_MID_<CALLSITE_FUNC_NAME_SANS_SPIDER_PREFIX>_<NO>
*/
enum spider_malloc_id {
  ")
          (middle (string-join
                   (sort
                    (mapcar
                     (lambda (pair)
                       (format "%s_%d" (car pair) (cdr pair)))
                     my-mdev-32524-ids)
                    #'string<)
                   ",
  "))
          (post "
};
"))
      (insert (concat pre middle post))
      (save-buffer))))
 
(defun my-mdev-32524 ()
  "Make changes for MDEV-32524. Assuming we are at the source root."
  (interactive)
  (my-mdev-32524-replace-ids)
  (my-mdev-32524-insert-enum-def))

Generated at Thu Feb 08 10:32:00 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.