[MDEV-28931] MTR prints detailed stack trace unconditionally, etc Created: 2022-06-23  Updated: 2022-12-16  Resolved: 2022-07-18

Status: Closed
Project: MariaDB Server
Component/s: Tests
Affects Version/s: 10.3, 10.4, 10.5, 10.6, 10.7
Fix Version/s: 10.3.36, 10.4.26, 10.5.17, 10.6.9, 10.7.5, 10.8.4, 10.9.2

Type: Bug Priority: Major
Reporter: Aleksey Midenkov Assignee: Aleksey Midenkov
Resolution: Fixed Votes: 0
Labels: None

Attachments: Text File new_buildbot_env.txt     Text File old_buildbot_env.txt     File print_env.test    
Issue Links:
Problem/Incident
causes MDEV-30242 MTR fails to report stack traces of a... Closed
Relates
relates to MDEV-30084 Shutdown hangs in some tests Closed
Sub-Tasks:
Key
Summary
Type
Status
Assignee
MDEV-29023 MTR hangs after multiple failures Technical task Closed Aleksey Midenkov  
MDEV-29025 MTR doesn't print stack trace for boo... Technical task Closed Aleksey Midenkov  

 Description   

66832e3a introduced change that prints core dumps like this:

#16 0x000055b86613e3d8 in do_handle_one_connection (connect=0x55b8699061f8, put_in_cache=true) at /home/midenok/src/mariadb/10.6/src/sql/sql_connect.cc:1418
        create_user = true
        thr_create_utime = 140576815727
        thd = 0x7f11c4000d58
#17 0x000055b86613e03a in handle_one_connection (arg=0x55b8699f84a8) at /home/midenok/src/mariadb/10.6/src/sql/sql_connect.cc:1312
        connect = 0x55b8699f84a8
#18 0x000055b8666b0a2f in pfs_spawn_thread (arg=0x55b86997b1d8) at /home/midenok/src/mariadb/10.6/src/storage/perfschema/pfs.cc:2201
        typed_arg = 0x55b86997b1d8
        user_arg = 0x55b8699f84a8
        user_start_routine = 0x55b86613dfe0 <handle_one_connection(void*)>
        pfs = 0x7f1203ebf7c0
        klass = 0x55b8691b5c80
#19 0x00007f120461fb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
        ret = <optimized out>
        pd = <optimized out>
        out = <optimized out>
        unwind_buf = {
          cancel_jmp_buf = {{
              jmp_buf = {140727175011760, -485359056267303735, 139715155310144, 2, 139715359668304, 140727175012112, 531522500273317065, 533571283848864969},
              mask_was_saved = 0
            }},
          priv = {
            pad = {0x0, 0x0, 0x0, 0x0},
            data = {
              prev = 0x0,
              cleanup = 0x0,
              canceltype = 0
            }
          }
        }
        not_first_call = <optimized out>

That's completely out of user-friendliness but serves as a measure for debugging hard-reproducible bugs. The proper way to implement this:

1. it must be controlled by command-line and environment variable;
2. detailed traces must be default for buildbots only, for user invocations normal stack traces should be printed.

Options for control

MTR_PRINT_CORE and --print-core can accept the following values:

no Don't print core
short Print normal trace like `bt` does
detailed Print detailed stack trace like in commit 66832e3a
custom:<code> Use GDB commands <code> to print stack trace
Default behaviour

Should be detected by environment. If BUILDMASTER environment is non-null, defaults to "detailed", otherwise "short".

Environment

MTR_PRINT_CORE=no|short|detailed|custom:code

Wrong values are silently ignored (falls back to default behaviour).
Overrides default behavior.

Command-line

--print-core{=no|short|detailed|custom:code}

When specified without value defaults to "short". Wrong values fail execution with error.
Overrides environment variable.



 Comments   
Comment by Sergei Golubchik [ 2022-06-23 ]

I'd suggest not to autodetect BUILDMASTER. Explicit --print-core and MTR_PRINT_CORE is better, we can configure buildbot to use one of those.

Comment by Aleksey Midenkov [ 2022-07-05 ]

Please review bb-10.3-midenok-MDEV-28931 (commits for MDEV-28931, MDEV-29025, MDEV-29023)

Comment by Aleksey Midenkov [ 2022-07-12 ]

More to check:

https://buildbot.askmonty.org/buildbot/builders/kvm-bintar-trusty-x86/builds/28282/steps/mtr/logs/stdio

Could not execute 'check-warnings' for testcase 'innodb.innodb_ctype_big5' (res: 5) server: 'mysqld.1:
The test './include/check-warnings.test' is not supported by this installation
Detected in file ./include/check-warnings.test at line 64
reason: OK
skipped
mysqltest failed but provided no output

Comment by Oleksandr Byelkin [ 2022-07-15 ]

OK to push after removing changes in formatting untouched lines (tabs)

Comment by Michael Widenius [ 2022-12-15 ]

Why did introduce a new variable instead of extending --core-file?

It would also be better if we would have a common prefix for related options, like we do with engine options, debug options etc. This make it easier to quickly find all related options.
I suggested that we change the option name to --core-print instead of --print-core.

The default is :short, which is probably ok normal users.
However when running a debug build (mainly only done by developers), it would probably be
better to have 'detailed' as default!

Comment by Marko Mäkelä [ 2022-12-16 ]

MDEV-30242 restored the old default of thread apply all bt instead of bt (showing a stack trace of only one thread).

Generated at Thu Feb 08 10:04:33 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.