[MDEV-34099] AddressSanitizer running out of memory regardless of stack_thread size - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Affects Version/s: 10.5
Fix Version/s: 10.5.26, 10.6.19, 10.11.9, 11.1.6, 11.2.5, 11.4.3
Component/s: Server
Labels:
- ASAN

Description

Example output for stack_thread=50M and stack_thread=30M

CURRENT_TEST: main.type_timestamp_hires12910Could not execute 'check-testcase' before testcase 'main.type_timestamp_hires' (res: 1):12911mysqltest: Logging to '/usr/share/mysql-test/var/2/tmp/check-mysqld_1.log'.12912mysqltest: Results saved in '/usr/share/mysql-test/var/2/tmp/check-mysqld_1.result'.12913mysqltest: Connecting to server localhost:16000 (socket /usr/share/mysql-test/var/tmp/2/mysqld.1.sock) as 'root', connection 'default', attempt 0 ...12914mysqltest: ... Connected.12915mysqltest: Start processing test commands from './include/check-testcase.test' ...12916mysqltest: At line 19: query 'let $tmp= `SELECT '$tmp' = 'No such row'`' failed: 1436: Thread stack overrun:  63936096 bytes used of a 52428800 byte stack, and 32000 bytes needed. Consider increasing the thread_stack system variable

CURRENT_TEST: main.type_timestamp_hires12878Could not execute 'check-testcase' before testcase 'main.type_timestamp_hires' (res: 1):12879mysqltest: Logging to '/usr/share/mysql-test/var/6/tmp/check-mysqld_1.log'.12880mysqltest: Results saved in '/usr/share/mysql-test/var/6/tmp/check-mysqld_1.result'.12881mysqltest: Connecting to server localhost:16060 (socket /usr/share/mysql-test/var/tmp/6/mysqld.1.sock) as 'root', connection 'default', attempt 0 ...12882mysqltest: ... Connected.12883mysqltest: Start processing test commands from './include/check-testcase.test' ...12884mysqltest: At line 19: query 'let $tmp= `SELECT '$tmp' = 'No such row'`' failed: 1436: Thread stack overrun:  42964576 bytes used of a 31457280 byte stack, and 32000 bytes needed. Consider increasing the thread_stack system variable.

Attachments

Issue Links

blocks

MDBF-740 bump clang-14 asan builder to use clang 20 (and do ubsan too)

Open

relates to

MDEV-31605 cmake/stack_direction.c does not work correctly on clang 16

Closed

MDEV-16660 Inadequate DEFAULT_THREAD_STACK size for AddressSanitizer

Closed

MDEV-33210 Several performance_schema tests fail due to explicit (too small for ASAN) thread_stack

Confirmed

Activity

Ascending order - Click to sort in descending order

Anson created issue - 2024-05-06 23:58

Anson made changes - 2024-05-06 23:58

Field	Original Value	New Value
Link		This issue relates to ~~MDEV-16660~~ [ ~~MDEV-16660~~ ]

Anson made changes - 2024-05-06 23:58

Link

This issue relates to MDEV-33210 [ MDEV-33210 ]

Anson made changes - 2024-05-06 23:59

Description

Regardless of how large we set the thread_stack variable ASAN seems to complain about running out of memory at the same point for numerous tests in the main suite. This was found on clang16 and with {{-DWITH_ASAN=YES}}. The error messages suggest that this seem to be related to the test runner itself as opposed to the actual engine.

Example output for stack_thread=50M and stack_thread=30M

{code:java}
CURRENT_TEST: main.type_timestamp_hires12910Could not execute 'check-testcase' before testcase 'main.type_timestamp_hires' (res: 1):12911mysqltest: Logging to '/usr/share/mysql-test/var/2/tmp/check-mysqld_1.log'.12912mysqltest: Results saved in '/usr/share/mysql-test/var/2/tmp/check-mysqld_1.result'.12913mysqltest: Connecting to server localhost:16000 (socket /usr/share/mysql-test/var/tmp/2/mysqld.1.sock) as 'root', connection 'default', attempt 0 ...12914mysqltest: ... Connected.12915mysqltest: Start processing test commands from './include/check-testcase.test' ...12916mysqltest: At line 19: query 'let $tmp= `SELECT '$tmp' = 'No such row'`' failed: 1436: Thread stack overrun: 63936096 bytes used of a 52428800 byte stack, and 32000 bytes needed. Consider increasing the thread_stack system variable

CURRENT_TEST: main.type_timestamp_hires12878Could not execute 'check-testcase' before testcase 'main.type_timestamp_hires' (res: 1):12879mysqltest: Logging to '/usr/share/mysql-test/var/6/tmp/check-mysqld_1.log'.12880mysqltest: Results saved in '/usr/share/mysql-test/var/6/tmp/check-mysqld_1.result'.12881mysqltest: Connecting to server localhost:16060 (socket /usr/share/mysql-test/var/tmp/6/mysqld.1.sock) as 'root', connection 'default', attempt 0 ...12882mysqltest: ... Connected.12883mysqltest: Start processing test commands from './include/check-testcase.test' ...12884mysqltest: At line 19: query 'let $tmp= `SELECT '$tmp' = 'No such row'`' failed: 1436: Thread stack overrun: 42964576 bytes used of a 31457280 byte stack, and 32000 bytes needed. Consider increasing the thread_stack system variable.
{code}

Regardless of how large we set the thread_stack variable ASAN seems to complain about running out of memory at the same point for numerous tests in the main suite. This was found on 10.5 using clang16 with {{-DWITH_ASAN=YES}}. The error messages suggest that this seem to be related to the test runner itself as opposed to the actual engine.

Example output for stack_thread=50M and stack_thread=30M

{code:java}
CURRENT_TEST: main.type_timestamp_hires12910Could not execute 'check-testcase' before testcase 'main.type_timestamp_hires' (res: 1):12911mysqltest: Logging to '/usr/share/mysql-test/var/2/tmp/check-mysqld_1.log'.12912mysqltest: Results saved in '/usr/share/mysql-test/var/2/tmp/check-mysqld_1.result'.12913mysqltest: Connecting to server localhost:16000 (socket /usr/share/mysql-test/var/tmp/2/mysqld.1.sock) as 'root', connection 'default', attempt 0 ...12914mysqltest: ... Connected.12915mysqltest: Start processing test commands from './include/check-testcase.test' ...12916mysqltest: At line 19: query 'let $tmp= `SELECT '$tmp' = 'No such row'`' failed: 1436: Thread stack overrun: 63936096 bytes used of a 52428800 byte stack, and 32000 bytes needed. Consider increasing the thread_stack system variable

CURRENT_TEST: main.type_timestamp_hires12878Could not execute 'check-testcase' before testcase 'main.type_timestamp_hires' (res: 1):12879mysqltest: Logging to '/usr/share/mysql-test/var/6/tmp/check-mysqld_1.log'.12880mysqltest: Results saved in '/usr/share/mysql-test/var/6/tmp/check-mysqld_1.result'.12881mysqltest: Connecting to server localhost:16060 (socket /usr/share/mysql-test/var/tmp/6/mysqld.1.sock) as 'root', connection 'default', attempt 0 ...12882mysqltest: ... Connected.12883mysqltest: Start processing test commands from './include/check-testcase.test' ...12884mysqltest: At line 19: query 'let $tmp= `SELECT '$tmp' = 'No such row'`' failed: 1436: Thread stack overrun: 42964576 bytes used of a 31457280 byte stack, and 32000 bytes needed. Consider increasing the thread_stack system variable.
{code}

Anson made changes - 2024-05-07 02:46

Description

Regardless of how large we set the thread_stack variable ASAN seems to complain about running out of memory at the same point for numerous tests in the main suite. This was found on 10.5 using clang16 with {{-DWITH_ASAN=YES}}. The error messages suggest that this seem to be related to the test runner itself as opposed to the actual engine. This does not occur when compiling in debug mode.

Example output for stack_thread=50M and stack_thread=30M

{code:java}
CURRENT_TEST: main.type_timestamp_hires12910Could not execute 'check-testcase' before testcase 'main.type_timestamp_hires' (res: 1):12911mysqltest: Logging to '/usr/share/mysql-test/var/2/tmp/check-mysqld_1.log'.12912mysqltest: Results saved in '/usr/share/mysql-test/var/2/tmp/check-mysqld_1.result'.12913mysqltest: Connecting to server localhost:16000 (socket /usr/share/mysql-test/var/tmp/2/mysqld.1.sock) as 'root', connection 'default', attempt 0 ...12914mysqltest: ... Connected.12915mysqltest: Start processing test commands from './include/check-testcase.test' ...12916mysqltest: At line 19: query 'let $tmp= `SELECT '$tmp' = 'No such row'`' failed: 1436: Thread stack overrun: 63936096 bytes used of a 52428800 byte stack, and 32000 bytes needed. Consider increasing the thread_stack system variable

CURRENT_TEST: main.type_timestamp_hires12878Could not execute 'check-testcase' before testcase 'main.type_timestamp_hires' (res: 1):12879mysqltest: Logging to '/usr/share/mysql-test/var/6/tmp/check-mysqld_1.log'.12880mysqltest: Results saved in '/usr/share/mysql-test/var/6/tmp/check-mysqld_1.result'.12881mysqltest: Connecting to server localhost:16060 (socket /usr/share/mysql-test/var/tmp/6/mysqld.1.sock) as 'root', connection 'default', attempt 0 ...12882mysqltest: ... Connected.12883mysqltest: Start processing test commands from './include/check-testcase.test' ...12884mysqltest: At line 19: query 'let $tmp= `SELECT '$tmp' = 'No such row'`' failed: 1436: Thread stack overrun: 42964576 bytes used of a 31457280 byte stack, and 32000 bytes needed. Consider increasing the thread_stack system variable.
{code}

Daniel Black added a comment - 2024-07-05 03:37

~~MDEV-31605~~ highlights lack of assumptions that can be made.

Daniel Black added a comment - 2024-07-05 03:37 MDEV-31605 highlights lack of assumptions that can be made.

Daniel Black made changes - 2024-07-05 03:37

Link

This issue relates to ~~MDEV-31605~~ [ ~~MDEV-31605~~ ]

Daniel Black made changes - 2024-07-05 03:56

Link

This issue blocks MDBF-740 [ MDBF-740 ]

Daniel Black made changes - 2024-07-05 03:57

Component/s

Server [ 13907 ]

Daniel Black made changes - 2024-07-05 03:57

Labels

ASAN

Daniel Black made changes - 2024-07-05 03:57

Priority

Minor [ 4 ]

Critical [ 2 ]

Daniel Black made changes - 2024-07-05 04:02

Assignee

Daniel Black [ danblack ]

Daniel Black made changes - 2024-07-05 04:02

Fix Version/s

10.5 [ 23123 ]

Daniel Black made changes - 2024-07-05 04:02

Status

Open [ 1 ]

In Progress [ 3 ]

Daniel Black added a comment - 2024-07-05 04:35

sanja, can I get a review on https://github.com/MariaDB/server/pull/3387 please.

It was apparently your idea to just return 0 here.

I tested with this on clang-18 WITH_ASAN on both 10.5 and 10.11 branch which passed the perfschema tests (including --big-test), where previously there was failures.

Daniel Black added a comment - 2024-07-05 04:35 sanja , can I get a review on https://github.com/MariaDB/server/pull/3387 please. It was apparently your idea to just return 0 here. I tested with this on clang-18 WITH_ASAN on both 10.5 and 10.11 branch which passed the perfschema tests (including --big-test), where previously there was failures.

Daniel Black made changes - 2024-07-05 04:35

Assignee	Daniel Black [ danblack ]	Oleksandr Byelkin [ sanja ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Oleksandr Byelkin added a comment - 2024-07-15 10:30

I belive there are tests which check stack overflow, so switching off this for debug build should have influence on the tests, I do not see. Was all tests really run ?

Oleksandr Byelkin added a comment - 2024-07-15 10:30 I belive there are tests which check stack overflow, so switching off this for debug build should have influence on the tests, I do not see. Was all tests really run ?

Oleksandr Byelkin added a comment - 2024-07-15 12:00

It looks like the tests switched off when _SANITIZE_ADDRESS_ (checking it)

Oleksandr Byelkin added a comment - 2024-07-15 12:00 It looks like the tests switched off when _ SANITIZE_ADDRESS _ (checking it)

Oleksandr Byelkin added a comment - 2024-07-15 13:54

OK to push

Oleksandr Byelkin added a comment - 2024-07-15 13:54 OK to push

Oleksandr Byelkin made changes - 2024-07-15 13:54

Assignee	Oleksandr Byelkin [ sanja ]	Daniel Black [ danblack ]
Status	In Review [ 10002 ]	Stalled [ 10000 ]

Daniel Black added a comment - 2024-07-15 22:53

Thanks anson, sanja, TheLinuxJedi

Daniel Black added a comment - 2024-07-15 22:53 Thanks anson , sanja , TheLinuxJedi

Daniel Black made changes - 2024-07-15 22:53

Fix Version/s		10.5.26 [ 29832 ]
Fix Version/s	10.5 [ 23123 ]
Resolution		Fixed [ 1 ]
Status	Stalled [ 10000 ]	Closed [ 6 ]

JiraAutomate made changes - 2024-07-15 22:53

Fix Version/s		10.6.19 [ 29833 ]
Fix Version/s		10.11.9 [ 29834 ]
Fix Version/s		11.1.6 [ 29835 ]
Fix Version/s		11.2.5 [ 29836 ]
Fix Version/s		11.4.3 [ 29837 ]

People

Assignee:: Daniel Black

Reporter:: Anson

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 2024-05-06 23:58

Updated:: 2024-08-12 07:49

Resolved:: 2024-07-15 22:53

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration