[MDEV-21968] mysql crashes on test main.xml on arch armhf Created: 2020-03-18  Updated: 2023-01-29

Status: Open
Project: MariaDB Server
Component/s: Server, Tests
Affects Version/s: 10.5
Fix Version/s: 10.5

Type: Bug Priority: Minor
Reporter: Otto Kekäläinen Assignee: VicenČ›iu Ciorbaru
Resolution: Unresolved Votes: 0
Labels: None


 Description   

While testing 10.5 I noticed that all of the main test suite passes on armhf apart from the main.ml test, which seems to crash mysqld:

main.xml                                 w2 [ fail ]
        Test ended at 2020-03-17 23:00:40
 
CURRENT_TEST: main.xml
mysqltest: At line 46: query 'SELECT extractValue(@xml,'/a/b[1]')' failed: 2013: Lost connection to MySQL server during query
 
The result from queries just before the failure was:
< snip >
extractValue(@xml,'/a/b/c/ancestor-or-self::*')
a11 b21 c1 b22 a12
SELECT extractValue(@xml,'//@ba');
extractValue(@xml,'//@ba')
ba11 ba12 ba21 ba22
SET @xml='<a><b>b</b><c>c</c></a>';
SELECT extractValue(@xml,'/a/b');
extractValue(@xml,'/a/b')
b
SELECT extractValue(@xml,'/a/c');
extractValue(@xml,'/a/c')
c
SELECT extractValue(@xml,'/a/child::b');
extractValue(@xml,'/a/child::b')
b
SELECT extractValue(@xml,'/a/child::c');
extractValue(@xml,'/a/child::c')
c
SET @xml='<a><b>b1</b><c>c1</c><b>b2</b><c>c2</c></a>';
SELECT extractValue(@xml,'/a/b[1]');
 
More results from queries before failure can be found in /<<BUILDDIR>>/mariadb-10.5-10.5.2+maria~sid~ubuntu20.04.1~1584481678.d9afbffc006/builddir/mysql-test/var/2/log/xml.log
 
 
Server [mysqld.1 - pid: 6860, winpid: 6860, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
200317 23:00:39 [ERROR] mysqld got signal 7 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.5.2-MariaDB-1:10.5.2+maria~sid~ubuntu20.04.1~1584481678.
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=3
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62255 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0xf10006b8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xf13b3d3c thread_stack 0x49000
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0xf100fe48): SELECT extractValue(@xml,'/a/b[1]')
Connection ID (thread ID): 95
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /dev/shm/var_auto_a5jR/2/mysqld.1/data
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             31776                31776                processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       31776                31776                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: |/usr/share/apport/apport %p %s %c %d %P
 
----------SERVER LOG END-------------
 
 
 - saving '/<<BUILDDIR>>/mariadb-10.5-10.5.2+maria~sid~ubuntu20.04.1~1584481678.d9afbffc006/builddir/mysql-test/var/2/log/main.xml/' to '/<<BUILDDIR>>/mariadb-10.5-10.5.2+maria~sid~ubuntu20.04.1~1584481678.d9afbffc006/builddir/mysql-test/var/log/main.xml/'
***Warnings generated in error logs during shutdown after running tests: main.view_alias main.win_std main.xml main.udf_notembedded main.userstat-badlogin-4824 main.udf_query_cache main.win_sum main.variables_community main.win_percent_cume main.win_lead_lag main.win_first_last_value main.win_percentile main.win_insert_select main.win_min_max main.win_i_s main.varbinary main.win_orderby main.win_rank main.update_ignore_216 main.xa_sync main.win_ntile main.win_empty_over main.windows main.win_nth_value
 
200317 23:00:39 [ERROR] mysqld got signal 7 ;
Attempting backtrace. You can use the following information to find out
 
main.delete                              w2 [ pass ]    220
main.delete_returning                    w2 [ pass ]     40
main.delete_returning_grant              w2 [ pass ]     21
main.win_big-mdev-11697                  w3 [ pass ]   5644
main.delimiter_command_case_sensitivity  w2 [ pass ]     22
main.win_bit                             w3 [ pass ]     10
main.deprecated_features                 w2 [ pass ]      3
main.derived_view                        w3 [ pass ]    950
main.drop-no_root                        w3 [ pass ]      5
main.empty_server_name-8224              w3 [ pass ]   1424
main.subselect_sj                        w4 [ pass ]   6488
main.derived                             w2 [ pass ]   2874
main.empty_string_literal                w3 [ pass ]     18
main.empty_table                         w4 [ pass ]      6
main.enforce_storage_engine              w3 [ pass ]     37
main.events_embedded                     w3 [ skipped ]  Test requires: embedded server
main.drop                                w2 [ pass ]    278
main.empty_user_table                    w2 [ pass ]     25
main.events_1                            w4 [ pass ]    397
main.events_grant                        w3 [ pass ]    154
main.events_microsec                     w2 [ pass ]      3
main.events_scheduling                   w2 [ pass ]   5984
main.default_storage_engine              w1 [ pass ]  31055
--------------------------------------------------------------------------
The servers were restarted 131 times
Spent 804.036 of 288 seconds executing testcases
 
Completed: Failed 1/805 tests, 99.88% were successful.
 
Failing test(s): main.xml

Full log at https://launchpadlibrarian.net/469475432/buildlog_ubuntu-focal-armhf.mariadb-10.5_1%3A10.5.2+maria~sid~ubuntu20.04.1~1584481678.d9afbffc006_BUILDING.txt.gz

Marking priority as minor, since this does not prevent further work, but only needs to be fixed by time 10.5 is uploaded into Debian in 6 months or so.



 Comments   
Comment by Otto Kekäläinen [ 2020-04-23 ]

Temporarily disabled in https://github.com/MariaDB/server/commit/6f0b621caf76cf636771cab2e78b6e1846b31c78 - we can't have this keeping the CI red permanently, that halts all other quality assurance / testing work. Please fix and re-enable when you have a chance.

Comment by Robert Bindar [ 2020-04-24 ]

Hi otto is there anything more to know about this failure (maybe more logs)? I've tried to reproduce the crash on arm for the past 6 hours or so with no success (very slow and a bit frustrating build times).

Comment by Otto Kekäläinen [ 2020-06-06 ]

This is now coming up on the Launchpad builds again:
https://launchpadlibrarian.net/482873654/buildlog_ubuntu-groovy-armhf.mariadb-10.5_1%3A10.5.4+maria~sid~ubuntu20.10.1~1591295719.2fcff310d02.10.5_BUILDING.txt.gz

Comment by Julien Fritsch [ 2020-12-07 ]

Removing the label since otto said it happened again.

Comment by Otto Kekäläinen [ 2020-12-08 ]

I just tested building the 10.5 HEAD on armhf at https://launchpad.net/~otto/+archive/ubuntu/mariadb/+builds?build_text=&build_state=all

main.xml                                 [ disabled ]  MDEV-21968 - Crash on armhf

The build currently passes as the test is disabled.

I did this change and re-built:

diff --git a/mysql-test/unstable-tests b/mysql-test/unstable-tests
index 8536f7fba3f..a3b20603b68 100644
--- a/mysql-test/unstable-tests
+++ b/mysql-test/unstable-tests
@@ -216,7 +216,6 @@ main.wait_timeout                      : MDEV-19023 - Lost connection to MySQL s
 main.win                               : Modified in 10.5.7
 main.windows_debug                     : Added in 10.5.7
 main.xa                                : MDEV-11769 - lock wait timeout
-main.xml                               : MDEV-21968 - Crash on armhf
 
 #-----------------------------------------------------------------------

Result:

main.xml                                 w1 [ fail ]
        Test ended at 2020-12-08 01:54:11
 
CURRENT_TEST: main.xml
mysqltest: At line 46: query 'SELECT extractValue(@xml,'/a/b[1]')' failed: 2013: Lost connection to MySQL server during query
 
The result from queries just before the failure was:
< snip >
extractValue(@xml,'/a/b/c/ancestor-or-self::*')
a11 b21 c1 b22 a12
SELECT extractValue(@xml,'//@ba');
extractValue(@xml,'//@ba')
ba11 ba12 ba21 ba22
SET @xml='<a><b>b</b><c>c</c></a>';
SELECT extractValue(@xml,'/a/b');
extractValue(@xml,'/a/b')
b
SELECT extractValue(@xml,'/a/c');
extractValue(@xml,'/a/c')
c
SELECT extractValue(@xml,'/a/child::b');
extractValue(@xml,'/a/child::b')
b
SELECT extractValue(@xml,'/a/child::c');
extractValue(@xml,'/a/child::c')
c
SET @xml='<a><b>b1</b><c>c1</c><b>b2</b><c>c2</c></a>';
SELECT extractValue(@xml,'/a/b[1]');
 
More results from queries before failure can be found in /<<BUILDDIR>>/mariadb-10.5-10.5.0~ubuntu20.04.1~1607367090.7c53006b6a1+10.5/builddir/mysql-test/var/1/log/xml.log
 
 
Server [mysqld.1 - pid: 10011, winpid: 10011, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
201208  1:54:09 [ERROR] mysqld got signal 7 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.5.9-MariaDB-1:10.5.0~ubuntu20.04.1~1607367090.7c53006b6a
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=6
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62298 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0xf1c006b8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xf1e68d2c thread_stack 0x49000
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0xf1c0ff68): SELECT extractValue(@xml,'/a/b[1]')
 
Connection ID (thread ID): 211
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /dev/shm/var_auto_BMMl/1/mysqld.1/data
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             31807                31807                processes 
Max open files            1024                 1024                 files     
Max locked memory         67108864             67108864             bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       31807                31807                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: |/usr/share/apport/apport %p %s %c %d %P %E
 
----------SERVER LOG END-------------
 
 
 - saving '/<<BUILDDIR>>/mariadb-10.5-10.5.0~ubuntu20.04.1~1607367090.7c53006b6a1+10.5/builddir/mysql-test/var/1/log/main.xml/' to '/<<BUILDDIR>>/mariadb-10.5-10.5.0~ubuntu20.04.1~1607367090.7c53006b6a1+10.5/builddir/mysql-test/var/log/main.xml/'
***Warnings generated in error logs during shutdown after running tests: main.varbinary main.win_bit main.udf_notembedded main.upgrade_MDEV-23102-1 main.upgrade_MDEV-23102-2 main.win_rank main.win_avg main.udf_query_cache main.view_alias main.win_big-mdev-10092 main.win_big-mdev-11697 main.warnings_engine_disabled main.variables_community main.win_sum main.win_as_arg_to_aggregate_func main.xml main.view_grant main.update_ignore_216 main.upgrade_MDEV-19650 main.win_std main.win_empty_over main.win_big main.userstat-badlogin-4824
 
201208  1:54:09 [ERROR] mysqld got signal 7 ;
Attempting backtrace. You can use the following information to find out

Full log at https://launchpadlibrarian.net/510316528/buildlog_ubuntu-focal-armhf.mariadb-10.5_1%3A10.5.0~ubuntu20.04.1~1607367090.7c53006b6a1+10.5_BUILDING.txt.gz

Test passes OK on other platforms, its just armhf that fails.

At https://launchpad.net/~otto/+archive/ubuntu/mariadb/+builds?build_text=&build_state=all there is also a build failure in risc64 visible, which I suspect is part of MDEV-23892 (thus added comment about it there).

Comment by Otto Kekäläinen [ 2021-09-20 ]

This is still happening across 10.5, 10.6 and 10.7. Recent example: https://launchpadlibrarian.net/559046741/buildlog_ubuntu-hirsute-armhf.mariadb-10.7_1%3A10.7.0~ubuntu21.04.1~1631999241.a60c5e60c84+10.7.gitlab.ci.benchmark_BUILDING.txt.gz

Comment by Otto Kekäläinen [ 2022-02-26 ]

This is still happening, recent log from armhf on Debian upload of 10.6.7: https://buildd.debian.org/status/fetch.php?pkg=mariadb-10.6&arch=armhf&ver=1%3A10.6.7-2%7Eexp1&stamp=1645785661&raw=0 and on Lauchpad build at https://launchpadlibrarian.net/587786980/buildlog_ubuntu-jammy-armhf.mariadb-10.6_1%3A10.6.7-2~ubuntu22.04.1~1645740357.412f6ddbbf9.dev.otto_BUILDING.txt.gz

Comment by Otto Kekäläinen [ 2023-01-29 ]

This is still hapening, recent log from armhf on Ubuntu build of 1.10.11.1-2: https://launchpadlibrarian.net/648568594/buildlog_ubuntu-lunar-armhf.mariadb_1%3A10.11.1-2~ubuntu23.04.1~1674979070.4c1845984a3.dev.otto_BUILDING.txt.gz

Generated at Thu Feb 08 09:11:12 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.