server crash on complex select (Assertion failure in file rem0rec.cc line 815)

InnoDB: Assertion failure in file /home/buildbot/buildbot/build/storage/innobase/rem/rem0rec.cc line 815

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/

InnoDB: about forcing recovery.

220624 8:11:56 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.3.28-MariaDB-log

key_buffer_size=134217728

read_buffer_size=131072

max_used_connections=178

max_threads=5002

thread_count=186

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11128102 K bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f2fc00009a8

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7f3048148eb0 thread_stack 0x49000

mysys/stacktrace.c:175(my_print_stacktrace)[0x5633d9a8356e]

sql/signal_handler.cc:209(handle_fatal_signal)[0x5633d94e30df]

sigaction.c:0(__restore_rt)[0x7f30629c3630]

:0(__GI_raise)[0x7f30617ce387]

:0(__GI_abort)[0x7f30617cfa78]

ut/ut0rbt.cc:218(rbt_tree_add_child(ib_rbt_t const*, ib_rbt_bound_t*, ib_rbt_node_t*) [clone .isra.4] [clone .part.5])[0x5633d921ce68]

rem/rem0rec.cc:815(rec_get_offsets_func(unsigned char const*, dict_index_t const*, unsigned short*, bool, unsigned long, mem_block_info_t**))[0x5633d9717c13]

fts/fts0fts.cc:5166(fts_get_doc_id_from_rec(dict_table_t*, unsigned char const*, dict_index_t const*, mem_block_info_t*))[0x5633d98a3a48]

row/row0sel.cc:3175(row_sel_store_mysql_rec(unsigned char*, row_prebuilt_t*, unsigned char const*, dtuple_t const*, bool, dict_index_t const*, unsigned short const*))[0x5633d921a723]

row/row0sel.cc:4478(row_search_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long))[0x5633d976207d]

handler/ha_innodb.cc:9274(ha_innobase::index_read(unsigned char*, unsigned char const*, unsigned int, ha_rkey_function))[0x5633d968b443]

sql/handler.cc:2906(handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function))[0x5633d94e8b78]

sql/sql_select.cc:20387(join_read_key2(THD*, st_join_table*, TABLE*, st_table_ref*))[0x5633d9346c6a]

sql/sql_select.cc:19709(sub_select(JOIN*, st_join_table*, bool))[0x5633d93366b9]

sql/sql_class.h:3890(THD::get_stmt_da())[0x5633d932ba5c]

sql/sql_select.cc:19719(sub_select(JOIN*, st_join_table*, bool))[0x5633d933672e]

sql/sql_select.cc:19252(do_select)[0x5633d935958b]

sql/sql_select.cc:3919(JOIN::exec())[0x5633d93597a3]

sql/item_subselect.cc:3929(subselect_single_select_engine::exec())[0x5633d9593375]

sql/item_subselect.cc:741(Item_subselect::exec())[0x5633d9591d6a]

sql/item_subselect.cc:1622(Item_exists_subselect::val_int())[0x5633d9592711]

sql/item.cc:9901(Item_cache_int::cache_value())[0x5633d94f4b2a]

sql/sql_list.h:440(base_list_iterator::next_fast())[0x5633d92b3037]

sql/sql_select.cc:3997(JOIN::exec_inner())[0x5633d935955c]

sql/sql_select.cc:3919(JOIN::exec())[0x5633d93597a3]

sql/item_subselect.cc:3929(subselect_single_select_engine::exec())[0x5633d9593375]

sql/item_subselect.cc:741(Item_subselect::exec())[0x5633d9591d6a]

sql/item_subselect.cc:1329(Item_singlerow_subselect::val_int())[0x5633d959242e]

sql/item.cc:6879(Item::save_int_in_field(Field*, bool))[0x5633d950660f]

sql/item.cc:6889(Item::save_in_field(Field*, bool))[0x5633d94f7c06]

sql/sql_class.h:3880(Field::sp_prepare_and_store_item(THD*, Item**))[0x5633d94c3c33]

sql/sql_class.h:6817(Sp_eval_expr_state::stop())[0x5633d926cf17]

sql/sp_rcontext.cc:624(sp_rcontext::set_variable(THD*, unsigned int, Item**))[0x5633d927b013]

sql/sp_head.cc:3733(sp_instr_set::exec_core(THD*, unsigned int*))[0x5633d927398f]

sql/sp_head.cc:3434(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x5633d9272dcb]

sql/sp_head.cc:1379(sp_head::execute(THD*, bool))[0x5633d926ea97]

sql/sp_head.cc:2124(sp_head::execute_function(THD*, Item**, unsigned int, Field*, sp_rcontext**, Query_arena*))[0x5633d9270652]

sql/item.cc:2997(Item_sp::execute_impl(THD*, Item**, unsigned int))[0x5633d9500db8]

sql/item.cc:2905(Item_sp::execute(THD*, bool*, Item**, unsigned int))[0x5633d9501001]

sql/item_func.h:2927(Item_func_sp::val_int())[0x5633d95647a1]

sql/item_cmpfunc.cc:917(Arg_comparator::compare_int_signed())[0x5633d95142e9]

sql/item_cmpfunc.cc:1786(Item_func_gt::val_int())[0x5633d9517a8a]

sql/sql_type.cc:3288(Type_handler_int_result::Item_val_bool(Item*) const)[0x5633d942c560]

sql/item_cmpfunc.cc:5092(Item_cond_or::val_int())[0x5633d95158ba]

sql/sql_type.cc:3288(Type_handler_int_result::Item_val_bool(Item*) const)[0x5633d942c560]

sql/item_cmpfunc.cc:5074(Item_cond_and::val_int())[0x5633d951583a]

sql/sql_select.cc:19809(evaluate_join_record(JOIN*, st_join_table*, int))[0x5633d932b7fd]

sql/sql_select.cc:19719(sub_select(JOIN*, st_join_table*, bool))[0x5633d933672e]

sql/sql_select.cc:19252(do_select)[0x5633d935958b]

sql/sql_select.cc:3919(JOIN::exec())[0x5633d93597a3]

sql/sql_select.cc:4325(mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5633d93598fa]

sql/sql_derived.cc:1168(mysql_derived_fill(THD*, LEX*, TABLE_LIST*))[0x5633d92c9acb]

sql/sql_derived.cc:199(mysql_handle_single_derived(LEX*, TABLE_LIST*, unsigned int))[0x5633d92c97dc]

sql/sql_select.cc:12846(st_join_table::preread_init())[0x5633d9336489]

sql/sql_select.cc:20642(join_init_read_record(st_join_table*))[0x5633d9347328]

sql/sql_join_cache.cc:2253(JOIN_CACHE::join_matching_records(bool))[0x5633d9407046]

sql/sql_join_cache.cc:2090(JOIN_CACHE::join_records(bool))[0x5633d9406b1c]

sql/sql_join_cache.cc:2136(JOIN_CACHE::join_records(bool))[0x5633d9406b54]

sql/sql_join_cache.cc:2136(JOIN_CACHE::join_records(bool))[0x5633d9406b54]

sql/sql_join_cache.cc:2136(JOIN_CACHE::join_records(bool))[0x5633d9406b54]

sql/sql_select.cc:19492(sub_select_cache(JOIN*, st_join_table*, bool))[0x5633d9336a9e]

sql/sql_select.cc:19253(do_select)[0x5633d935943b]

sql/sql_select.cc:3919(JOIN::exec())[0x5633d93597a3]

sql/sql_select.cc:4325(mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5633d93598fa]

sql/sql_select.cc:382(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5633d935a41c]

sql/sql_parse.cc:6318(execute_sqlcom_select(THD*, TABLE_LIST*))[0x5633d9209739]

sql/sql_parse.cc:3848(mysql_execute_command(THD*))[0x5633d9303007]

sql/sql_parse.cc:7841(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x5633d9305f4c]

sql/sql_parse.cc:1855(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x5633d9307d20]

sql/sql_parse.cc:1401(do_command(THD*))[0x5633d9309a4b]

sql/sql_connect.cc:1403(do_handle_one_connection(CONNECT*))[0x5633d93dff66]

sql/sql_connect.cc:1310(handle_one_connection)[0x5633d93e007d]

pthread_create.c:0(start_thread)[0x7f30629bbea5]

/lib64/libc.so.6(clone+0x6d)[0x7f306189696d]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x7f2fc000fd58): ELECT

                dateTable.targetDate as date

               ,ifnull(jobResult.jobTotalCount, 0) as jobTotalCount

               ,ifnull(jobResult.jobSuccessCount, 0) as jobSuccessCount

               ,ifnull(jobResult.jobFailCount, 0) as jobFailCount

               ,ifnull(jobResult.jobStopCount, 0) as jobStopCount

               ,ifnull(jobResult.jobTimeoutCount, 0) as jobTimeoutCount

          FROM

          (

            SELECT a.targetDate

            FROM (

        SELECT date_sub(@endDatetime, interval -1 SECOND) - INTERVAL (a.a + (10 * b.a) + (100 * c.a) + (1000 * d.a)) hour as targetDate

                FROM (select 0 as a union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) as a

                CROSS JOIN (select 0 as a union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) as b

                CROSS JOIN (select 0 as a union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) as c

                CROSS JOIN (select 0 as a union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) as d

            ) a

        WHERE a.targetDate BETWEEN @startDatetime AND @endDatetime

        ) dateTable

        LEFT OUTER JOIN

        (

        SELECT

                jobHistory.applyStartTime as jobStartTime

               ,count(*) as jobTotalCount

               ,SUM(case when jobHistory.jobStatus = '1' THEN 1 ELSE 0 END) as jobSuccessCount

               ,SUM(case when jobHistory.jobStatus = '2' THEN 1 ELSE 0 END) as jobFailCount

               ,SUM(case when jobHistory.jobStatus = '3' THEN 1 ELSE 0 END) as jobStopCount

               ,SUM(case when jobHistory.jobStatus = '4' THEN 1 ELSE 0 END) as jobTimeoutCount

          FROM (

                SELECT date_format(date_sub(start_time, interval @interval MINUTE), '%Y-%m-%d %H:00:00') as applyStartTime

                       ,start_time as startTime

                       ,result_code as jobStatus

                FROM tenant004_rpa.tb_running_result_log his

                LEFT OUTER JOIN tenant004_rpa.tb_common_object process

                ON his.TARGET_ID = process.OBJECT_ID

                AND his.VERSION = process.VERSION

                LEFT OUTER JOIN tenant004_rpa.v_job_history_workflow workflow

                ON workflow.processJobId = his.JOB_ID

                LEFT OUTER JOIN tenant004_rpa.v_job_history_bot bot

                ON bot.botSelectType = his.BOT_SELECT_TYPE

                AND bot.botId = his.BOT_ID

                 WHERE job_result_type != 9

                    AND

                    (tenant004_rpa.fn_user_is_admin('abcd') > 0 OR

                        (

                        ((his.job_type = 1 OR his.job_type = 11) AND

                        (process.CREATE_USR_ID = 'abcd' OR

                        his.TARGET_ID IN (

                        SELECT upper_map.OBJECT_ID

                        FROM tenant004_rpa.tb_object_map AS upper_map

                        INNER JOIN tenant004_rpa.tb_rpa_group_object_mapping AS OBJ_MAPPING ON upper_map.UPPER_OBJECT_ID = OBJ_MAPPING.OBJECT_ID

                        INNER JOIN tenant004_rpa.tb_rpa_group_user_mapping AS USER_MAPPING ON OBJ_MAPPING.GROUP_ID = USER_MAPPING.GROUP_ID

                        WHERE OBJ_MAPPING.OBJECT_TYPE_ID = 1 AND OBJ_MAPPING.AUTH_E = 1 AND USER_MAPPING.USER_ID = 'abcd'

                        )))

                        OR

                        ((his.job_type = 2 OR his.job_type = 12) AND

                        (workflow.workflowCreateUsrId = 'abcd' OR

                        workflow.workflowId IN (

                        SELECT OBJECT_ID

                        FROM tenant004_rpa.tb_rpa_group_object_mapping AS OBJ_MAPPING

                        INNER JOIN tenant004_rpa.tb_rpa_group_user_mapping AS USER_MAPPING ON OBJ_MAPPING.GROUP_ID = USER_MAPPING.GROUP_ID

                        WHERE OBJ_MAPPING.OBJECT_TYPE_ID = 10 AND OBJ_MAPPING.AUTH_E = 1 AND USER_MAPPING.USER_ID = 'abcd'

                        )))

                        )

                    )

                        AND his.start_time BETWEEN '2022-02-03T15:00:00.000Z' AND '2022-02-04T14:59:59.000Z'

        ) AS jobHistory

        GROUP BY jobHistory.applyStartTime

        ) jobResult

        ON date_format(dateTable.targetDate, '%Y-%m-%d %H:%m:%s') = date_format(jobResult.jobStartTime, '%Y-%m-%d %H:%m:%s')

        ORDER BY dateTable.targetDate

Connection ID (thread ID): 20

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains

information that should help you find out what is causing the crash.

Writing a core file...

Working directory at /DATA/rpa/pkgs/mariadb/data

Resource Limits:

Limit                     Soft Limit           Hard Limit           Units     

Max cpu time              unlimited            unlimited            seconds   

Max file size             unlimited            unlimited            bytes     

Max data size             unlimited            unlimited            bytes     

Max stack size            8388608              unlimited            bytes     

Max core file size        unlimited            unlimited            bytes     

Max resident set          unlimited            unlimited            bytes     

Max processes             4096                 126677               processes 

Max open files            40960                40960                files     

Max locked memory         65536                65536                bytes     

Max address space         unlimited            unlimited            bytes     

Max file locks            unlimited            unlimited            locks     

Max pending signals       126677               126677               signals   

Max msgqueue size         819200               819200               bytes     

Max nice priority         0                    0                    

Max realtime priority     0                    0                    

Max realtime timeout      unlimited            unlimited            us        

Core pattern: core