Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4246

ControllerSegmentTable: RWLock() threw: Permission denied

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 1.5.3
    • Fix Version/s: N/A
    • Component/s: N/A
    • Labels:
      None
    • Environment:
      ColumnStore 1.5/MariaDB 10.5
      Deployment: Virtualized, On Premise
      Platform: Red Hat Enterprise Linux

      Description

      customer reported the following error:

      terminate called after throwing an instance of 'boost::filesystem::filesystem_error'
        what():  boost::filesystem::status: Permission denied: "/tmp/columnstore_tmp_files/rdwrscratch"
      200812 11:00:35 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.5.4-2-MariaDB-enterprise-log
      key_buffer_size=134217728
      read_buffer_size=131072
      max_used_connections=2
      max_threads=153
      thread_count=4
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467809 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7ef48c0009b8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7effc40aec90 thread_stack 0x49000
      ??:0(my_print_stacktrace)[0x5557ad9fd3ae]
      ??:0(handle_fatal_signal)[0x5557ad3d73e7]
      sigaction.c:0(__restore_rt)[0x7effd01d0630]
      2020-08-12 11:00:35 10 [Warning] Access denied for user 'repl'@'x01tgisdb6a.vsi.uat.dbs.com'
      :0(__GI_raise)[0x7effce299377]
      :0(__GI_abort)[0x7effce29aa68]
      ??:0(__gnu_cxx::__verbose_terminate_handler())[0x7effce9937d5]
      ??:0(std::rethrow_exception(std::__exception_ptr::exception_ptr))[0x7effce991746]
      ??:0(std::terminate())[0x7effce991773]
      ??:0(__cxa_throw)[0x7effce991993]
      ??:0(boost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code*))[0x7effb652501f]
      ??:0(idbdatafile::IDBPolicy::init(bool, bool, std::string const&, long))[0x7effb78d46c6]
      ??:0(idbdatafile::IDBPolicy::configIDBPolicy())[0x7effb78d50c3]
      ??:0(WriteEngine::Config::checkReload())[0x7effc526e6a2]
      ??:0(WriteEngine::Config::initConfigCache())[0x7effc526f796]
      ??:0(is_columnstore_files_plugin_init(void*))[0x7effc5f6548f]
      ??:0(is_columnstore_files_plugin_init(void*))[0x7effc5f67584]
      ??:0(get_schema_tables_result(JOIN*, enum_schema_table_state))[0x5557ad250ffa]
      ??:0(JOIN::exec_inner())[0x5557ad235f35]
      ??:0(JOIN::exec())[0x5557ad236483]
      ??:0(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5557ad234646]
      ??:0(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5557ad2351ac]
      ??:0(mysql_execute_command(THD*))[0x5557ad1db64a]
      ??:0(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x5557ad132096]
      ??:0(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x5557ad13a84a]
      ??:0(sp_instr_stmt::execute(THD*, unsigned int*))[0x5557ad13b2bc]
      ??:0(sp_head::execute(THD*, bool))[0x5557ad136068]
      ??:0(sp_head::execute_procedure(THD*, List<Item>*))[0x5557ad13761f]
      ??:0(comp_ne_creator(bool))[0x5557ad1cc24a]
      ??:0(Sql_cmd_call::execute(THD*))[0x5557ad1d021a]
      ??:0(mysql_execute_command(THD*))[0x5557ad1d8a22]
      ??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x5557ad1dcc22]
      ??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x5557ad1defbc]
      ??:0(do_command(THD*))[0x5557ad1e096b]
      ??:0(do_handle_one_connection(CONNECT*, bool))[0x5557ad2cb109]
      ??:0(handle_one_connection)[0x5557ad2cb3a4]
      ??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x5557ad649c3d]
      pthread_create.c:0(start_thread)[0x7effd01c8ea5]
      ??:0(__clone)[0x7effce3618cd]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7ef48c097018): INSERT INTO columnstore_info.columnstore_files (SELECT  NAME_CONST('dbname',_latin1'gistgt' COLLATE 'latin1_swedish_ci'),  NAME_CONST('tbname',_latin1'local_cust' COLLATE 'latin1_swedish_ci'), sum(file_size), 0 FROM information_schema.columnstore_files WHERE find_in_set(object_id,  NAME_CONST('object_ids',_latin1'77734,77735,77736,77737,77738,77739,77740,77741,77742,77743,77744,77745,77746,77747,77748,77749,77750' COLLATE 'latin1_swedish_ci')))
      Connection ID (thread ID): 7
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
      

      pure virtual method called
      terminate called without an active exception
      200813  9:51:19 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.5.4-2-MariaDB-enterprise-log
      key_buffer_size=134217728
      read_buffer_size=131072
      max_used_connections=7
      max_threads=153
      thread_count=9
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467809 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7f67800009b8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f7284079c90 thread_stack 0x49000
      ??:0(my_print_stacktrace)[0x556cc97b13ae]
      ??:0(handle_fatal_signal)[0x556cc918b3e7]
      sigaction.c:0(__restore_rt)[0x7f72b20bc630]
      :0(__GI_raise)[0x7f72b0185377]
      :0(__GI_abort)[0x7f72b0186a68]
      ??:0(__gnu_cxx::__verbose_terminate_handler())[0x7f72b087f7d5]
      ??:0(std::rethrow_exception(std::__exception_ptr::exception_ptr))[0x7f72b087d746]
      ??:0(std::terminate())[0x7f72b087d773]
      ??:0(__cxa_pure_virtual)[0x7f72b087e2df]
      2020-08-13  9:51:19 67574 [Warning] Access denied for user 'repl'@'x01tgisdb6a.vsi.uat.dbs.com'
      ??:0(cal_impl_if::processWhere(st_select_lex&, cal_impl_if::gp_walk_info&, boost::shared_ptr<execplan::CalpontSelectExecutionPlan>&, List<Item>&, std::vector<Item*, std::allocator<Item*> > const&))[0x7f729d4b52da]
      ??:0(cal_impl_if::getSelectPlan(cal_impl_if::gp_walk_info&, st_select_lex&, boost::shared_ptr<execplan::CalpontSelectExecutionPlan>&, bool, bool, std::vector<Item*, std::allocator<Item*> > const&))[0x7f729d4c82eb]
      ??:0(ha_cs_impl_select_next(unsigned char*, TABLE*))[0x7f729d47438f]
      ??:0(ha_mcs_impl_rnd_init(TABLE*, std::vector<Item*, std::allocator<Item*> > const&))[0x7f729d47b766]
      ??:0(ha_mcs::rnd_init(bool))[0x7f729d463729]
      ??:0(handler::ha_rnd_init_with_error(bool))[0x556cc9192257]
      ??:0(init_read_record(READ_RECORD*, THD*, TABLE*, SQL_SELECT*, SORT_INFO*, int, bool, bool))[0x556cc92d0119]
      ??:0(mysql_delete(THD*, TABLE_LIST*, Item*, SQL_I_List<st_order>*, unsigned long long, unsigned long long, select_result*))[0x556cc92e7833]
      ??:0(mysql_execute_command(THD*))[0x556cc8f8d665]
      ??:0(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x556cc8ee6096]
      ??:0(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x556cc8eee84a]
      ??:0(sp_instr_stmt::execute(THD*, unsigned int*))[0x556cc8eef2bc]
      ??:0(sp_head::execute(THD*, bool))[0x556cc8eea068]
      ??:0(sp_head::execute_procedure(THD*, List<Item>*))[0x556cc8eeb61f]
      ??:0(comp_ne_creator(bool))[0x556cc8f8024a]
      ??:0(Sql_cmd_call::execute(THD*))[0x556cc8f8421a]
      ??:0(mysql_execute_command(THD*))[0x556cc8f8ca22]
      ??:0(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x556cc8ee6096]
      ??:0(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x556cc8eee84a]
      ??:0(sp_instr_stmt::execute(THD*, unsigned int*))[0x556cc8eef2bc]
      ??:0(sp_head::execute(THD*, bool))[0x556cc8eea068]
      ??:0(sp_head::execute_procedure(THD*, List<Item>*))[0x556cc8eeb61f]
      ??:0(comp_ne_creator(bool))[0x556cc8f8024a]
      ??:0(Sql_cmd_call::execute(THD*))[0x556cc8f8421a]
      ??:0(mysql_execute_command(THD*))[0x556cc8f8ca22]
      ??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x556cc8f90c22]
      ??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x556cc8f92fbc]
      ??:0(do_command(THD*))[0x556cc8f9496b]
      ??:0(do_handle_one_connection(CONNECT*, bool))[0x556cc907f109]
      ??:0(handle_one_connection)[0x556cc907f3a4]
      ??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x556cc93fdc3d]
      pthread_create.c:0(start_thread)[0x7f72b20b4ea5]
      ??:0(__clone)[0x7f72b024d8cd]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7f6783742780): DELETE from  gistgt.global_cust where asof_dt= NAME_CONST('p_BusinessDate',TIMESTAMP'0000-00-00 00:00:00')
      Connection ID (thread ID): 67573
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
      

      2020-08-13 10:35:57 51 [Note] Start binlog_dump to slave_server(2000), pos(x01tgisdb4a-bin.000024, 1002)
      2020-08-13 10:35:57 52 [Warning] Access denied for user 'repuser'@'xxx.com' (using password: YES)
      2020-08-13 10:35:57 53 [Warning] Access denied for user 'repl'@'xxx.com'
      2020-08-13 10:35:59 54 [Warning] Access denied for user 'repl'@'xxx.com'
      2020-08-13 10:36:01 55 [Warning] Access denied for user 'repl'@'xxx.com'
      ..................
      2020-08-13 10:36:28 81 [Warning] Access denied for user 'repl'@'xxx.xxx.com'
      2020-08-13 10:36:30 82 [Warning] Access denied for user 'repl'@'xxx.xxx.com'
      RWLock failed to attach to the InfiniDB-shm-00020000 shared mem segment, got Permission denied
      ControllerSegmentTable: RWLock() threw: Permission denied
      terminate called after throwing an instance of 'boost::interprocess::interprocess_exception'
        what():  Permission denied
      200813 10:36:30 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.5.4-2-MariaDB-enterprise-log
      key_buffer_size=134217728
      read_buffer_size=131072
      max_used_connections=3
      max_threads=153
      thread_count=5
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467809 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7f00740009b8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f0be00b6c90 thread_stack 0x49000
      ??:0(my_print_stacktrace)[0x55d0d97953ae]
      ??:0(handle_fatal_signal)[0x55d0d916f3e7]
      sigaction.c:0(__restore_rt)[0x7f0beac3f630]
      :0(__GI_raise)[0x7f0be8d08377]
      :0(__GI_abort)[0x7f0be8d09a68]
      ??:0(__gnu_cxx::__verbose_terminate_handler())[0x7f0be94027d5]
      ??:0(std::rethrow_exception(std::__exception_ptr::exception_ptr))[0x7f0be9400746]
      ??:0(std::terminate())[0x7f0be9400773]
      ??:0(std::terminate())[0x7f0be9400786]
      ??:0(__cxa_call_unexpected)[0x7f0be94003c2]
      ??:0(BRM::DBRM::DBRM(bool))[0x7f0bd25a13a2]
      ??:0(execplan::SessionManager::SessionManager())[0x7f0bd45648a0]
      ??:0(execplan::CalpontSystemCatalog::CalpontSystemCatalog())[0x7f0bd4524cc5]
      ??:0(execplan::CalpontSystemCatalog::makeCalpontSystemCatalog(unsigned int))[0x7f0bd4525448]
      ??:0(ha_cs_impl_pushdown_init(mcs_handler_info*, TABLE*))[0x7f0be094633f]
      ??:0(ha_columnstore_select_handler::init_scan())[0x7f0be092b2a8]
      ??:0(Pushdown_select::execute())[0x55d0d8fec44c]
      ??:0(JOIN::exec_inner())[0x55d0d8fcd8bc]
      ??:0(JOIN::exec())[0x55d0d8fce483]
      ??:0(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x55d0d8fcc646]
      ??:0(handle_select(THD*, LEX*, select_result*, unsigned long))[0x55d0d8fcd1ac]
      /usr/sbin/mariadbd(+0x62ce1b)[0x55d0d8e32e1b]
      ??:0(mysql_execute_command(THD*))[0x55d0d8f71c6c]
      ??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x55d0d8f74c22]
      ??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x55d0d8f76fbc]
      ??:0(do_command(THD*))[0x55d0d8f7896b]
      ??:0(do_handle_one_connection(CONNECT*, bool))[0x55d0d9063109]
      ??:0(handle_one_connection)[0x55d0d90633a4]
      ??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x55d0d93e1c3d]
      pthread_create.c:0(start_thread)[0x7f0beac37ea5]
      ??:0(__clone)[0x7f0be8dd08cd]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7f00740161a0): select count(1) from gistgt.global_cust
      Connection ID (thread ID): 72
      Status: NOT_KILLED
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              toddstoffel Todd Stoffel
              Reporter:
              allen.lee@mariadb.com Allen Lee
              Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.