[MCOL-4246] ControllerSegmentTable: RWLock() threw: Permission denied Created: 2020-08-13  Updated: 2021-04-19  Resolved: 2020-08-26

Status: Closed
Project: MariaDB ColumnStore
Component/s: N/A
Affects Version/s: 1.5.3
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Allen Lee (Inactive) Assignee: Todd Stoffel (Inactive)
Resolution: Duplicate Votes: 2
Labels: None
Environment:

ColumnStore 1.5/MariaDB 10.5
Deployment: Virtualized, On Premise
Platform: Red Hat Enterprise Linux


Issue Links:
Duplicate
is duplicated by MCOL-4265 /tmp/columnstore_tmp_files permission... Closed

 Description   

customer reported the following error:

terminate called after throwing an instance of 'boost::filesystem::filesystem_error'
  what():  boost::filesystem::status: Permission denied: "/tmp/columnstore_tmp_files/rdwrscratch"
200812 11:00:35 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.5.4-2-MariaDB-enterprise-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=2
max_threads=153
thread_count=4
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467809 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7ef48c0009b8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7effc40aec90 thread_stack 0x49000
??:0(my_print_stacktrace)[0x5557ad9fd3ae]
??:0(handle_fatal_signal)[0x5557ad3d73e7]
sigaction.c:0(__restore_rt)[0x7effd01d0630]
2020-08-12 11:00:35 10 [Warning] Access denied for user 'repl'@'x01tgisdb6a.vsi.uat.dbs.com'
:0(__GI_raise)[0x7effce299377]
:0(__GI_abort)[0x7effce29aa68]
??:0(__gnu_cxx::__verbose_terminate_handler())[0x7effce9937d5]
??:0(std::rethrow_exception(std::__exception_ptr::exception_ptr))[0x7effce991746]
??:0(std::terminate())[0x7effce991773]
??:0(__cxa_throw)[0x7effce991993]
??:0(boost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code*))[0x7effb652501f]
??:0(idbdatafile::IDBPolicy::init(bool, bool, std::string const&, long))[0x7effb78d46c6]
??:0(idbdatafile::IDBPolicy::configIDBPolicy())[0x7effb78d50c3]
??:0(WriteEngine::Config::checkReload())[0x7effc526e6a2]
??:0(WriteEngine::Config::initConfigCache())[0x7effc526f796]
??:0(is_columnstore_files_plugin_init(void*))[0x7effc5f6548f]
??:0(is_columnstore_files_plugin_init(void*))[0x7effc5f67584]
??:0(get_schema_tables_result(JOIN*, enum_schema_table_state))[0x5557ad250ffa]
??:0(JOIN::exec_inner())[0x5557ad235f35]
??:0(JOIN::exec())[0x5557ad236483]
??:0(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x5557ad234646]
??:0(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5557ad2351ac]
??:0(mysql_execute_command(THD*))[0x5557ad1db64a]
??:0(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x5557ad132096]
??:0(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x5557ad13a84a]
??:0(sp_instr_stmt::execute(THD*, unsigned int*))[0x5557ad13b2bc]
??:0(sp_head::execute(THD*, bool))[0x5557ad136068]
??:0(sp_head::execute_procedure(THD*, List<Item>*))[0x5557ad13761f]
??:0(comp_ne_creator(bool))[0x5557ad1cc24a]
??:0(Sql_cmd_call::execute(THD*))[0x5557ad1d021a]
??:0(mysql_execute_command(THD*))[0x5557ad1d8a22]
??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x5557ad1dcc22]
??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x5557ad1defbc]
??:0(do_command(THD*))[0x5557ad1e096b]
??:0(do_handle_one_connection(CONNECT*, bool))[0x5557ad2cb109]
??:0(handle_one_connection)[0x5557ad2cb3a4]
??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x5557ad649c3d]
pthread_create.c:0(start_thread)[0x7effd01c8ea5]
??:0(__clone)[0x7effce3618cd]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7ef48c097018): INSERT INTO columnstore_info.columnstore_files (SELECT  NAME_CONST('dbname',_latin1'gistgt' COLLATE 'latin1_swedish_ci'),  NAME_CONST('tbname',_latin1'local_cust' COLLATE 'latin1_swedish_ci'), sum(file_size), 0 FROM information_schema.columnstore_files WHERE find_in_set(object_id,  NAME_CONST('object_ids',_latin1'77734,77735,77736,77737,77738,77739,77740,77741,77742,77743,77744,77745,77746,77747,77748,77749,77750' COLLATE 'latin1_swedish_ci')))
Connection ID (thread ID): 7
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

pure virtual method called
terminate called without an active exception
200813  9:51:19 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.5.4-2-MariaDB-enterprise-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=7
max_threads=153
thread_count=9
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467809 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f67800009b8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f7284079c90 thread_stack 0x49000
??:0(my_print_stacktrace)[0x556cc97b13ae]
??:0(handle_fatal_signal)[0x556cc918b3e7]
sigaction.c:0(__restore_rt)[0x7f72b20bc630]
:0(__GI_raise)[0x7f72b0185377]
:0(__GI_abort)[0x7f72b0186a68]
??:0(__gnu_cxx::__verbose_terminate_handler())[0x7f72b087f7d5]
??:0(std::rethrow_exception(std::__exception_ptr::exception_ptr))[0x7f72b087d746]
??:0(std::terminate())[0x7f72b087d773]
??:0(__cxa_pure_virtual)[0x7f72b087e2df]
2020-08-13  9:51:19 67574 [Warning] Access denied for user 'repl'@'x01tgisdb6a.vsi.uat.dbs.com'
??:0(cal_impl_if::processWhere(st_select_lex&, cal_impl_if::gp_walk_info&, boost::shared_ptr<execplan::CalpontSelectExecutionPlan>&, List<Item>&, std::vector<Item*, std::allocator<Item*> > const&))[0x7f729d4b52da]
??:0(cal_impl_if::getSelectPlan(cal_impl_if::gp_walk_info&, st_select_lex&, boost::shared_ptr<execplan::CalpontSelectExecutionPlan>&, bool, bool, std::vector<Item*, std::allocator<Item*> > const&))[0x7f729d4c82eb]
??:0(ha_cs_impl_select_next(unsigned char*, TABLE*))[0x7f729d47438f]
??:0(ha_mcs_impl_rnd_init(TABLE*, std::vector<Item*, std::allocator<Item*> > const&))[0x7f729d47b766]
??:0(ha_mcs::rnd_init(bool))[0x7f729d463729]
??:0(handler::ha_rnd_init_with_error(bool))[0x556cc9192257]
??:0(init_read_record(READ_RECORD*, THD*, TABLE*, SQL_SELECT*, SORT_INFO*, int, bool, bool))[0x556cc92d0119]
??:0(mysql_delete(THD*, TABLE_LIST*, Item*, SQL_I_List<st_order>*, unsigned long long, unsigned long long, select_result*))[0x556cc92e7833]
??:0(mysql_execute_command(THD*))[0x556cc8f8d665]
??:0(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x556cc8ee6096]
??:0(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x556cc8eee84a]
??:0(sp_instr_stmt::execute(THD*, unsigned int*))[0x556cc8eef2bc]
??:0(sp_head::execute(THD*, bool))[0x556cc8eea068]
??:0(sp_head::execute_procedure(THD*, List<Item>*))[0x556cc8eeb61f]
??:0(comp_ne_creator(bool))[0x556cc8f8024a]
??:0(Sql_cmd_call::execute(THD*))[0x556cc8f8421a]
??:0(mysql_execute_command(THD*))[0x556cc8f8ca22]
??:0(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x556cc8ee6096]
??:0(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x556cc8eee84a]
??:0(sp_instr_stmt::execute(THD*, unsigned int*))[0x556cc8eef2bc]
??:0(sp_head::execute(THD*, bool))[0x556cc8eea068]
??:0(sp_head::execute_procedure(THD*, List<Item>*))[0x556cc8eeb61f]
??:0(comp_ne_creator(bool))[0x556cc8f8024a]
??:0(Sql_cmd_call::execute(THD*))[0x556cc8f8421a]
??:0(mysql_execute_command(THD*))[0x556cc8f8ca22]
??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x556cc8f90c22]
??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x556cc8f92fbc]
??:0(do_command(THD*))[0x556cc8f9496b]
??:0(do_handle_one_connection(CONNECT*, bool))[0x556cc907f109]
??:0(handle_one_connection)[0x556cc907f3a4]
??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x556cc93fdc3d]
pthread_create.c:0(start_thread)[0x7f72b20b4ea5]
??:0(__clone)[0x7f72b024d8cd]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f6783742780): DELETE from  gistgt.global_cust where asof_dt= NAME_CONST('p_BusinessDate',TIMESTAMP'0000-00-00 00:00:00')
Connection ID (thread ID): 67573
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

2020-08-13 10:35:57 51 [Note] Start binlog_dump to slave_server(2000), pos(x01tgisdb4a-bin.000024, 1002)
2020-08-13 10:35:57 52 [Warning] Access denied for user 'repuser'@'xxx.com' (using password: YES)
2020-08-13 10:35:57 53 [Warning] Access denied for user 'repl'@'xxx.com'
2020-08-13 10:35:59 54 [Warning] Access denied for user 'repl'@'xxx.com'
2020-08-13 10:36:01 55 [Warning] Access denied for user 'repl'@'xxx.com'
..................
2020-08-13 10:36:28 81 [Warning] Access denied for user 'repl'@'xxx.xxx.com'
2020-08-13 10:36:30 82 [Warning] Access denied for user 'repl'@'xxx.xxx.com'
RWLock failed to attach to the InfiniDB-shm-00020000 shared mem segment, got Permission denied
ControllerSegmentTable: RWLock() threw: Permission denied
terminate called after throwing an instance of 'boost::interprocess::interprocess_exception'
  what():  Permission denied
200813 10:36:30 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.5.4-2-MariaDB-enterprise-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=3
max_threads=153
thread_count=5
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467809 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f00740009b8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f0be00b6c90 thread_stack 0x49000
??:0(my_print_stacktrace)[0x55d0d97953ae]
??:0(handle_fatal_signal)[0x55d0d916f3e7]
sigaction.c:0(__restore_rt)[0x7f0beac3f630]
:0(__GI_raise)[0x7f0be8d08377]
:0(__GI_abort)[0x7f0be8d09a68]
??:0(__gnu_cxx::__verbose_terminate_handler())[0x7f0be94027d5]
??:0(std::rethrow_exception(std::__exception_ptr::exception_ptr))[0x7f0be9400746]
??:0(std::terminate())[0x7f0be9400773]
??:0(std::terminate())[0x7f0be9400786]
??:0(__cxa_call_unexpected)[0x7f0be94003c2]
??:0(BRM::DBRM::DBRM(bool))[0x7f0bd25a13a2]
??:0(execplan::SessionManager::SessionManager())[0x7f0bd45648a0]
??:0(execplan::CalpontSystemCatalog::CalpontSystemCatalog())[0x7f0bd4524cc5]
??:0(execplan::CalpontSystemCatalog::makeCalpontSystemCatalog(unsigned int))[0x7f0bd4525448]
??:0(ha_cs_impl_pushdown_init(mcs_handler_info*, TABLE*))[0x7f0be094633f]
??:0(ha_columnstore_select_handler::init_scan())[0x7f0be092b2a8]
??:0(Pushdown_select::execute())[0x55d0d8fec44c]
??:0(JOIN::exec_inner())[0x55d0d8fcd8bc]
??:0(JOIN::exec())[0x55d0d8fce483]
??:0(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x55d0d8fcc646]
??:0(handle_select(THD*, LEX*, select_result*, unsigned long))[0x55d0d8fcd1ac]
/usr/sbin/mariadbd(+0x62ce1b)[0x55d0d8e32e1b]
??:0(mysql_execute_command(THD*))[0x55d0d8f71c6c]
??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x55d0d8f74c22]
??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x55d0d8f76fbc]
??:0(do_command(THD*))[0x55d0d8f7896b]
??:0(do_handle_one_connection(CONNECT*, bool))[0x55d0d9063109]
??:0(handle_one_connection)[0x55d0d90633a4]
??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x55d0d93e1c3d]
pthread_create.c:0(start_thread)[0x7f0beac37ea5]
??:0(__clone)[0x7f0be8dd08cd]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f00740161a0): select count(1) from gistgt.global_cust
Connection ID (thread ID): 72
Status: NOT_KILLED



 Comments   
Comment by Todd Stoffel (Inactive) [ 2020-08-20 ]

What Linux user are the Columnstore processes running as and which user owns /tmp/columnstore_tmp_files/?

Comment by Todd Stoffel (Inactive) [ 2020-08-26 ]

Work around:

chmod 755 /tmp/columnstore_tmp_files

Generated at Thu Feb 08 02:48:54 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.