[MCOL-5636] FairThreadScheduler::sendErrorMsg and BPPSeeder::sendErrorMsg crash PP trying to send to a nullptr sock Created: 2024-01-05  Updated: 2024-01-17  Resolved: 2024-01-17

Status: Closed
Project: MariaDB ColumnStore
Component/s: PrimProc
Affects Version/s: 23.02.4, 23.10.0
Fix Version/s: 23.10.1, 23.02.7

Type: Bug Priority: Major
Reporter: Roman Assignee: Roman
Resolution: Fixed Votes: 1
Labels: None

Sprint: 2023-12
Assigned for Review: Denis Khalikov Denis Khalikov

 Description   

There is a crash that is caused by FairThreadScheduler trying to send an error over an empty socket. This might happen if the Job that causes an exception in FairThreadScheduler has been received via same node communication messaging queue.

crash trace

Date/time: 2024-01-04 14:11:47
Signal: 6
 
/usr/bin/PrimProc(+0xb74b6)[0x5613d1fb04b6]
/lib64/libpthread.so.0(+0x12cf0)[0x7fc87b116cf0]
/lib64/libc.so.6(gsignal+0x10f)[0x7fc879b31aff]
/lib64/libc.so.6(abort+0x127)[0x7fc879b04ea5]
/lib64/libc.so.6(+0x21d79)[0x7fc879b04d79]
/lib64/libc.so.6(+0x47456)[0x7fc879b2a456]
/lib64/libthreadpool.so(_ZN10threadpool14FairThreadPool12sendErrorMsgEjjN5boost10shared_ptrIN11messageqcpp8IOSocketEEE+0x1ab)[0x7fc87aeb6b7b]
/lib64/libthreadpool.so(+0x18a6e)[0x7fc87ae9da6e]
/usr/bin/PrimProc(+0xb8f97)[0x5613d1fb1f97]
/lib64/libpthread.so.0(+0x81ca)[0x7fc87b10c1ca]
/lib64/libc.so.6(clone+0x43)[0x7fc879b1ce73]

crash trace analyzer results:

Line decoded:
/lib64/libthreadpool.so(_ZN10threadpool14FairThreadPool12sendErrorMsgEjjN5boost10shared_ptrIN11messageqcpp8IOSocketEEE+0x1ab)[0x7fc87aeb6b7b]
File:
/usr/src/debug/MariaDB-/src_0/storage/columnstore/columnstore/utils/threadpool/fair_threadpool.cpp:311
 
result:
void FairThreadPool::sendErrorMsg(uint32_t id, uint32_t step, primitiveprocessor::SP_UM_IOSOCK sock)
{
  ISMPacketHeader ism;
  PrimitiveHeader ph = {0, 0, 0, 0, 0, 0};
 
  ism.Status = logging::primitiveServerErr;
  ph.UniqueID = id;
  ph.StepID = step;
  messageqcpp::ByteStream msg(sizeof(ISMPacketHeader) + sizeof(PrimitiveHeader)); <--- 311
  msg.append((uint8_t*)&ism, sizeof(ism));
  msg.append((uint8_t*)&ph, sizeof(ph));
 
  sock->write(msg);
}


Generated at Thu Feb 08 02:59:20 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.