ProcMon crashed with backtrace when swap space usage was high

Build tested: 1.0.7-1

getsoftwareinfo Fri Jan 20 15:00:42 2017

Name : mariadb-columnstore-platform
Version : 1.0.7
Release : 1
Architecture: x86_64
Install Date: Thu 19 Jan 2017 04:12:30 PM CST
Group : Applications/Databases
Size : 10001266
License : Copyright (c) 2016 MariaDB Corporation Ab., all rights reserved; redistributable under the terms of the GPL, see the file COPYING for details.

This issue occurred when I was running Autopilot's concurrency test suite on a single server VM with 8GB memory and 4GB swap space. procMon crash while running the queryAndDataMod test. When I run the test again after a system restart, it completed successfully. It seems that prior tests in the concurrency test suite already used much of the system memory and got the system to a swapping or soon-to-be-swapping state. The system got restarted due to swap space utilization being high. At the same time, the test script kept submitting cpimport jobs and failed.

This is seem to be the scenario that I mentioned previously. When ColumnStore opens files (not using directory IO) for operations such as cpimport, the Linux OS would cached the data in the buffer cache. Supposedly, when memory is needed, the OS would release buffer cache to make memory available. It is quite often the OS starts using the swap space before release all buffer cache. That creates the situation that ColumnStore would detect swap space utilization is high and it restarts the system.

err.log entries:

Jan 20 11:53:05 mcsce7 ServerMonitor[19187]: 05.743388 |0|0|0| C 09 CAL0000: Swap Space usage over Major threashold, perform OAM command restartSystem
Jan 20 11:55:00 mcsce7 ProcessMonitor[18704]: 46.773451 |0|0|0| C 18 CAL0000: *****Calpont Process Restarting: PrimProc, old PID = 19293
Jan 20 11:55:03 mcsce7 snmpmanager[17932]: 03.276921 |0|0|0| E 11 CAL0000: configAlarm error: API Failure return in setAlarmConfig API
Jan 20 11:55:04 mcsce7 joblist[17932]: 03.277014 |0|0|0| C 05 CAL0000: /home/builder/mariadb-columnstore-server/mariadb-columnstore-engine/dbcon/joblist/distributedenginecomm.cpp @ 382 DEC: lost connection to 127.0.0.1
Jan 20 11:55:04 mcsce7 snmpmanager[17922]: 03.278244 |0|0|0| E 11 CAL0000: configAlarm error: API Failure return in setAlarmConfig API
Jan 20 11:55:04 mcsce7 joblist[17922]: 03.733889 |0|0|0| C 05 CAL0000: /home/builder/mariadb-columnstore-server/mariadb-columnstore-engine/dbcon/joblist/distributedenginecomm.cpp @ 382 DEC: lost connection to 127.0.0.1
Jan 20 11:55:04 mcsce7 snmpmanager[17988]: 03.277518 |0|0|0| E 11 CAL0000: configAlarm error: API Failure return in setAlarmConfig API
Jan 20 11:55:04 mcsce7 snmpmanager[17904]: 03.277224 |0|0|0| E 11 CAL0000: configAlarm error: API Failure return in setAlarmConfig API
Jan 20 11:55:04 mcsce7 joblist[17988]: 03.735440 |0|0|0| C 05 CAL0000: /home/builder/mariadb-columnstore-server/mariadb-columnstore-engine/dbcon/joblist/distributedenginecomm.cpp @ 382 DEC: lost connection to 127.0.0.1
Jan 20 11:55:04 mcsce7 joblist[17904]: 03.735459 |0|0|0| C 05 CAL0000: /home/builder/mariadb-columnstore-server/mariadb-columnstore-engine/dbcon/joblist/distributedenginecomm.cpp @ 382 DEC: lost connection to 127.0.0.1
Jan 20 11:55:05 mcsce7 snmpmanager[17961]: 03.277381 |0|0|0| E 11 CAL0000: configAlarm error: API Failure return in setAlarmConfig API
Jan 20 11:55:05 mcsce7 joblist[17961]: 04.778843 |0|0|0| C 05 CAL0000: /home/builder/mariadb-columnstore-server/mariadb-columnstore-engine/dbcon/joblist/distributedenginecomm.cpp @ 382 DEC: lost connection to 127.0.0.1
Jan 20 11:55:06 mcsce7 joblist[18153]: 05.193886 |0|0|0| C 05 CAL0000: /home/builder/mariadb-columnstore-server/mariadb-columnstore-engine/dbcon/joblist/distributedenginecomm.cpp @ 382 DEC: lost connection to 127.0.0.1
Jan 20 11:55:06 mcsce7 writeenginesplit[17988]: 05.252961 |0|0|0| E 33 CAL0000: #033[0;31mReceived a Cpimport Failure from PM1#033[0m
Jan 20 11:55:06 mcsce7 writeenginesplit[17988]: 05.788091 |0|0|0| E 33 CAL0087: BulkLoad Error: #033[0;31mReceived a Cpimport Failure from PM1#033[0m
Jan 20 11:55:06 mcsce7 writeenginesplit[17932]: 05.253115 |0|0|0| E 33 CAL0000: #033[0;31mReceived a Cpimport Failure from PM1#033[0m
Jan 20 11:55:06 mcsce7 writeenginesplit[17961]: 05.253176 |0|0|0| E 33 CAL0000: #033[0;31mReceived a Cpimport Failure from PM1#033[0m
Jan 20 11:55:06 mcsce7 joblist[18224]: 05.264251 |0|0|0| C 05 CAL0000: /home/builder/mariadb-columnstore-server/mariadb-columnstore-engine/dbcon/joblist/distributedenginecomm.cpp @ 382 DEC: lost connection to 127.0.0.1
Jan 20 11:55:06 mcsce7 writeenginesplit[17904]: 05.473043 |0|0|0| E 33 CAL0000: #033[0;31mReceived a Cpimport Failure from PM1#033[0m
Jan 20 11:55:06 mcsce7 joblist[18233]: 05.775979 |0|0|0| C 05 CAL0000: /home/builder/mariadb-columnstore-server/mariadb-columnstore-engine/dbcon/joblist/distributedenginecomm.cpp @ 382 DEC:

• • • Error in `/usr/local/mariadb/columnstore/bin/ProcMon': double free or corruption (out): 0x00007f2d50000060 ***
      ======= Backtrace: =========
      /lib64/libc.so.6(+0x7d053)[0x7f2da9af3053]
      /usr/local/mariadb/columnstore/bin/ProcMon(_ZN3oam21DeviceNetworkConfig_sD1Ev+0x8d)[0x42654d]
      /usr/local/mariadb/columnstore/bin/ProcMon(ZNSt6vectorIN3oam21DeviceNetworkConfig_sESaIS1_EE13_M_insert_auxEN9gnu_cxx17normal_iteratorIPS1_S3_EERKS1+0x1a4)[0x429024]
      /usr/local/mariadb/columnstore/lib/liboamcpp.so.1(_ZN3oam3Oam15getSystemConfigERKSsRNS_18ModuleTypeConfig_sE+0x2105)[0x7f2dac1f1ef5]
      /usr/local/mariadb/columnstore/bin/ProcMon(_Z16processStatusMSGPN11messageqcpp8IOSocketE+0x47af)[0x41f80f]
      /lib64/libpthread.so.0(+0x7dc5)[0x7f2daa867dc5]
      /lib64/libc.so.6(clone+0x6d)[0x7f2da9b6cced]
      ======= Memory map: ========
      00400000-00463000 r-xp 00000000 fd:01 68944041 /usr/local/mariadb/columnstore/bin/ProcMon
      00662000-00663000 r--p 00062000 fd:01 68944041 /usr/local/mariadb/columnstore/bin/ProcMon
      00663000-00664000 rw-p 00063000 fd:01 68944041 /usr/local/mariadb/columnstore/bin/ProcMon
      00664000-00665000 rw-p 00000000 00:00 0
      00e35000-01004000 rw-p 00000000 00:00 0 [heap]
      7f2ce3fe0000-7f2ce3fe1000 ---p 00000000 00:00 0
      7f2ce3fe1000-7f2ce47e1000 rw-p 00000000 00:00 0
      7f2ce47e1000-7f2ce47e2000 ---p 00000000 00:00 0
      7f2ce47e2000-7f2ce4fe2000 rw-p 00000000 00:00 0 [stack:1929]
      7f2ce4fe2000-7f2ce4fe3000 ---p 00000000 00:00 0
      7f2ce4fe3000-7f2ce57e3000 rw-p 00000000 00:00 0
      7f2ce57e3000-7f2ce57e4000 ---p 00000000 00:00 0
      7f2ce57e4000-7f2ce5fe4000 rw-p 00000000 00:00 0
      7f2ce5fe4000-7f2ce5fe5000 ---p 00000000 00:00 0
      7f2ce5fe5000-7f2ce67e5000 rw-p 00000000 00:00 0
      7f2ce67e5000-7f2ce67e6000 ---p 00000000 00:00 0
      7f2ce67e6000-7f2ce6fe6000 rw-p 00000000 00:00 0
      7f2ce6fe6000-7f2ce6fe7000 ---p 00000000 00:00 0
      7f2ce6fe7000-7f2ce77e7000 rw-p 00000000 00:00 0
      7f2ce77e7000-7f2ce77e8000 ---p 00000000 00:00 0
      7f2ce77e8000-7f2ce7fe8000 rw-p 00000000 00:00 0
      7f2ce7fe8000-7f2ce7fe9000 ---p 00000000 00:00 0
      7f2ce7fe9000-7f2ce87e9000 rw-p 00000000 00:00 0
      7f2ce87e9000-7f2ce87ea000 ---p 00000000 00:00 0
      7f2ce87ea000-7f2ce8fea000 rw-p 00000000 00:00 0
      7f2ce8fea000-7f2ce8feb000 ---p 00000000 00:00 0
      7f2ce8feb000-7f2ce97eb000 rw-p 00000000 00:00 0
      7f2ce97eb000-7f2ce97ec000 ---p 00000000 00:00 0
      7f2ce97ec000-7f2ce9fec000 rw-p 00000000 00:00 0
      7f2ce9fec000-7f2ce9fed000 ---p 00000000 00:00 0
      7f2ce9fed000-7f2cea7ed000 rw-p 00000000 00:00 0
      7f2cea7ed000-7f2cea7ee000 ---p 00000000 00:00 0
      7f2cea7ee000-7f2ceafee000 rw-p 00000000 00:00 0
      7f2ceafee000-7f2ceafef000 ---p 00000000 00:00 0
      7f2ceafef000-7f2ceb7ef000 rw-p 00000000 00:00 0
      7f2ceb7ef000-7f2ceb7f0000 ---p 00000000 00:00 0
      7f2ceb7f0000-7f2cebff0000 rw-p 00000000 00:00 0
      7f2cebff0000-7f2cebff1000 ---p 00000000 00:00 0
      7f2cebff1000-7f2cec7f1000 rw-p 00000000 00:00 0
      7f2cec7f1000-7f2cec7f2000 ---p 00000000 00:00 0
      7f2cec7f2000-7f2cecff2000 rw-p 00000000 00:00 0
      7f2cecff2000-7f2cecff3000 ---p 00000000 00:00 0
      7f2cecff3000-7f2ced7f3000 rw-p 00000000 00:00 0
      7f2ced7f3000-7f2ced7f4000 ---p 00000000 00:00 0
      7f2ced7f4000-7f2cedff4000 rw-p 00000000 00:00 0
      7f2cedff4000-7f2cedff5000 ---p 00000000 00:00 0
      7f2cedff5000-7f2cee7f5000 rw-p 00000000 00:00 0
      7f2cee7f5000-7f2cee7f6000 ---p 00000000 00:00 0
      7f2cee7f6000-7f2ceeff6000 rw-p 00000000 00:00 0
      7f2ceeff6000-7f2ceeff7000 ---p 00000000 00:00 0
      7f2ceeff7000-7f2cef7f7000 rw-p 00000000 00:00 0
      7f2cef7f7000-7f2cef7f8000 ---p 00000000 00:00 0
      7f2cef7f8000-7f2cefff8000 rw-p 00000000 00:00 0
      7f2cefff8000-7f2cefff9000 ---p 00000000 00:00 0
      7f2cefff9000-7f2cf07f9000 rw-p 00000000 00:00 0
      7f2cf07f9000-7f2cf07fa000 ---p 00000000 00:00 0
      7f2cf07fa000-7f2cf0ffa000 rw-p 00000000 00:00 0
      7f2cf0ffa000-7f2cf0ffb000 ---p 00000000 00:00 0
      7f2cf0ffb000-7f2cf17fb000 rw-p 00000000 00:00 0
      7f2cf17fb000-7f2cf17fc000 ---p 00000000 00:00 0
      7f2cf17fc000-7f2cf1ffc000 rw-p 00000000 00:00 0
      7f2cf1ffc000-7f2cf1ffd000 ---p 00000000 00:00 0
      7f2cf1ffd000-7f2cf27fd000 rw-p 00000000 00:00 0
      7f2cf27fd000-7f2cf27fe000 ---p 00000000 00:00 0
      7f2cf27fe000-7f2cf2ffe000 rw-p 00000000 00:00 0
      7f2cf2ffe000-7f2cf2fff000 ---p 00000000 00:00 0
      7f2cf2fff000-7f2cf37ff000 rw-p 00000000 00:00 0
      7f2cf37ff000-7f2cf3800000 ---p 00000000 00:00 0
      7f2cf3800000-7f2cf4000000 rw-p 00000000 00:00 0
      7f2cf4000000-7f2cf4021000 rw-p 00000000 00:00 0
      7f2cf4021000-7f2cf8000000 ---p 00000000 00:00 0
      7f2cf8000000-7f2cf8021000 rw-p 00000000 00:00 0
      7f2cf8021000-7f2cfc000000 ---p 00000000 00:00 0
      7f2cfc7f9000-7f2cfc7fa000 ---p 00000000 00:00 0
      7f2cfc7fa000-7f2cfcffa000 rw-p 00000000 00:00 0
      7f2cfcffa000-7f2cfcffb000 ---p 00000000 00:00 0
      7f2cfcffb000-7f2cfd7fb000 rw-p 00000000 00:00 0
      7f2cfd7fb000-7f2cfd7fc000 ---p 00000000 00:00 0
      7f2cfd7fc000-7f2cfdffc000 rw-p 00000000 00:00 0
      7f2cfdffc000-7f2cfdffd000 ---p 00000000 00:00 0
      7f2cfdffd000-7f2cfe7fd000 rw-p 00000000 00:00 0
      7f2cfe7fd000-7f2cfe7fe000 ---p 00000000 00:00 0
      7f2cfe7fe000-7f2cfeffe000 rw-p 00000000 00:00 0
      7f2cfeffe000-7f2cfefff000 ---p 00000000 00:00 0
      7f2cfefff000-7f2cff7ff000 rw-p 00000000 00:00 0
      7f2cff7ff000-7f2cff800000 ---p 00000000 00:00 0
      7f2cff800000-7f2d00000000 rw-p 00000000 00:00 0
      7f2d00000000-7f2d00030000 rw-p 00000000 00:00 0
      7f2d00030000-7f2d04000000 ---p 00000000 00:00 0
      7f2d04000000-7f2d04030000 rw-p 00000000 00:00 0
      7f2d04030000-7f2d08000000 ---p 00000000 00:00 0
      7f2d08000000-7f2d08030000 rw-p 00000000 00:00 0
      7f2d08030000-7f2d0c000000 ---p 00000000 00:00 0
      7f2d0c000000-7f2d0c030000 rw-p 00000000 00:00 0
      7f2d0c030000-7f2d10000000 ---p 00000000 00:00 0
      7f2d10000000-7f2d10021000 rw-p 00000000 00:00 0
      7f2d10021000-7f2d14000000 ---p 00000000 00:00 0
      7f2d14000000-7f2d14021000 rw-p 00000000 00:00 0
      7f2d14021000-7f2d18000000 ---p 00000000 00:00 0
      7f2d18000000-7f2d18021000 rw-p 00000000 00:00 0
      7f2d18021000-7f2d1c000000 ---p 00000000 00:00 0
      7f2d1c7f9000-7f2d1c7fa000 ---p 00000000 00:00 0
      7f2d1c7fa000-7f2d1cffa000 rw-p 00000000 00:00 0
      7f2d1cffa000-7f2d1cffb000 ---p 00000000 00:00 0
      7f2d1cffb000-7f2d1d7fb000 rw-p 00000000 00:00 0
      7f2d1d7fb000-7f2d1d7fc000 ---p 00000000 00:00 0
      7f2d1d7fc000-7f2d1dffc000 rw-p 00000000 00:00 0
      7f2d1dffc000-7f2d1dffd000 ---p 00000000 00:00 0
      7f2d1dffd000-7f2d1e7fd000 rw-p 00000000 00:00 0
      7f2d1e7fd000-7f2d1e7fe000 ---p 00000000 00:00 0
      7f2d1e7fe000-7f2d1effe000 rw-p 00000000 00:00 0
      7f2d1effe000-7f2d1efff000 ---p 00000000 00:00 0
      7f2d1efff000-7f2d1f7ff000 rw-p 00000000 00:00 0
      7f2d1f7ff000-7f2d1f800000 ---p 00000000 00:00 0
      7f2d1f800000-7f2d20000000 rw-p 00000000 00:00 0
      7f2d20000000-7f2d20021000 rw-p 00000000 00:00 0
      7f2d20021000-7f2d24000000 ---p 00000000 00:00 0
      7f2d24000000-7f2d24021000 rw-p 00000000 00:00 0
      7f2d24021000-7f2d28000000 ---p 00000000 00:00 0
      7f2d28000000-7f2d28021000 rw-p 00000000 00:00 0
      7f2d28021000-7f2d2c000000 ---p 00000000 00:00 0
      7f2d2c7f9000-7f2d2c7fa000 ---p 00000000 00:00 0
      7f2d2c7fa000-7f2d2cffa000 rw-p 00000000 00:00 0
      7f2d2cffa000-7f2d2cffb000 ---p 00000000 00:00 0
      7f2d2cffb000-7f2d2d7fb000 rw-p 00000000 00:00 0
      7f2d2d7fb000-7f2d2d7fc000 ---p 00000000 00:00 0
      7f2d2d7fc000-7f2d2dffc000 rw-p 00000000 00:00 0
      7f2d2dffc000-7f2d2dffd000 ---p 00000000 00:00 0
      7f2d2dffd000-7f2d2e7fd000 rw-p 00000000 00:00 0
      7f2d2e7fd000-7f2d2e7fe000 ---p 00000000 00:00 0
      7f2d2e7fe000-7f2d2effe000 rw-p 00000000 00:00 0
      7f2d2effe000-7f2d2efff000 ---p 00000000 00:00 0