2020-01-31 3:19:01 1226743 [Warning] Aborted connection 1226743 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:19:21 1226829 [Warning] Aborted connection 1226829 to db: 'unconnected' user: 'wpwsutv' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:19:42 1226836 [Warning] Aborted connection 1226836 to db: 'unconnected' user: 'wpwsutv' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:20:02 1226842 [Warning] Aborted connection 1226842 to db: 'unconnected' user: 'wpwsutv' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:20:23 1226847 [Warning] Aborted connection 1226847 to db: 'unconnected' user: 'wpwsutv' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:20:42 1226852 [Warning] Aborted connection 1226852 to db: 'unconnected' user: 'wpwsutv' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:21:03 1226860 [Warning] Aborted connection 1226860 to db: 'unconnected' user: 'wpwsutv' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:21:23 1226868 [Warning] Aborted connection 1226868 to db: 'unconnected' user: 'wpwsutv' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:21:42 1226877 [Warning] Aborted connection 1226877 to db: 'unconnected' user: 'wpwsutv' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:22:01 1226887 [Warning] Aborted connection 1226887 to db: 'unconnected' user: 'wpwsutv' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:22:19 1226894 [Warning] Aborted connection 1226894 to db: 'unconnected' user: 'wpwsutv' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:22:37 1226903 [Warning] Aborted connection 1226903 to db: 'unconnected' user: 'wpwsutv' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:22:51 1226915 [Warning] Aborted connection 1226915 to db: 'unconnected' user: 'wpwsutv' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:32:55 1227146 [Warning] Aborted connection 1227146 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:46:00 1227522 [Warning] Aborted connection 1227522 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:46:11 1227561 [Warning] Aborted connection 1227561 to db: 'unconnected' user: 'ksarn' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 3:48:48 1206336 [Warning] Aborted connection 1206336 to db: 'gots' user: 'gotsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 3:59:12 1224803 [Warning] Aborted connection 1224803 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 4:17:17 1228376 [Warning] Aborted connection 1228376 to db: 'shockeripe' user: 'shockeripe' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 4:18:09 1225780 [Warning] Aborted connection 1225780 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 4:42:25 1229093 [Warning] Aborted connection 1229093 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 4:42:30 1229098 [Warning] Aborted connection 1229098 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 5:04:51 1229736 [Warning] Aborted connection 1229736 to db: 'openrecords' user: 'openrecords' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 5:05:12 1229788 [Warning] Aborted connection 1229788 to db: 'unconnected' user: 'openrecords' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 5:16:56 1226591 [Warning] Aborted connection 1226591 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 5:18:13 1230131 [Warning] Aborted connection 1230131 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 5:22:34 1226787 [Warning] Aborted connection 1226787 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 5:54:21 1231148 [Warning] Aborted connection 1231148 to db: 'wpsem' user: 'wpsem' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:01:20 1189131 [Warning] Aborted connection 1189131 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:01:57 1208963 [Warning] Aborted connection 1208963 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:03:11 1231365 [Warning] Aborted connection 1231365 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:03:24 1231419 [Warning] Aborted connection 1231419 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:03:25 1231444 [Warning] Aborted connection 1231444 to db: 'unconnected' user: 'idtgroup' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:03:46 1231457 [Warning] Aborted connection 1231457 to db: 'unconnected' user: 'idtgroup' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:04:07 1231468 [Warning] Aborted connection 1231468 to db: 'unconnected' user: 'idtgroup' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:04:27 1231474 [Warning] Aborted connection 1231474 to db: 'unconnected' user: 'idtgroup' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:04:39 1231488 [Warning] Aborted connection 1231488 to db: 'unconnected' user: 'idtgroup' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:05:48 1194899 [Warning] Aborted connection 1194899 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 6:15:32 1230254 [Warning] Aborted connection 1230254 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:15:37 1227904 [Warning] Aborted connection 1227904 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:17:31 1231788 [Warning] Aborted connection 1231788 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:17:39 1231840 [Warning] Aborted connection 1231840 to db: 'unconnected' user: 'wpms' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:18:01 1231843 [Warning] Aborted connection 1231843 to db: 'unconnected' user: 'wpms' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:34:35 1210064 [Warning] Aborted connection 1210064 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:36:44 1232311 [Warning] Aborted connection 1232311 to db: 'ulrich' user: 'ulrich' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:38:49 1206335 [Warning] Aborted connection 1206335 to db: 'gots' user: 'gotsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:51:11 1232713 [Warning] Aborted connection 1232713 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:51:34 1232750 [Warning] Aborted connection 1232750 to db: 'unconnected' user: 'wpcomm305' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:51:56 1230113 [Warning] Aborted connection 1230113 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:51:58 1232760 [Warning] Aborted connection 1232760 to db: 'unconnected' user: 'wpcomm305' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:52:22 1232773 [Warning] Aborted connection 1232773 to db: 'unconnected' user: 'wpcomm305' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 6:52:37 1232789 [Warning] Aborted connection 1232789 to db: 'unconnected' user: 'wpcomm305' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 7:00:27 1211511 [Warning] Aborted connection 1211511 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 7:21:56 1228427 [Warning] Aborted connection 1228427 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 7:46:56 1231785 [Warning] Aborted connection 1231785 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 7:55:03 1213273 [Warning] Aborted connection 1213273 to db: 'suma' user: 'sumadbuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 7:55:18 1213279 [Warning] Aborted connection 1213279 to db: 'suma' user: 'sumadbuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 7:59:32 1233848 [Warning] Aborted connection 1233848 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:22:01 1234598 [Warning] Aborted connection 1234598 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:22:11 1235017 [Warning] Aborted connection 1235017 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:22:39 1232772 [Warning] Aborted connection 1232772 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:22:57 1231781 [Warning] Aborted connection 1231781 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:23:28 1235712 [Warning] Aborted connection 1235712 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:23:44 1235697 [Warning] Aborted connection 1235697 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:24:06 1235764 [Warning] Aborted connection 1235764 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:24:26 1235770 [Warning] Aborted connection 1235770 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:24:41 1231407 [Warning] Aborted connection 1231407 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:24:56 1235694 [Warning] Aborted connection 1235694 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:25:01 1235729 [Warning] Aborted connection 1235729 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:25:10 1235782 [Warning] Aborted connection 1235782 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:25:14 1235795 [Warning] Aborted connection 1235795 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:25:20 1232295 [Warning] Aborted connection 1232295 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:25:23 1235813 [Warning] Aborted connection 1235813 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:25:56 1235815 [Warning] Aborted connection 1235815 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:25:57 1235819 [Warning] Aborted connection 1235819 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:26:17 1235831 [Warning] Aborted connection 1235831 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:26:25 1235817 [Warning] Aborted connection 1235817 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:26:41 1235854 [Warning] Aborted connection 1235854 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:26:54 1235870 [Warning] Aborted connection 1235870 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:27:07 1235884 [Warning] Aborted connection 1235884 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:27:40 1235864 [Warning] Aborted connection 1235864 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:27:43 1235855 [Warning] Aborted connection 1235855 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:28:18 1235888 [Warning] Aborted connection 1235888 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:28:26 1235931 [Warning] Aborted connection 1235931 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:29:00 1235929 [Warning] Aborted connection 1235929 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:29:04 1235958 [Warning] Aborted connection 1235958 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:29:21 1235952 [Warning] Aborted connection 1235952 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:29:32 1235983 [Warning] Aborted connection 1235983 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:29:33 1235977 [Warning] Aborted connection 1235977 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:29:38 1235826 [Warning] Aborted connection 1235826 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:29:42 1235991 [Warning] Aborted connection 1235991 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:29:50 1235995 [Warning] Aborted connection 1235995 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:30:10 1236002 [Warning] Aborted connection 1236002 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:30:15 1235809 [Warning] Aborted connection 1235809 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:30:25 1235907 [Warning] Aborted connection 1235907 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:30:28 1235808 [Warning] Aborted connection 1235808 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:30:29 1236012 [Warning] Aborted connection 1236012 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:30:31 1235830 [Warning] Aborted connection 1235830 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:30:32 1235996 [Warning] Aborted connection 1235996 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:30:43 1236000 [Warning] Aborted connection 1236000 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:30:47 1236032 [Warning] Aborted connection 1236032 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:30:50 1236031 [Warning] Aborted connection 1236031 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:30:52 1236024 [Warning] Aborted connection 1236024 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:30:53 1236034 [Warning] Aborted connection 1236034 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:30:57 1236026 [Warning] Aborted connection 1236026 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:02 1235998 [Warning] Aborted connection 1235998 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:02 1232296 [Warning] Aborted connection 1232296 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:06 1236037 [Warning] Aborted connection 1236037 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:09 1236025 [Warning] Aborted connection 1236025 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:10 1236047 [Warning] Aborted connection 1236047 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:13 1235829 [Warning] Aborted connection 1235829 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:15 1236044 [Warning] Aborted connection 1236044 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:20 1236014 [Warning] Aborted connection 1236014 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:26 1236058 [Warning] Aborted connection 1236058 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:28 1236053 [Warning] Aborted connection 1236053 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:29 1236059 [Warning] Aborted connection 1236059 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:29 1236061 [Warning] Aborted connection 1236061 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:36 1236041 [Warning] Aborted connection 1236041 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:43 1236073 [Warning] Aborted connection 1236073 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:43 1236062 [Warning] Aborted connection 1236062 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:45 1236030 [Warning] Aborted connection 1236030 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:46 1236043 [Warning] Aborted connection 1236043 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:50 1236077 [Warning] Aborted connection 1236077 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:31:57 1236075 [Warning] Aborted connection 1236075 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:02 1236083 [Warning] Aborted connection 1236083 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:06 1236082 [Warning] Aborted connection 1236082 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:07 1236087 [Warning] Aborted connection 1236087 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:08 1236071 [Warning] Aborted connection 1236071 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:10 1236051 [Warning] Aborted connection 1236051 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:11 1236050 [Warning] Aborted connection 1236050 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:11 1236093 [Warning] Aborted connection 1236093 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:14 1236089 [Warning] Aborted connection 1236089 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:23 1236110 [Warning] Aborted connection 1236110 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:26 1236067 [Warning] Aborted connection 1236067 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:30 1236109 [Warning] Aborted connection 1236109 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:35 1236119 [Warning] Aborted connection 1236119 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:35 1236105 [Warning] Aborted connection 1236105 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:36 1236111 [Warning] Aborted connection 1236111 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:41 1236127 [Warning] Aborted connection 1236127 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:47 1236118 [Warning] Aborted connection 1236118 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:50 1236134 [Warning] Aborted connection 1236134 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:57 1236138 [Warning] Aborted connection 1236138 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:32:58 1236133 [Warning] Aborted connection 1236133 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:02 1236142 [Warning] Aborted connection 1236142 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:03 1236085 [Warning] Aborted connection 1236085 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:04 1236135 [Warning] Aborted connection 1236135 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:07 1236040 [Warning] Aborted connection 1236040 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:15 1236143 [Warning] Aborted connection 1236143 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:15 1236146 [Warning] Aborted connection 1236146 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:17 1236074 [Warning] Aborted connection 1236074 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:24 1236153 [Warning] Aborted connection 1236153 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:28 1236147 [Warning] Aborted connection 1236147 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:31 1236162 [Warning] Aborted connection 1236162 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:31 1236160 [Warning] Aborted connection 1236160 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:34 1236116 [Warning] Aborted connection 1236116 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:35 1236054 [Warning] Aborted connection 1236054 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:36 1236129 [Warning] Aborted connection 1236129 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:39 1236167 [Warning] Aborted connection 1236167 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:46 1236163 [Warning] Aborted connection 1236163 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:50 1236182 [Warning] Aborted connection 1236182 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:52 1236180 [Warning] Aborted connection 1236180 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:33:55 1236179 [Warning] Aborted connection 1236179 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:02 1236173 [Warning] Aborted connection 1236173 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:05 1236196 [Warning] Aborted connection 1236196 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:06 1236114 [Warning] Aborted connection 1236114 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:08 1236199 [Warning] Aborted connection 1236199 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:09 1236155 [Warning] Aborted connection 1236155 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:16 1236154 [Warning] Aborted connection 1236154 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:16 1236202 [Warning] Aborted connection 1236202 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:19 1236203 [Warning] Aborted connection 1236203 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:27 1236213 [Warning] Aborted connection 1236213 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:28 1236214 [Warning] Aborted connection 1236214 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:32 1236207 [Warning] Aborted connection 1236207 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:40 1236210 [Warning] Aborted connection 1236210 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:41 1236220 [Warning] Aborted connection 1236220 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:41 1236186 [Warning] Aborted connection 1236186 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:46 1236187 [Warning] Aborted connection 1236187 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:54 1236244 [Warning] Aborted connection 1236244 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:56 1236222 [Warning] Aborted connection 1236222 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:57 1236238 [Warning] Aborted connection 1236238 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:34:58 1236192 [Warning] Aborted connection 1236192 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:00 1236230 [Warning] Aborted connection 1236230 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:06 1236183 [Warning] Aborted connection 1236183 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:08 1236232 [Warning] Aborted connection 1236232 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:12 1236245 [Warning] Aborted connection 1236245 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:13 1236260 [Warning] Aborted connection 1236260 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:19 1236257 [Warning] Aborted connection 1236257 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:21 1236211 [Warning] Aborted connection 1236211 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:21 1236267 [Warning] Aborted connection 1236267 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:25 1236269 [Warning] Aborted connection 1236269 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:35 1236249 [Warning] Aborted connection 1236249 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:38 1236270 [Warning] Aborted connection 1236270 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:40 1236274 [Warning] Aborted connection 1236274 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:45 1236278 [Warning] Aborted connection 1236278 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:48 1236258 [Warning] Aborted connection 1236258 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:52 1236284 [Warning] Aborted connection 1236284 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:52 1236289 [Warning] Aborted connection 1236289 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:53 1236262 [Warning] Aborted connection 1236262 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:35:56 1236290 [Warning] Aborted connection 1236290 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:01 1236292 [Warning] Aborted connection 1236292 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:06 1236296 [Warning] Aborted connection 1236296 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:08 1236303 [Warning] Aborted connection 1236303 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:08 1236304 [Warning] Aborted connection 1236304 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:13 1236306 [Warning] Aborted connection 1236306 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:14 1236305 [Warning] Aborted connection 1236305 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:20 1236310 [Warning] Aborted connection 1236310 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:23 1236219 [Warning] Aborted connection 1236219 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:24 1236184 [Warning] Aborted connection 1236184 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:25 1236280 [Warning] Aborted connection 1236280 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:28 1236273 [Warning] Aborted connection 1236273 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:30 1236320 [Warning] Aborted connection 1236320 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:30 1236321 [Warning] Aborted connection 1236321 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:31 1236317 [Warning] Aborted connection 1236317 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:33 1236272 [Warning] Aborted connection 1236272 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:34 1236261 [Warning] Aborted connection 1236261 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:43 1236318 [Warning] Aborted connection 1236318 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:48 1236340 [Warning] Aborted connection 1236340 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:52 1236341 [Warning] Aborted connection 1236341 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:36:53 1236315 [Warning] Aborted connection 1236315 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:00 1236023 [Warning] Aborted connection 1236023 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:01 1236348 [Warning] Aborted connection 1236348 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:03 1236347 [Warning] Aborted connection 1236347 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:04 1236359 [Warning] Aborted connection 1236359 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:07 1236131 [Warning] Aborted connection 1236131 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:20 1236371 [Warning] Aborted connection 1236371 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:26 1236379 [Warning] Aborted connection 1236379 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:27 1236380 [Warning] Aborted connection 1236380 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:28 1236382 [Warning] Aborted connection 1236382 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:29 1236227 [Warning] Aborted connection 1236227 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:31 1236333 [Warning] Aborted connection 1236333 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:32 1236325 [Warning] Aborted connection 1236325 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:33 1236265 [Warning] Aborted connection 1236265 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:37 1236369 [Warning] Aborted connection 1236369 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:40 1236367 [Warning] Aborted connection 1236367 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:41 1236372 [Warning] Aborted connection 1236372 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:42 1236361 [Warning] Aborted connection 1236361 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:43 1236391 [Warning] Aborted connection 1236391 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:52 1236385 [Warning] Aborted connection 1236385 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:37:53 1236395 [Warning] Aborted connection 1236395 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:00 1236332 [Warning] Aborted connection 1236332 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:03 1236401 [Warning] Aborted connection 1236401 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:03 1236389 [Warning] Aborted connection 1236389 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:08 1236398 [Warning] Aborted connection 1236398 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:09 1236404 [Warning] Aborted connection 1236404 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:13 1236412 [Warning] Aborted connection 1236412 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:14 1236365 [Warning] Aborted connection 1236365 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:17 1236337 [Warning] Aborted connection 1236337 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:17 1236400 [Warning] Aborted connection 1236400 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:18 1236418 [Warning] Aborted connection 1236418 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:21 1236330 [Warning] Aborted connection 1236330 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:21 1236410 [Warning] Aborted connection 1236410 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:25 1236417 [Warning] Aborted connection 1236417 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:26 1236432 [Warning] Aborted connection 1236432 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:32 1236426 [Warning] Aborted connection 1236426 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:36 1236440 [Warning] Aborted connection 1236440 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:38 1236435 [Warning] Aborted connection 1236435 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:42 1236429 [Warning] Aborted connection 1236429 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:45 1236442 [Warning] Aborted connection 1236442 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:51 1236433 [Warning] Aborted connection 1236433 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:52 1236446 [Warning] Aborted connection 1236446 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:38:53 1236428 [Warning] Aborted connection 1236428 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:02 1236456 [Warning] Aborted connection 1236456 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:04 1236394 [Warning] Aborted connection 1236394 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:07 1236461 [Warning] Aborted connection 1236461 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:07 1236397 [Warning] Aborted connection 1236397 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:08 1236393 [Warning] Aborted connection 1236393 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:10 1236458 [Warning] Aborted connection 1236458 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:10 1236455 [Warning] Aborted connection 1236455 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:18 1236430 [Warning] Aborted connection 1236430 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:18 1236451 [Warning] Aborted connection 1236451 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:21 1236467 [Warning] Aborted connection 1236467 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:26 1236465 [Warning] Aborted connection 1236465 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:28 1236472 [Warning] Aborted connection 1236472 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:37 1236471 [Warning] Aborted connection 1236471 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:40 1236460 [Warning] Aborted connection 1236460 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:41 1236479 [Warning] Aborted connection 1236479 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:46 1236477 [Warning] Aborted connection 1236477 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:48 1236424 [Warning] Aborted connection 1236424 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:51 1236406 [Warning] Aborted connection 1236406 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:39:51 1236434 [Warning] Aborted connection 1236434 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:00 1236487 [Warning] Aborted connection 1236487 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:03 1236490 [Warning] Aborted connection 1236490 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:10 1236476 [Warning] Aborted connection 1236476 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:12 1236469 [Warning] Aborted connection 1236469 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:12 1236486 [Warning] Aborted connection 1236486 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:14 1236501 [Warning] Aborted connection 1236501 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:16 1236495 [Warning] Aborted connection 1236495 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:19 1236466 [Warning] Aborted connection 1236466 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:28 1236513 [Warning] Aborted connection 1236513 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:28 1236514 [Warning] Aborted connection 1236514 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:38 1236504 [Warning] Aborted connection 1236504 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:42 1236520 [Warning] Aborted connection 1236520 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:43 1236511 [Warning] Aborted connection 1236511 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:48 1236475 [Warning] Aborted connection 1236475 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:50 1236521 [Warning] Aborted connection 1236521 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:40:59 1236524 [Warning] Aborted connection 1236524 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:02 1236527 [Warning] Aborted connection 1236527 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:09 1236493 [Warning] Aborted connection 1236493 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:12 1236541 [Warning] Aborted connection 1236541 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:14 1236531 [Warning] Aborted connection 1236531 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:19 1236542 [Warning] Aborted connection 1236542 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:20 1236526 [Warning] Aborted connection 1236526 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:25 1236549 [Warning] Aborted connection 1236549 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:28 1236548 [Warning] Aborted connection 1236548 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:29 1236496 [Warning] Aborted connection 1236496 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:36 1236558 [Warning] Aborted connection 1236558 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:38 1236551 [Warning] Aborted connection 1236551 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:42 1236561 [Warning] Aborted connection 1236561 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:44 1236563 [Warning] Aborted connection 1236563 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:46 1236509 [Warning] Aborted connection 1236509 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:51 1236530 [Warning] Aborted connection 1236530 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:56 1236567 [Warning] Aborted connection 1236567 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:56 1236574 [Warning] Aborted connection 1236574 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:41:58 1236569 [Warning] Aborted connection 1236569 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:01 1236576 [Warning] Aborted connection 1236576 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:01 1236508 [Warning] Aborted connection 1236508 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:03 1236516 [Warning] Aborted connection 1236516 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:09 1236589 [Warning] Aborted connection 1236589 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:16 1236556 [Warning] Aborted connection 1236556 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:18 1236488 [Warning] Aborted connection 1236488 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:19 1236559 [Warning] Aborted connection 1236559 to db: 'kfbf' user: 'kfbf' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:20 1236607 [Warning] Aborted connection 1236607 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:25 1236595 [Warning] Aborted connection 1236595 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:29 1236596 [Warning] Aborted connection 1236596 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:36 1236619 [Warning] Aborted connection 1236619 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:39 1236621 [Warning] Aborted connection 1236621 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:43 1236611 [Warning] Aborted connection 1236611 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:47 1236613 [Warning] Aborted connection 1236613 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:54 1236625 [Warning] Aborted connection 1236625 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:42:58 1236599 [Warning] Aborted connection 1236599 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:01 1236631 [Warning] Aborted connection 1236631 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:04 1236628 [Warning] Aborted connection 1236628 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:04 1236630 [Warning] Aborted connection 1236630 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:11 1236634 [Warning] Aborted connection 1236634 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:13 1236615 [Warning] Aborted connection 1236615 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:16 1236580 [Warning] Aborted connection 1236580 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:18 1236645 [Warning] Aborted connection 1236645 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:19 1236577 [Warning] Aborted connection 1236577 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:23 1236588 [Warning] Aborted connection 1236588 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:23 1236639 [Warning] Aborted connection 1236639 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:29 1236652 [Warning] Aborted connection 1236652 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:36 1236661 [Warning] Aborted connection 1236661 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:37 1236668 [Warning] Aborted connection 1236668 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:38 1236667 [Warning] Aborted connection 1236667 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:43 1236564 [Warning] Aborted connection 1236564 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:45 1236649 [Warning] Aborted connection 1236649 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:50 1236677 [Warning] Aborted connection 1236677 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:54 1236680 [Warning] Aborted connection 1236680 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:55 1236679 [Warning] Aborted connection 1236679 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:43:58 1236470 [Warning] Aborted connection 1236470 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:07 1236644 [Warning] Aborted connection 1236644 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:07 1236682 [Warning] Aborted connection 1236682 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:09 1236689 [Warning] Aborted connection 1236689 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:13 1236691 [Warning] Aborted connection 1236691 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:15 1236656 [Warning] Aborted connection 1236656 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:19 1236692 [Warning] Aborted connection 1236692 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:20 1236481 [Warning] Aborted connection 1236481 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:24 1236699 [Warning] Aborted connection 1236699 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:26 1236686 [Warning] Aborted connection 1236686 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:30 1236703 [Warning] Aborted connection 1236703 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:40 1236717 [Warning] Aborted connection 1236717 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:41 1236716 [Warning] Aborted connection 1236716 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:41 1236713 [Warning] Aborted connection 1236713 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:50 1236637 [Warning] Aborted connection 1236637 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:53 1236728 [Warning] Aborted connection 1236728 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:55 1236594 [Warning] Aborted connection 1236594 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:44:55 1236733 [Warning] Aborted connection 1236733 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:03 1236734 [Warning] Aborted connection 1236734 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:04 1236735 [Warning] Aborted connection 1236735 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:17 1236742 [Warning] Aborted connection 1236742 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:21 1236746 [Warning] Aborted connection 1236746 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:23 1236741 [Warning] Aborted connection 1236741 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:27 1236696 [Warning] Aborted connection 1236696 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:34 1236749 [Warning] Aborted connection 1236749 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:35 1236721 [Warning] Aborted connection 1236721 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:36 1236662 [Warning] Aborted connection 1236662 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:39 1236753 [Warning] Aborted connection 1236753 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:46 1236740 [Warning] Aborted connection 1236740 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:53 1236759 [Warning] Aborted connection 1236759 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:54 1236415 [Warning] Aborted connection 1236415 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:58 1236767 [Warning] Aborted connection 1236767 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:45:59 1236751 [Warning] Aborted connection 1236751 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:05 1236745 [Warning] Aborted connection 1236745 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:05 1236779 [Warning] Aborted connection 1236779 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:13 1236771 [Warning] Aborted connection 1236771 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:21 1236786 [Warning] Aborted connection 1236786 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:24 1236790 [Warning] Aborted connection 1236790 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:25 1236789 [Warning] Aborted connection 1236789 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:25 1236761 [Warning] Aborted connection 1236761 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:29 1236794 [Warning] Aborted connection 1236794 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:30 1236738 [Warning] Aborted connection 1236738 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:36 1236787 [Warning] Aborted connection 1236787 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:44 1236800 [Warning] Aborted connection 1236800 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:51 1236754 [Warning] Aborted connection 1236754 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:51 1236799 [Warning] Aborted connection 1236799 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:51 1236801 [Warning] Aborted connection 1236801 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:52 1236810 [Warning] Aborted connection 1236810 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:55 1236806 [Warning] Aborted connection 1236806 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:46:58 1236781 [Warning] Aborted connection 1236781 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:47:08 1236813 [Warning] Aborted connection 1236813 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:47:12 1236821 [Warning] Aborted connection 1236821 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:47:14 1236714 [Warning] Aborted connection 1236714 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:47:20 1236823 [Warning] Aborted connection 1236823 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:47:29 1236818 [Warning] Aborted connection 1236818 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:47:29 1236833 [Warning] Aborted connection 1236833 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:47:30 1236797 [Warning] Aborted connection 1236797 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:47:31 1236685 [Warning] Aborted connection 1236685 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:47:34 1236838 [Warning] Aborted connection 1236838 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:47:43 1236844 [Warning] Aborted connection 1236844 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:47:52 1236848 [Warning] Aborted connection 1236848 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:47:53 1236843 [Warning] Aborted connection 1236843 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:47:56 1236845 [Warning] Aborted connection 1236845 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:00 1236819 [Warning] Aborted connection 1236819 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:10 1236850 [Warning] Aborted connection 1236850 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:13 1236863 [Warning] Aborted connection 1236863 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:15 1236860 [Warning] Aborted connection 1236860 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:17 1236858 [Warning] Aborted connection 1236858 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:27 1236872 [Warning] Aborted connection 1236872 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:28 1236825 [Warning] Aborted connection 1236825 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:28 1236868 [Warning] Aborted connection 1236868 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:31 1236846 [Warning] Aborted connection 1236846 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:33 1236874 [Warning] Aborted connection 1236874 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:33 1236807 [Warning] Aborted connection 1236807 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:37 1236834 [Warning] Aborted connection 1236834 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:39 1236836 [Warning] Aborted connection 1236836 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:44 1236885 [Warning] Aborted connection 1236885 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:45 1236887 [Warning] Aborted connection 1236887 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:49 1236865 [Warning] Aborted connection 1236865 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:53 1236890 [Warning] Aborted connection 1236890 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:48:57 1236895 [Warning] Aborted connection 1236895 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:01 1236897 [Warning] Aborted connection 1236897 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:01 1236894 [Warning] Aborted connection 1236894 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:08 1236899 [Warning] Aborted connection 1236899 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:12 1236762 [Warning] Aborted connection 1236762 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:12 1236882 [Warning] Aborted connection 1236882 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:14 1236903 [Warning] Aborted connection 1236903 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:15 1236869 [Warning] Aborted connection 1236869 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:18 1236900 [Warning] Aborted connection 1236900 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:27 1236886 [Warning] Aborted connection 1236886 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:29 1236909 [Warning] Aborted connection 1236909 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:30 1236912 [Warning] Aborted connection 1236912 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:31 1236904 [Warning] Aborted connection 1236904 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:33 1236906 [Warning] Aborted connection 1236906 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:33 1236913 [Warning] Aborted connection 1236913 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:33 1236911 [Warning] Aborted connection 1236911 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:44 1236924 [Warning] Aborted connection 1236924 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:45 1236926 [Warning] Aborted connection 1236926 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:51 1236921 [Warning] Aborted connection 1236921 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:52 1236892 [Warning] Aborted connection 1236892 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:52 1236923 [Warning] Aborted connection 1236923 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:49:58 1236920 [Warning] Aborted connection 1236920 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:00 1236918 [Warning] Aborted connection 1236918 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:04 1236930 [Warning] Aborted connection 1236930 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:09 1236934 [Warning] Aborted connection 1236934 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:10 1236889 [Warning] Aborted connection 1236889 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:14 1236932 [Warning] Aborted connection 1236932 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:17 1236720 [Warning] Aborted connection 1236720 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:18 1236929 [Warning] Aborted connection 1236929 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:24 1236943 [Warning] Aborted connection 1236943 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:26 1236938 [Warning] Aborted connection 1236938 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:30 1236946 [Warning] Aborted connection 1236946 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:31 1236908 [Warning] Aborted connection 1236908 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:35 1236949 [Warning] Aborted connection 1236949 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:37 1236952 [Warning] Aborted connection 1236952 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:40 1236916 [Warning] Aborted connection 1236916 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:45 1236959 [Warning] Aborted connection 1236959 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:46 1236956 [Warning] Aborted connection 1236956 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:47 1236957 [Warning] Aborted connection 1236957 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:54 1236937 [Warning] Aborted connection 1236937 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:50:59 1236968 [Warning] Aborted connection 1236968 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:02 1236953 [Warning] Aborted connection 1236953 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:02 1236933 [Warning] Aborted connection 1236933 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:10 1236980 [Warning] Aborted connection 1236980 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:13 1236700 [Warning] Aborted connection 1236700 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:14 1236983 [Warning] Aborted connection 1236983 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:18 1236981 [Warning] Aborted connection 1236981 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:18 1236987 [Warning] Aborted connection 1236987 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:30 1236970 [Warning] Aborted connection 1236970 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:32 1237005 [Warning] Aborted connection 1237005 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:34 1236947 [Warning] Aborted connection 1236947 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:34 1237013 [Warning] Aborted connection 1237013 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:35 1237006 [Warning] Aborted connection 1237006 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:38 1236961 [Warning] Aborted connection 1236961 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:41 1236992 [Warning] Aborted connection 1236992 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:44 1237015 [Warning] Aborted connection 1237015 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:49 1237026 [Warning] Aborted connection 1237026 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:51:51 1237020 [Warning] Aborted connection 1237020 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:01 1237021 [Warning] Aborted connection 1237021 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:05 1237032 [Warning] Aborted connection 1237032 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:06 1237036 [Warning] Aborted connection 1237036 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:09 1236994 [Warning] Aborted connection 1236994 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:12 1237024 [Warning] Aborted connection 1237024 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:23 1237037 [Warning] Aborted connection 1237037 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:25 1236883 [Warning] Aborted connection 1236883 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:31 1237008 [Warning] Aborted connection 1237008 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:34 1237034 [Warning] Aborted connection 1237034 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:34 1237051 [Warning] Aborted connection 1237051 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:38 1237055 [Warning] Aborted connection 1237055 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:38 1236975 [Warning] Aborted connection 1236975 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:41 1237042 [Warning] Aborted connection 1237042 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:46 1237062 [Warning] Aborted connection 1237062 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:49 1237052 [Warning] Aborted connection 1237052 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:50 1237070 [Warning] Aborted connection 1237070 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:52:59 1237074 [Warning] Aborted connection 1237074 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:53:04 1237023 [Warning] Aborted connection 1237023 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:53:05 1237080 [Warning] Aborted connection 1237080 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:53:06 1237076 [Warning] Aborted connection 1237076 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:53:07 1237081 [Warning] Aborted connection 1237081 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:53:17 1237069 [Warning] Aborted connection 1237069 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:53:20 1237096 [Warning] Aborted connection 1237096 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:53:22 1237088 [Warning] Aborted connection 1237088 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:53:24 1237028 [Warning] Aborted connection 1237028 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:53:30 1237093 [Warning] Aborted connection 1237093 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:53:34 1237065 [Warning] Aborted connection 1237065 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:53:35 1237105 [Warning] Aborted connection 1237105 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:53:39 1237109 [Warning] Aborted connection 1237109 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:53:48 1236817 [Warning] Aborted connection 1236817 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:58:34 1237107 [Warning] Aborted connection 1237107 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 8:58:34 1237056 [Warning] Aborted connection 1237056 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 9:01:53 1211510 [Warning] Aborted connection 1211510 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 9:09:57 1237095 [Warning] Aborted connection 1237095 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 9:15:48 1237286 [Warning] Aborted connection 1237286 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 9:19:41 1237454 [Warning] Aborted connection 1237454 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 9:39:32 1238345 [Warning] Aborted connection 1238345 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 9:41:32 1238458 [Warning] Aborted connection 1238458 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 9:45:33 1238546 [Warning] Aborted connection 1238546 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 9:45:54 1237287 [Warning] Aborted connection 1237287 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 9:49:33 1238662 [Warning] Aborted connection 1238662 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 10:11:56 1237071 [Warning] Aborted connection 1237071 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 10:18:48 1224559 [Warning] Aborted connection 1224559 to db: 'gots' user: 'gotsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 10:36:57 1237622 [Warning] Aborted connection 1237622 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 10:41:56 1237825 [Warning] Aborted connection 1237825 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 11:06:56 1239421 [Warning] Aborted connection 1239421 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 11:26:57 1241343 [Warning] Aborted connection 1241343 to db: 'openrecords' user: 'openrecords' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 11:32:29 1242098 [Warning] Aborted connection 1242098 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 11:32:30 1242104 [Warning] Aborted connection 1242104 to db: 'unconnected' user: 'wpcomm306' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 11:32:32 1242105 [Warning] Aborted connection 1242105 to db: 'unconnected' user: 'wpcomm306' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 11:33:56 1238677 [Warning] Aborted connection 1238677 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 11:37:37 1237072 [Warning] Aborted connection 1237072 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 11:52:30 1240242 [Warning] Aborted connection 1240242 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 12:19:25 1240398 [Warning] Aborted connection 1240398 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 12:33:48 1218344 [Warning] Aborted connection 1218344 to db: 'gots' user: 'gotsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 12:52:22 1241175 [Warning] Aborted connection 1241175 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 13:16:04 1245152 [Warning] Aborted connection 1245152 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 13:18:58 1245238 [Warning] Aborted connection 1245238 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 13:21:47 1242151 [Warning] Aborted connection 1242151 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 13:22:21 1245380 [Warning] Aborted connection 1245380 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 13:27:07 1245478 [Warning] Aborted connection 1245478 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 13:38:59 1242661 [Warning] Aborted connection 1242661 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 13:56:49 1243602 [Warning] Aborted connection 1243602 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 14:01:11 1246195 [Warning] Aborted connection 1246195 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 14:01:20 1231385 [Warning] Aborted connection 1231385 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 14:31:53 1247497 [Warning] Aborted connection 1247497 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 14:35:08 1244548 [Warning] Aborted connection 1244548 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 14:35:53 1247624 [Warning] Aborted connection 1247624 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 14:38:49 1232399 [Warning] Aborted connection 1232399 to db: 'gots' user: 'gotsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 14:39:53 1247792 [Warning] Aborted connection 1247792 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 14:42:53 1247889 [Warning] Aborted connection 1247889 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 15:10:34 1248429 [Warning] Aborted connection 1248429 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 15:22:51 1245447 [Warning] Aborted connection 1245447 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 15:36:56 1245970 [Warning] Aborted connection 1245970 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 16:07:23 1250873 [Warning] Aborted connection 1250873 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:31:07 1236055 [Warning] Aborted connection 1236055 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:33:41 1236191 [Warning] Aborted connection 1236191 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:33:41 1236190 [Warning] Aborted connection 1236190 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:35:29 1236246 [Warning] Aborted connection 1236246 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:36:45 1236343 [Warning] Aborted connection 1236343 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:37:28 1236189 [Warning] Aborted connection 1236189 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:37:28 1236392 [Warning] Aborted connection 1236392 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:39:23 1236453 [Warning] Aborted connection 1236453 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:51:58 1249430 [Warning] Aborted connection 1249430 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 16:53:02 1236996 [Warning] Aborted connection 1236996 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:53:45 1237067 [Warning] Aborted connection 1237067 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:53:46 1237089 [Warning] Aborted connection 1237089 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:53:48 1237114 [Warning] Aborted connection 1237114 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:53:48 1237123 [Warning] Aborted connection 1237123 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:53:48 1237120 [Warning] Aborted connection 1237120 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:53:49 1237125 [Warning] Aborted connection 1237125 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 16:53:49 1232398 [Warning] Aborted connection 1232398 to db: 'gots' user: 'gotsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 16:58:55 1247755 [Warning] Aborted connection 1247755 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 17:01:53 1237364 [Warning] Aborted connection 1237364 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 17:08:13 1237122 [Warning] Aborted connection 1237122 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 17:21:31 1257414 [Warning] Aborted connection 1257414 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Got timeout reading communication packets) 2020-01-31 17:54:51 1246539 [Warning] Aborted connection 1246539 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Got an error reading communication packets) 200131 18:00:15 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=501 max_threads=502 thread_count=83 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f785c48a548 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f7833b12cf0 thread_stack 0x49000 *** buffer overflow detected ***: /usr/sbin/mysqld terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7f78da19d567] /lib64/libc.so.6(+0x1166e2)[0x7f78da19b6e2] /lib64/libc.so.6(+0x1184c7)[0x7f78da19d4c7] /usr/sbin/mysqld(my_addr_resolve+0xda)[0x5579154d794a] /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x5579154c00c2] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x557914f563af] /lib64/libpthread.so.0(+0xf630)[0x7f78dbdea630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x557914e84151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x557914fc2545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x557914d78ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x557914d6ed4b] /usr/sbin/mysqld(+0x6ca792)[0x557914d79792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x557914da16da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557914da365d] /usr/sbin/mysqld(+0x65e3f3)[0x557914d0d3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x557914d0ccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x557914da0b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557914da365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x557914da3761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x557914da4377] /usr/sbin/mysqld(+0x58fd84)[0x557914c3ed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x557914d4bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x557914d63b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x557914d63cd2] /usr/sbin/mysqld(+0x6b55e7)[0x557914d645e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x557914d64677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x557914d5356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x557914d5507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x557914e30afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x557914e30bdd] /lib64/libpthread.so.0(+0x7ea5)[0x7f78dbde2ea5] /lib64/libc.so.6(clone+0x6d)[0x7f78da1838cd] ======= Memory map: ======== 5579146af000-557915afd000 r-xp 00000000 fd:00 2101454 /usr/sbin/mysqld 557915cfc000-557915e28000 r--p 0144d000 fd:00 2101454 /usr/sbin/mysqld 557915e28000-557915ee2000 rw-p 01579000 fd:00 2101454 /usr/sbin/mysqld 557915ee2000-557916791000 rw-p 00000000 00:00 0 5579177ef000-55791b72a000 rw-p 00000000 00:00 0 [heap] 7f782d93a000-7f782d93b000 ---p 00000000 00:00 0 7f782d93b000-7f782d985000 rw-p 00000000 00:00 0 7f782da66000-7f782da67000 ---p 00000000 00:00 0 7f782da67000-7f782dab1000 rw-p 00000000 00:00 0 7f782dcbe000-7f782dcbf000 ---p 00000000 00:00 0 7f782dcbf000-7f782dd09000 rw-p 00000000 00:00 0 7f782ddea000-7f782ddeb000 ---p 00000000 00:00 0 7f782ddeb000-7f782de35000 rw-p 00000000 00:00 0 7f782de80000-7f782de81000 ---p 00000000 00:00 0 7f782de81000-7f782decb000 rw-p 00000000 00:00 0 7f782decb000-7f782decc000 ---p 00000000 00:00 0 7f782decc000-7f782df16000 rw-p 00000000 00:00 0 7f782df61000-7f782df62000 ---p 00000000 00:00 0 7f782df62000-7f782dfac000 rw-p 00000000 00:00 0 7f782e0d8000-7f782e0d9000 ---p 00000000 00:00 0 7f782e0d9000-7f782e123000 rw-p 00000000 00:00 0 7f782e2e5000-7f782e2e6000 ---p 00000000 00:00 0 7f782e2e6000-7f782e330000 rw-p 00000000 00:00 0 7f782e37b000-7f782e37c000 ---p 00000000 00:00 0 7f782e37c000-7f782e3c6000 rw-p 00000000 00:00 0 7f782e45c000-7f782e45d000 ---p 00000000 00:00 0 7f782e45d000-7f782e4a7000 rw-p 00000000 00:00 0 7f782e7e0000-7f782e7e1000 ---p 00000000 00:00 0 7f782e7e1000-7f782e82b000 rw-p 00000000 00:00 0 7f782e876000-7f782e877000 ---p 00000000 00:00 0 7f782e877000-7f782e8c1000 rw-p 00000000 00:00 0 7f782ea38000-7f782ea39000 ---p 00000000 00:00 0 7f782ea39000-7f782ea83000 rw-p 00000000 00:00 0 7f782eace000-7f782eacf000 ---p 00000000 00:00 0 7f782eacf000-7f782eb19000 rw-p 00000000 00:00 0 7f782eb19000-7f782eb1a000 ---p 00000000 00:00 0 7f782eb1a000-7f782eb64000 rw-p 00000000 00:00 0 7f782ebaf000-7f782ebb0000 ---p 00000000 00:00 0 7f782ebb0000-7f782ebfa000 rw-p 00000000 00:00 0 7f782ec90000-7f782ec91000 ---p 00000000 00:00 0 7f782ec91000-7f782ecdb000 rw-p 00000000 00:00 0 7f782edbc000-7f782edbd000 ---p 00000000 00:00 0 7f782edbd000-7f782ee07000 rw-p 00000000 00:00 0 7f782eee8000-7f782eee9000 ---p 00000000 00:00 0 7f782eee9000-7f782ef33000 rw-p 00000000 00:00 0 7f782efc9000-7f782efca000 ---p 00000000 00:00 0 7f782efca000-7f782f014000 rw-p 00000000 00:00 0 7f782f1d6000-7f782f1d7000 ---p 00000000 00:00 0 7f782f1d7000-7f782f221000 rw-p 00000000 00:00 0 7f782f26c000-7f782f26d000 ---p 00000000 00:00 0 7f782f26d000-7f782f2b7000 rw-p 00000000 00:00 0 7f782f398000-7f782f399000 ---p 00000000 00:00 0 7f782f399000-7f782f3e3000 rw-p 00000000 00:00 0 7f782f3e3000-7f782f3e4000 ---p 00000000 00:00 0 7f782f3e4000-7f782f42e000 rw-p 00000000 00:00 0 7f782f479000-7f782f47a000 ---p 00000000 00:00 0 7f782f47a000-7f782f4c4000 rw-p 00000000 00:00 0 7f782f4c4000-7f782f4c5000 ---p 00000000 00:00 0 7f782f4c5000-7f782f50f000 rw-p 00000000 00:00 0 7f782f55a000-7f782f55b000 ---p 00000000 00:00 0 7f782f55b000-7f782f5a5000 rw-p 00000000 00:00 0 7f782f5f0000-7f782f5f1000 ---p 00000000 00:00 0 7f782f5f1000-7f782f63b000 rw-p 00000000 00:00 0 7f782f686000-7f782f687000 ---p 00000000 00:00 0 7f782f687000-7f782f6d1000 rw-p 00000000 00:00 0 7f782f6d1000-7f782f6d2000 ---p 00000000 00:00 0 7f782f6d2000-7f782f71c000 rw-p 00000000 00:00 0 7f782f767000-7f782f768000 ---p 00000000 00:00 0 7f782f768000-7f782f7b2000 rw-p 00000000 00:00 0 7f782f974000-7f782f975000 ---p 00000000 00:00 0 7f782f975000-7f782f9bf000 rw-p 00000000 00:00 0 7f782faa0000-7f782faa1000 ---p 00000000 00:00 0 7f782faa1000-7f782faeb000 rw-p 00000000 00:00 0 7f782faeb000-7f782faec000 ---p 00000000 00:00 0 7f782faec000-7f782fb36000 rw-p 00000000 00:00 0 7f782fc17000-7f782fc18000 ---p 00000000 00:00 0 7f782fc18000-7f782fc62000 rw-p 00000000 00:00 0 7f782fc62000-7f782fc63000 ---p 00000000 00:00 0 7f782fc63000-7f782fcad000 rw-p 00000000 00:00 0 7f782fe24000-7f782fe25000 ---p 00000000 00:00 0 7f782fe25000-7f782fe6f000 rw-p 00000000 00:00 0 7f782fe6f000-7f782fe70000 ---p 00000000 00:00 0 7f782fe70000-7f782feba000 rw-p 00000000 00:00 0 7f782ff9b000-7f782ff9c000 ---p 00000000 00:00 0 7f782ff9c000-7f782ffe6000 rw-p 00000000 00:00 0 7f7830031000-7f7830032000 ---p 00000000 00:00 0 7f7830032000-7f783007c000 rw-p 00000000 00:00 0 7f78300c7000-7f78300c8000 ---p 00000000 00:00 0 7f78300c8000-7f7830112000 rw-p 00000000 00:00 0 7f7830112000-7f7830113000 ---p 00000000 00:00 0 7f7830113000-7f783015d000 rw-p 00000000 00:00 0 7f783015d000-7f783015e000 ---p 00000000 00:00 0 7f783015e000-7f78301a8000 rw-p 00000000 00:00 0 7f78301f3000-7f78301f4000 ---p 00000000 00:00 0 7f78301f4000-7f783023e000 rw-p 00000000 00:00 0 7f783023e000-7f783023f000 ---p 00000000 00:00 0 7f783023f000-7f7830289000 rw-p 00000000 00:00 0 7f7830289000-7f783028a000 ---p 00000000 00:00 0 7f783028a000-7f78302d4000 rw-p 00000000 00:00 0 7f78303b5000-7f78303b6000 ---p 00000000 00:00 0 7f78303b6000-7f7830400000 rw-p 00000000 00:00 0 7f783044b000-7f783044c000 ---p 00000000 00:00 0 7f783044c000-7f7830496000 rw-p 00000000 00:00 0 7f78304e1000-7f78304e2000 ---p 00000000 00:00 0 7f78304e2000-7f783052c000 rw-p 00000000 00:00 0 7f78308fb000-7f78308fc000 ---p 00000000 00:00 0 7f78308fc000-7f7830946000 rw-p 00000000 00:00 0 7f7830946000-7f7830947000 ---p 00000000 00:00 0 7f7830947000-7f7830991000 rw-p 00000000 00:00 0 7f7830991000-7f7830992000 ---p 00000000 00:00 0 7f7830992000-7f78309dc000 rw-p 00000000 00:00 0 7f78309dc000-7f78309dd000 ---p 00000000 00:00 0 7f78309dd000-7f7830a27000 rw-p 00000000 00:00 0 7f7830be9000-7f7830bea000 ---p 00000000 00:00 0 7f7830bea000-7f7830c34000 rw-p 00000000 00:00 0 7f7830c34000-7f7830c35000 ---p 00000000 00:00 0 7f7830c35000-7f7830c7f000 rw-p 00000000 00:00 0 7f7830d60000-7f7830d61000 ---p 00000000 00:00 0 7f7830d61000-7f7830dab000 rw-p 00000000 00:00 0 7f7830dab000-7f7830dac000 ---p 00000000 00:00 0 7f7830dac000-7f7830df6000 rw-p 00000000 00:00 0 7f7830df6000-7f7830df7000 ---p 00000000 00:00 0 7f7830df7000-7f7830e41000 rw-p 00000000 00:00 0 7f7830e8c000-7f7830e8d000 ---p 00000000 00:00 0 7f7830e8d000-7f7830ed7000 rw-p 00000000 00:00 0 7f7830ed7000-7f7830ed8000 ---p 00000000 00:00 0 7f7830ed8000-7f7830f22000 rw-p 00000000 00:00 0 7f7830f22000-7f7830f23000 ---p 00000000 00:00 0 7f7830f23000-7f7830f6d000 rw-p 00000000 00:00 0 7f783104e000-7f783104f000 ---p 00000000 00:00 0 7f783104f000-7f7831099000 rw-p 00000000 00:00 0 7f7831099000-7f783109a000 ---p 00000000 00:00 0 7f783109a000-7f78310e4000 rw-p 00000000 00:00 0 7f783117a000-7f783117b000 ---p 00000000 00:00 0 7f783117b000-7f78311c5000 rw-p 00000000 00:00 0 7f7831210000-7f7831211000 ---p 00000000 00:00 0 7f7831211000-7f783125b000 rw-p 00000000 00:00 0 7f783125b000-7f783125c000 ---p 00000000 00:00 0 7f783125c000-7f78312a6000 rw-p 00000000 00:00 0 7f78312a6000-7f78312a7000 ---p 00000000 00:00 0 7f78312a7000-7f78312f1000 rw-p 00000000 00:00 0 7f7831387000-7f7831388000 ---p 00000000 00:00 0 7f7831388000-7f78313d2000 rw-p 00000000 00:00 0 7f78313d2000-7f78313d3000 ---p 00000000 00:00 0 7f78313d3000-7f783141d000 rw-p 00000000 00:00 0 7f783141d000-7f783141e000 ---p 00000000 00:00 0 7f783141e000-7f7831468000 rw-p 00000000 00:00 0 7f7831594000-7f7831595000 ---p 00000000 00:00 0 7f7831595000-7f78315df000 rw-p 00000000 00:00 0 7f78315df000-7f78315e0000 ---p 00000000 00:00 0 7f78315e0000-7f783162a000 rw-p 00000000 00:00 0 7f783170b000-7f783170c000 ---p 00000000 00:00 0 7f783170c000-7f7831756000 rw-p 00000000 00:00 0 7f78317ec000-7f78317ed000 ---p 00000000 00:00 0 7f78317ed000-7f7831837000 rw-p 00000000 00:00 0 7f7831918000-7f7831919000 ---p 00000000 00:00 0 7f7831919000-7f7831963000 rw-p 00000000 00:00 0 7f7831963000-7f7831964000 ---p 00000000 00:00 0 7f7831964000-7f78319ae000 rw-p 00000000 00:00 0 7f78319ae000-7f78319af000 ---p 00000000 00:00 0 7f78319af000-7f78319f9000 rw-p 00000000 00:00 0 7f7831a44000-7f7831a45000 ---p 00000000 00:00 0 7f7831a45000-7f7831a8f000 rw-p 00000000 00:00 0 7f7831a8f000-7f7831a90000 ---p 00000000 00:00 0 7f7831a90000-7f7831ada000 rw-p 00000000 00:00 0 7f7831b70000-7f7831b71000 ---p 00000000 00:00 0 7f7831b71000-7f7831bbb000 rw-p 00000000 00:00 0 7f7831e13000-7f7831e14000 ---p 00000000 00:00 0 7f7831e14000-7f7831e5e000 rw-p 00000000 00:00 0 7f7831e5e000-7f7831e5f000 ---p 00000000 00:00 0 7f7831e5f000-7f7831ea9000 rw-p 00000000 00:00 0 7f7831ef4000-7f7831ef5000 ---p 00000000 00:00 0 7f7831ef5000-7f7831f3f000 rw-p 00000000 00:00 0 7f7831f3f000-7f7831f40000 ---p 00000000 00:00 0 7f7831f40000-7f7831f8a000 rw-p 00000000 00:00 0 7f7832020000-7f7832021000 ---p 00000000 00:00 0 7f7832021000-7f783206b000 rw-p 00000000 00:00 0 7f783206b000-7f783206c000 ---p 00000000 00:00 0 7f783206c000-7f78320b6000 rw-p 00000000 00:00 0 7f78320b6000-7f78320b7000 ---p 00000000 00:00 0 7f78320b7000-7f7832101000 rw-p 00000000 00:00 0 7f783214c000-7f783214d000 ---p 00000000 00:00 0 7f783214d000-7f7832197000 rw-p 00000000 00:00 0 7f783222d000-7f783222e000 ---p 00000000 00:00 0 7f783222e000-7f7832278000 rw-p 00000000 00:00 0 7f7832278000-7f7832279000 ---p 00000000 00:00 0 7f7832279000-7f78322c3000 rw-p 00000000 00:00 0 7f78322c3000-7f78322c4000 ---p 00000000 00:00 0 7f78322c4000-7f783230e000 rw-p 00000000 00:00 0 7f7832485000-7f7832486000 ---p 00000000 00:00 0 7f7832486000-7f78324d0000 rw-p 00000000 00:00 0 7f7832566000-7f7832567000 ---p 00000000 00:00 0 7f7832567000-7f78325b1000 rw-p 00000000 00:00 0 7f78325fc000-7f78325fd000 ---p 00000000 00:00 0 7f78325fd000-7f7832647000 rw-p 00000000 00:00 0 7f78326dd000-7f78326de000 ---p 00000000 00:00 0 7f78326de000-7f7832728000 rw-p 00000000 00:00 0 7f78327be000-7f78327bf000 ---p 00000000 00:00 0 7f78327bf000-7f7832809000 rw-p 00000000 00:00 0 7f7832980000-7f7832981000 ---p 00000000 00:00 0 7f7832981000-7f78329cb000 rw-p 00000000 00:00 0 7f7832af7000-7f7832af8000 ---p 00000000 00:00 0 7f7832af8000-7f7832b42000 rw-p 00000000 00:00 0 7f7832b42000-7f7832b43000 ---p 00000000 00:00 0 7f7832b43000-7f7832b8d000 rw-p 00000000 00:00 0 7f7832c23000-7f7832c24000 ---p 00000000 00:00 0 7f7832c24000-7f7832c6e000 rw-p 00000000 00:00 0 7f7832d4f000-7f7832d50000 ---p 00000000 00:00 0 7f7832d50000-7f7832d9a000 rw-p 00000000 00:00 0 7f7832e7b000-7f7832e7c000 ---p 00000000 00:00 0 7f7832e7c000-7f7832ec6000 rw-p 00000000 00:00 0 7f783303d000-7f783303e000 ---p 00000000 00:00 0 7f783303e000-7f7833088000 rw-p 00000000 00:00 0 7f7833088000-7f7833089000 ---p 00000000 00:00 0 7f7833089000-7f78330d3000 rw-p 00000000 00:00 0 7f78331ff000-7f7833200000 ---p 00000000 00:00 0 7f7833200000-7f783324a000 rw-p 00000000 00:00 0 7f7833295000-7f7833296000 ---p 00000000 00:00 0 7f7833296000-7f78332e0000 rw-p 00000000 00:00 0 7f78334a2000-7f78334a3000 ---p 00000000 00:00 0 7f78334a3000-7f78334ed000 rw-p 00000000 00:00 0 7f7833538000-7f7833539000 ---p 00000000 00:00 0 7f7833539000-7f7833583000 rw-p 00000000 00:00 0 7f7833583000-7f7833584000 ---p 00000000 00:00 0 7f7833584000-7f78335ce000 rw-p 00000000 00:00 0 7f7833619000-7f783361a000 ---p 00000000 00:00 0 7f783361a000-7f7833664000 rw-p 00000000 00:00 0 7f7833664000-7f7833665000 ---p 00000000 00:00 0 7f7833665000-7f78336af000 rw-p 00000000 00:00 0 7f7833790000-7f7833791000 ---p 00000000 00:00 0 7f7833791000-7f78337db000 rw-p 00000000 00:00 0 7f7833826000-7f7833827000 ---p 00000000 00:00 0 7f7833827000-7f7833871000 rw-p 00000000 00:00 0 7f78338bc000-7f78338bd000 ---p 00000000 00:00 0 7f78338bd000-7f7833907000 rw-p 00000000 00:00 0 7f783399d000-7f783399e000 ---p 00000000 00:00 0 7f783399e000-7f78339e8000 rw-p 00000000 00:00 0 7f7833ac9000-7f7833aca000 ---p 00000000 00:00 0 7f7833aca000-7f7833b14000 rw-p 00000000 00:00 0 7f7833b14000-7f7833b15000 ---p 00000000 00:00 0 7f7833b15000-7f7833b5f000 rw-p 00000000 00:00 0 7f7833b5f000-7f7833b60000 ---p 00000000 00:00 0 7f7833b60000-7f7833baa000 rw-p 00000000 00:00 0 7f7833baa000-7f7833bab000 ---p 00000000 00:00 0 7f7833bab000-7f7833bf5000 rw-p 00000000 00:00 0 7f7833c40000-7f7833c41000 ---p 00000000 00:00 0 7f7833c41000-7f7833c8b000 rw-p 00000000 00:00 0 7f7833d21000-7f7833d22000 ---p 00000000 00:00 0 7f7833d22000-7f7833d6c000 rw-p 00000000 00:00 0 7f7833d6c000-7f7833d6d000 ---p 00000000 00:00 0 7f7833d6d000-7f7833db7000 rw-p 00000000 00:00 0 7f7833db7000-7f7833db8000 ---p 00000000 00:00 0 7f7833db8000-7f7833e02000 rw-p 00000000 00:00 0 7f7833ee3000-7f7833ee4000 ---p 00000000 00:00 0 7f7833ee4000-7f7833f2e000 rw-p 00000000 00:00 0 7f7833f2e000-7f7833f2f000 ---p 00000000 00:00 0 7f7833f2f000-7f7833f79000 rw-p 00000000 00:00 0 7f7833fc4000-7f7833fc5000 ---p 00000000 00:00 0 7f7833fc5000-7f783400f000 rw-p 00000000 00:00 0 7f78341d1000-7f78341d2000 ---p 00000000 00:00 0 7f78341d2000-7f783421c000 rw-p 00000000 00:00 0 7f783421c000-7f783421d000 ---p 00000000 00:00 0 7f783421d000-7f7834267000 rw-p 00000000 00:00 0 7f7834267000-7f7834268000 ---p 00000000 00:00 0 7f7834268000-7f78342b2000 rw-p 00000000 00:00 0 7f78342b2000-7f78342b3000 ---p 00000000 00:00 0 7f78342b3000-7f78342fd000 rw-p 00000000 00:00 0 7f78342fd000-7f78342fe000 ---p 00000000 00:00 0 7f78342fe000-7f7834348000 rw-p 00000000 00:00 0 7f7834348000-7f7834349000 ---p 00000000 00:00 0 7f7834349000-7f7834393000 rw-p 00000000 00:00 0 7f78344bf000-7f78344c0000 ---p 00000000 00:00 0 7f78344c0000-7f783450a000 rw-p 00000000 00:00 0 7f783450a000-7f783450b000 ---p 00000000 00:00 0 7f783450b000-7f7834555000 rw-p 00000000 00:00 0 7f7834555000-7f7834556000 ---p 00000000 00:00 0 7f7834556000-7f78345a0000 rw-p 00000000 00:00 0 7f78345a0000-7f78345a1000 ---p 00000000 00:00 0 7f78345a1000-7f78345eb000 rw-p 00000000 00:00 0 7f78345eb000-7f78345ec000 ---p 00000000 00:00 0 7f78345ec000-7f7834636000 rw-p 00000000 00:00 0 7f7834636000-7f7834637000 ---p 00000000 00:00 0 7f7834637000-7f7834681000 rw-p 00000000 00:00 0 7f7834681000-7f7834682000 ---p 00000000 00:00 0 7f7834682000-7f78346cc000 rw-p 00000000 00:00 0 7f7834762000-7f7834763000 ---p 00000000 00:00 0 7f7834763000-7f78347ad000 rw-p 00000000 00:00 0 7f78347ad000-7f78347ae000 ---p 00000000 00:00 0 7f78347ae000-7f78347f8000 rw-p 00000000 00:00 0 7f78347f8000-7f78347f9000 ---p 00000000 00:00 0 7f78347f9000-7f7834843000 rw-p 00000000 00:00 0 7f783488e000-7f783488f000 ---p 00000000 00:00 0 7f783488f000-7f78348d9000 rw-p 00000000 00:00 0 7f7834924000-7f7834925000 ---p 00000000 00:00 0 7f7834925000-7f783496f000 rw-p 00000000 00:00 0 7f783496f000-7f7834970000 ---p 00000000 00:00 0 7f7834970000-7f78349ba000 rw-p 00000000 00:00 0 7f78349ba000-7f78349bb000 ---p 00000000 00:00 0 7f78349bb000-7f7834a05000 rw-p 00000000 00:00 0 7f7834a05000-7f7834a06000 ---p 00000000 00:00 0 7f7834a06000-7f7834a50000 rw-p 00000000 00:00 0 7f7834ae6000-7f7834ae7000 ---p 00000000 00:00 0 7f7834ae7000-7f7834b31000 rw-p 00000000 00:00 0 7f7834c5d000-7f7834c5e000 ---p 00000000 00:00 0 7f7834c5e000-7f7834ca8000 rw-p 00000000 00:00 0 7f7834d3e000-7f7834d3f000 ---p 00000000 00:00 0 7f7834d3f000-7f7834d89000 rw-p 00000000 00:00 0 7f7834e6a000-7f7834e6b000 ---p 00000000 00:00 0 7f7834e6b000-7f7834eb5000 rw-p 00000000 00:00 0 7f7834f00000-7f7834f01000 ---p 00000000 00:00 0 7f7834f01000-7f7834f4b000 rw-p 00000000 00:00 0 7f7834f4b000-7f7834f4c000 ---p 00000000 00:00 0 7f7834f4c000-7f7834f96000 rw-p 00000000 00:00 0 7f7834f96000-7f7834f97000 ---p 00000000 00:00 0 7f7834f97000-7f7834fe1000 rw-p 00000000 00:00 0 7f7835077000-7f7835078000 ---p 00000000 00:00 0 7f7835078000-7f78350c2000 rw-p 00000000 00:00 0 7f7835158000-7f7835159000 ---p 00000000 00:00 0 7f7835159000-7f78351a3000 rw-p 00000000 00:00 0 7f783531a000-7f7835782000 r-xp 00000000 fd:00 2343798 /usr/lib64/mysql/plugin/ha_mroonga.so 7f7835782000-7f7835981000 ---p 00468000 fd:00 2343798 /usr/lib64/mysql/plugin/ha_mroonga.so 7f7835981000-7f7835b08000 r--p 00467000 fd:00 2343798 /usr/lib64/mysql/plugin/ha_mroonga.so 7f7835b08000-7f7835b5d000 rw-p 005ee000 fd:00 2343798 /usr/lib64/mysql/plugin/ha_mroonga.so 7f7835b5d000-7f7835b64000 rw-p 00000000 00:00 0 7f7835baf000-7f7835bb0000 ---p 00000000 00:00 0 7f7835bb0000-7f7835bfa000 rw-p 00000000 00:00 0 7f7835c45000-7f7835c46000 ---p 00000000 00:00 0 7f7835c46000-7f7835c90000 rw-p 00000000 00:00 0 7f7835cdb000-7f7835cdc000 ---p 00000000 00:00 0 7f7835cdc000-7f7835d26000 rw-p 00000000 00:00 0 7f7835d26000-7f7835d27000 ---p 00000000 00:00 0 7f7835d27000-7f7835d71000 rw-p 00000000 00:00 0 7f7835d71000-7f7835d72000 ---p 00000000 00:00 0 7f7835d72000-7f7835dbc000 rw-p 00000000 00:00 0 7f7835dbc000-7f7835dbd000 ---p 00000000 00:00 0 7f7835dbd000-7f7835e07000 rw-p 00000000 00:00 0 7f7835e52000-7f7835e53000 ---p 00000000 00:00 0 7f7835e53000-7f7835e9d000 rw-p 00000000 00:00 0 7f7835f33000-7f7835f34000 ---p 00000000 00:00 0 7f7835f34000-7f7835f7e000 rw-p 00000000 00:00 0 7f7835fc9000-7f7835fca000 ---p 00000000 00:00 0 7f7835fca000-7f7836014000 rw-p 00000000 00:00 0 7f7836014000-7f7836015000 ---p 00000000 00:00 0 7f7836015000-7f783605f000 rw-p 00000000 00:00 0 7f783605f000-7f7836060000 ---p 00000000 00:00 0 7f7836060000-7f78360aa000 rw-p 00000000 00:00 0 7f78360aa000-7f78360ab000 ---p 00000000 00:00 0 7f78360ab000-7f78360f5000 rw-p 00000000 00:00 0 7f78360f5000-7f78360f6000 ---p 00000000 00:00 0 7f78360f6000-7f7836140000 rw-p 00000000 00:00 0 7f7836140000-7f7836141000 ---p 00000000 00:00 0 7f7836141000-7f783618b000 rw-p 00000000 00:00 0 7f783618b000-7f783618c000 ---p 00000000 00:00 0 7f783618c000-7f78361d6000 rw-p 00000000 00:00 0 7f783626c000-7f783626d000 ---p 00000000 00:00 0 7f783626d000-7f78362b7000 rw-p 00000000 00:00 0 7f78363e3000-7f78363e4000 ---p 00000000 00:00 0 7f78363e4000-7f783642e000 rw-p 00000000 00:00 0 7f783642e000-7f783642f000 ---p 00000000 00:00 0 7f783642f000-7f7836479000 rw-p 00000000 00:00 0 7f78364c4000-7f78364c5000 ---p 00000000 00:00 0 7f78364c5000-7f783650f000 rw-p 00000000 00:00 0 7f783655a000-7f783655b000 ---p 00000000 00:00 0 7f783655b000-7f78365a5000 rw-p 00000000 00:00 0 7f78365a5000-7f78365a6000 ---p 00000000 00:00 0 7f78365a6000-7f78365f0000 rw-p 00000000 00:00 0 7f7836767000-7f7836768000 ---p 00000000 00:00 0 7f7836768000-7f78367b2000 rw-p 00000000 00:00 0 7f78367b2000-7f78367b3000 ---p 00000000 00:00 0 7f78367b3000-7f78367fd000 rw-p 00000000 00:00 0 7f78367fd000-7f78367fe000 ---p 00000000 00:00 0 7f78367fe000-7f7836ffe000 rw-p 00000000 00:00 0 7f7836ffe000-7f7836fff000 ---p 00000000 00:00 0 7f7836fff000-7f78377ff000 rw-p 00000000 00:00 0 7f78377ff000-7f7837800000 ---p 00000000 00:00 0 7f7837800000-7f7838000000 rw-p 00000000 00:00 0 7f7838000000-7f78397b4000 rw-p 00000000 00:00 0 7f78397b4000-7f783c000000 ---p 00000000 00:00 0 7f783c000000-7f783df42000 rw-p 00000000 00:00 0 7f783df42000-7f7840000000 ---p 00000000 00:00 0 7f7840000000-7f7842cc8000 rw-p 00000000 00:00 0 7f7842cc8000-7f7844000000 ---p 00000000 00:00 0 7f7844000000-7f7846394000 rw-p 00000000 00:00 0 7f7846394000-7f7848000000 ---p 00000000 00:00 0 7f7848000000-7f78498bc000 rw-p 00000000 00:00 0 7f78498bc000-7f784c000000 ---p 00000000 00:00 0 7f784c000000-7f784da8d000 rw-p 00000000 00:00 0 7f784da8d000-7f7850000000 ---p 00000000 00:00 0 7f7850000000-7f7851917000 rw-p 00000000 00:00 0 7f7851917000-7f7854000000 ---p 00000000 00:00 0 7f7854000000-7f7855ef7000 rw-p 00000000 00:00 0 7f7855ef7000-7f7858000000 ---p 00000000 00:00 0 7f78581d2000-7f78581d3000 ---p 00000000 00:00 0 7f78581d3000-7f785821d000 rw-p 00000000 00:00 0 7f7858268000-7f7858269000 ---p 00000000 00:00 0 7f7858269000-7f78582b3000 rw-p 00000000 00:00 0 7f78582b3000-7f78582b4000 ---p 00000000 00:00 0 7f78582b4000-7f78582fe000 rw-p 00000000 00:00 0 7f7858556000-7f7858557000 ---p 00000000 00:00 0 7f7858557000-7f78585a1000 rw-p 00000000 00:00 0 7f7858682000-7f7858683000 ---p 00000000 00:00 0 7f7858683000-7f78586cd000 rw-p 00000000 00:00 0 7f78587f9000-7f78587fa000 ---p 00000000 00:00 0 7f78587fa000-7f7858ffa000 rw-p 00000000 00:00 0 7f7858ffa000-7f7858ffb000 ---p 00000000 00:00 0 7f7858ffb000-7f78597fb000 rw-p 00000000 00:00 0 7f78597fb000-7f78597fc000 ---p 00000000 00:00 0 7f78597fc000-7f7859ffc000 rw-p 00000000 00:00 0 7f7859ffc000-7f7859ffd000 ---p 00000000 00:00 0 7f7859ffd000-7f785a7fd000 rw-p 00000000 00:00 0 7f785a7fd000-7f785a7fe000 ---p 00000000 00:00 0 7f785a7fe000-7f785affe000 rw-p 00000000 00:00 0 7f785affe000-7f785afff000 ---p 00000000 00:00 0 7f785afff000-7f785b7ff000 rw-p 00000000 00:00 0 7f785b7ff000-7f785b800000 ---p 00000000 00:00 0 7f785b800000-7f785c000000 rw-p 00000000 00:00 0 7f785c000000-7f785d8f5000 rw-p 00000000 00:00 0 7f785d8f5000-7f7860000000 ---p 00000000 00:00 0 7f7860000000-7f786222f000 rw-p 00000000 00:00 0 7f786222f000-7f7864000000 ---p 00000000 00:00 0 7f7864000000-7f7865ae6000 rw-p 00000000 00:00 0 7f7865ae6000-7f7868000000 ---p 00000000 00:00 0 7f7868000000-7f786a69b000 rw-p 00000000 00:00 0 7f786a69b000-7f786c000000 ---p 00000000 00:00 0 7f786c000000-7f786d8b3000 rw-p 00000000 00:00 0 7f786d8b3000-7f7870000000 ---p 00000000 00:00 0 7f7870003000-7f7870004000 ---p 00000000 00:00 0 7f7870004000-7f787004e000 rw-p 00000000 00:00 0 7f787004e000-7f787004f000 ---p 00000000 00:00 0 7f787004f000-7f7870099000 rw-p 00000000 00:00 0 7f7870099000-7f787009a000 ---p 00000000 00:00 0 7f787009a000-7f78700e4000 rw-p 00000000 00:00 0 7f78701c5000-7f78701c6000 ---p 00000000 00:00 0 7f78701c6000-7f7870210000 rw-p 00000000 00:00 0 7f787025b000-7f787025c000 ---p 00000000 00:00 0 7f787025c000-7f78702a6000 rw-p 00000000 00:00 0 7f78702a6000-7f78702a7000 ---p 00000000 00:00 0 7f78702a7000-7f78702f1000 rw-p 00000000 00:00 0 7f78702f1000-7f7870303000 r-xp 00000000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f7870303000-7f7870502000 ---p 00012000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f7870502000-7f7870505000 r--p 00011000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f7870505000-7f7870506000 rw-p 00014000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f7870506000-7f787050c000 r-xp 00000000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f787050c000-7f787070b000 ---p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f787070b000-7f787070c000 r--p 00005000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f787070c000-7f787070d000 rw-p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f787070d000-7f787070e000 ---p 00000000 00:00 0 7f787070e000-7f7870f0e000 rw-p 00000000 00:00 0 7f7870f0e000-7f7870f0f000 ---p 00000000 00:00 0 7f7870f0f000-7f787170f000 rw-p 00000000 00:00 0 7f787189e000-7f787189f000 ---p 00000000 00:00 0 7f787189f000-7f78718e9000 rw-p 00000000 00:00 0 7f7871934000-7f7871935000 ---p 00000000 00:00 0 7f7871935000-7f787197f000 rw-p 00000000 00:00 0 7f787197f000-7f7871980000 ---p 00000000 00:00 0 7f7871980000-7f78719ca000 rw-p 00000000 00:00 0 7f78719ca000-7f78719cb000 ---p 00000000 00:00 0 7f78719cb000-7f7871a15000 rw-p 00000000 00:00 0 7f7871a15000-7f7871a16000 ---p 00000000 00:00 0 7f7871a16000-7f7871a60000 rw-p 00000000 00:00 0 7f7871a60000-7f7871a61000 ---p 00000000 00:00 0 7f7871a61000-7f7871aab000 rw-p 00000000 00:00 0 7f7871aab000-7f7871aac000 ---p 00000000 00:00 0 7f7871aac000-7f7871af6000 rw-p 00000000 00:00 0 7f7871af6000-7f7871af7000 ---p 00000000 00:00 0 7f7871af7000-7f7871b41000 rw-p 00000000 00:00 0 7f7871b41000-7f7871b42000 ---p 00000000 00:00 0 7f7871b42000-7f7871b8c000 rw-p 00000000 00:00 0 7f7871b8c000-7f7871b8d000 ---p 00000000 00:00 0 7f7871b8d000-7f7871bd7000 rw-p 00000000 00:00 0 7f7871bd7000-7f7871bd8000 ---p 00000000 00:00 0 7f7871bd8000-7f7871c22000 rw-p 00000000 00:00 0 7f7871c22000-7f7871c23000 ---p 00000000 00:00 0 7f7871c23000-7f7871c6d000 rw-p 00000000 00:00 0 7f7871d03000-7f7871d04000 ---p 00000000 00:00 0 7f7871d04000-7f7871d4e000 rw-p 00000000 00:00 0 7f7871d4e000-7f7871d4f000 ---p 00000000 00:00 0 7f7871d4f000-7f7871d99000 rw-p 00000000 00:00 0 7f7871d99000-7f7871d9a000 ---p 00000000 00:00 0 7f7871d9a000-7f7871de4000 rw-p 00000000 00:00 0 7f7871de4000-7f7871de5000 ---p 00000000 00:00 0 7f7871de5000-7f7871e2f000 rw-p 00000000 00:00 0 7f7871e2f000-7f7871e30000 ---p 00000000 00:00 0 7f7871e30000-7f7871e7a000 rw-p 00000000 00:00 0 7f7871ec5000-7f7871ec6000 ---p 00000000 00:00 0 7f7871ec6000-7f7871f10000 rw-p 00000000 00:00 0 7f7871f10000-7f7871f11000 ---p 00000000 00:00 0 7f7871f11000-7f7872711000 rw-p 00000000 00:00 0 7f7872711000-7f7872712000 ---p 00000000 00:00 0 7f7872712000-7f7872f12000 rw-p 00000000 00:00 0 7f7872f12000-7f7872f13000 ---p 00000000 00:00 0 7f7872f13000-7f7873713000 rw-p 00000000 00:00 0 7f7873713000-7f7873714000 ---p 00000000 00:00 0 7f7873714000-7f7873f14000 rw-p 00000000 00:00 0 7f7873f14000-7f7873f15000 ---p 00000000 00:00 0 7f7873f15000-7f7874715000 rw-p 00000000 00:00 0 7f7874715000-7f7874716000 ---p 00000000 00:00 0 7f7874716000-7f7874f16000 rw-p 00000000 00:00 0 7f7874f16000-7f7874f17000 ---p 00000000 00:00 0 7f7874f17000-7f7875717000 rw-p 00000000 00:00 0 7f7875717000-7f7875718000 ---p 00000000 00:00 0 7f7875718000-7f7875f18000 rw-p 00000000 00:00 0 7f7875f18000-7f7875f19000 ---p 00000000 00:00 0 7f7875f19000-7f7876719000 rw-p 00000000 00:00 0 7f7876719000-7f787671a000 ---p 00000000 00:00 0 7f787671a000-7f7876f1a000 rw-p 00000000 00:00 0 7f7876f1a000-7f7876f1b000 ---p 00000000 00:00 0 7f7876f1b000-7f787771b000 rw-p 00000000 00:00 0 7f787771b000-7f787771c000 ---p 00000000 00:00 0 7f787771c000-7f7877f1c000 rw-p 00000000 00:00 0 7f7877f1c000-7f7877f1d000 ---p 00000000 00:00 0 7f7877f1d000-7f787871d000 rw-p 00000000 00:00 0 7f787871d000-7f787871e000 ---p 00000000 00:00 0 7f787871e000-7f7878f1e000 rw-p 00000000 00:00 0 7f7878f68000-7f7878f69000 ---p 00000000 00:00 0 7f7878f69000-7f7878fb3000 rw-p 00000000 00:00 0 7f7878ffe000-7f7878fff000 ---p 00000000 00:00 0 7f7878fff000-7f78797ff000 rw-p 00000000 00:00 0 7f78797ff000-7f7879800000 ---p 00000000 00:00 0 7f7879800000-7f787a000000 rw-p 00000000 00:00 0 7f787a000000-7f78bc000000 rw-p 00000000 00:00 0 7f78bc000000-7f78bdb5b000 rw-p 00000000 00:00 0 7f78bdb5b000-7f78c0000000 ---p 00000000 00:00 0 7f78c0026000-7f78c0032000 r-xp 00000000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f78c0032000-7f78c0231000 ---p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f78c0231000-7f78c0232000 r--p 0000b000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f78c0232000-7f78c0233000 rw-p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f78c0233000-7f78c0bd9000 rw-p 00000000 00:00 0 7f78c0cf8000-7f78c0cf9000 ---p 00000000 00:00 0 7f78c0cf9000-7f78c0d43000 rw-p 00000000 00:00 0 7f78c0d43000-7f78c0d44000 ---p 00000000 00:00 0 7f78c0d44000-7f78c0d8e000 rw-p 00000000 00:00 0 7f78c0d8e000-7f78c0d8f000 ---p 00000000 00:00 0 7f78c0d8f000-7f78d0000000 rw-p 00000000 00:00 0 7f78d0000000-7f78d2210000 rw-p 00000000 00:00 0 7f78d2210000-7f78d4000000 ---p 00000000 00:00 0 7f78d4029000-7f78d402a000 ---p 00000000 00:00 0 7f78d402a000-7f78d4692000 rw-p 00000000 00:00 0 7f78d4692000-7f78d4693000 ---p 00000000 00:00 0 7f78d4693000-7f78d6d22000 rw-p 00000000 00:00 0 7f78d6d22000-7f78d6d23000 ---p 00000000 00:00 0 7f78d6d23000-7f78d7523000 rw-p 00000000 00:00 0 7f78d7523000-7f78d7526000 r-xp 00000000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f78d7526000-7f78d7725000 ---p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f78d7725000-7f78d7726000 r--p 00002000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f78d7726000-7f78d7727000 rw-p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f78d7727000-7f78d7735000 r-xp 00000000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f78d7735000-7f78d7935000 ---p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f78d7935000-7f78d7936000 r--p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f78d7936000-7f78d7937000 rw-p 0000f000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f78d7937000-7f78d7946000 r-xp 00000000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f78d7946000-7f78d7b45000 ---p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f78d7b45000-7f78d7b46000 r--p 0000e000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f78d7b46000-7f78d7b47000 rw-p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f78d7b47000-7f78d7b5e000 r-xp 00000000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f78d7b5e000-7f78d7d5d000 ---p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f78d7d5d000-7f78d7d5e000 r--p 00016000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f78d7d5e000-7f78d7d5f000 rw-p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f78d7d5f000-7f78d7dbf000 r-xp 00000000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f78d7dbf000-7f78d7fbf000 ---p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f78d7fbf000-7f78d7fc0000 r--p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f78d7fc0000-7f78d7fc1000 rw-p 00061000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f78d7fc1000-7f78d7fc5000 r-xp 00000000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f78d7fc5000-7f78d81c4000 ---p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f78d81c4000-7f78d81c5000 r--p 00003000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f78d81c5000-7f78d81c6000 rw-p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f78d81c6000-7f78d81f7000 r-xp 00000000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f78d81f7000-7f78d83f6000 ---p 00031000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f78d83f6000-7f78d83f8000 r--p 00030000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f78d83f8000-7f78d83f9000 rw-p 00032000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f78d83f9000-7f78d83fc000 r-xp 00000000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f78d83fc000-7f78d85fb000 ---p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f78d85fb000-7f78d85fc000 r--p 00002000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f78d85fc000-7f78d85fd000 rw-p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f78d85fd000-7f78d86d6000 r-xp 00000000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f78d86d6000-7f78d88d5000 ---p 000d9000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f78d88d5000-7f78d88e3000 r--p 000d8000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f78d88e3000-7f78d88e6000 rw-p 000e6000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f78d88e6000-7f78d8930000 r-xp 00000000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f78d8930000-7f78d8b30000 ---p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f78d8b30000-7f78d8b31000 r--p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f78d8b31000-7f78d8b33000 rw-p 0004b000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f78d8b33000-7f78d8b48000 r-xp 00000000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f78d8b48000-7f78d8d47000 ---p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f78d8d47000-7f78d8d48000 r--p 00014000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f78d8d48000-7f78d8d49000 rw-p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f78d8d49000-7f78d8d97000 r-xp 00000000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f78d8d97000-7f78d8f97000 ---p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f78d8f97000-7f78d8f99000 r--p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f78d8f99000-7f78d8f9a000 rw-p 00050000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f78d8f9a000-7f78d8fb0000 r-xp 00000000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f78d8fb0000-7f78d91af000 ---p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f78d91af000-7f78d91b0000 r--p 00015000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f78d91b0000-7f78d91b1000 rw-p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f78d91b1000-7f78d91b3000 rw-p 00000000 00:00 0 7f78d91b3000-7f78d91b7000 r-xp 00000000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f78d91b7000-7f78d93b6000 ---p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f78d93b6000-7f78d93b7000 r--p 00003000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f78d93b7000-7f78d93b8000 rw-p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f78d93b8000-7f78d9435000 r-xp 00000000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f78d9435000-7f78d9634000 ---p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f78d9634000-7f78d9635000 r--p 0007c000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f78d9635000-7f78d9638000 rw-p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f78d9638000-7f78d9639000 rw-p 00000000 00:00 0 7f78d9639000-7f78d964d000 r-xp 00000000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f78d964d000-7f78d984c000 ---p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f78d984c000-7f78d984d000 r--p 00013000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f78d984d000-7f78d984e000 rw-p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f78d984e000-7f78d9872000 r-xp 00000000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f78d9872000-7f78d9a71000 ---p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f78d9a71000-7f78d9a72000 r--p 00023000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f78d9a72000-7f78d9a73000 rw-p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f78d9a73000-7f78d9a75000 rw-p 00000000 00:00 0 7f78d9a75000-7f78d9a7c000 r-xp 00000000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f78d9a7c000-7f78d9c7b000 ---p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f78d9c7b000-7f78d9c7c000 r--p 00006000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f78d9c7c000-7f78d9c7d000 rw-p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f78d9c7d000-7f78d9c81000 r-xp 00000000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f78d9c81000-7f78d9e80000 ---p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f78d9e80000-7f78d9e81000 r--p 00003000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f78d9e81000-7f78d9e82000 rw-p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f78d9e82000-7f78d9e84000 r-xp 00000000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f78d9e84000-7f78da083000 ---p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f78da083000-7f78da084000 r--p 00001000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f78da084000-7f78da085000 rw-p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f78da085000-7f78da248000 r-xp 00000000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f78da248000-7f78da448000 ---p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f78da448000-7f78da44c000 r--p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f78da44c000-7f78da44e000 rw-p 001c7000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f78da44e000-7f78da453000 rw-p 00000000 00:00 0 7f78da453000-7f78da554000 r-xp 00000000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f78da554000-7f78da753000 ---p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f78da753000-7f78da754000 r--p 00100000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f78da754000-7f78da755000 rw-p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f78da755000-7f78da83e000 r-xp 00000000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f78da83e000-7f78daa3d000 ---p 000e9000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f78daa3d000-7f78daa45000 r--p 000e8000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f78daa45000-7f78daa47000 rw-p 000f0000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f78daa47000-7f78daa5c000 rw-p 00000000 00:00 0 7f78daa5c000-7f78daa5e000 r-xp 00000000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f78daa5e000-7f78dac5e000 ---p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f78dac5e000-7f78dac5f000 r--p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f78dac5f000-7f78dac60000 rw-p 00003000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f78dac60000-7f78dae96000 r-xp 00000000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f78dae96000-7f78db096000 ---p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f78db096000-7f78db0b2000 r--p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f78db0b2000-7f78db0bf000 rw-p 00252000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f78db0bf000-7f78db0c3000 rw-p 00000000 00:00 0 7f78db0c3000-7f78db12a000 r-xp 00000000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f78db12a000-7f78db32a000 ---p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f78db32a000-7f78db32e000 r--p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f78db32e000-7f78db335000 rw-p 0006b000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f78db335000-7f78db364000 r-xp 00000000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f78db364000-7f78db564000 ---p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f78db564000-7f78db565000 r--p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f78db565000-7f78db566000 rw-p 00030000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f78db566000-7f78db56e000 r-xp 00000000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f78db56e000-7f78db76d000 ---p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f78db76d000-7f78db76e000 r--p 00007000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f78db76e000-7f78db76f000 rw-p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f78db76f000-7f78db79d000 rw-p 00000000 00:00 0 7f78db79d000-7f78db7b2000 r-xp 00000000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f78db7b2000-7f78db9b1000 ---p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f78db9b1000-7f78db9b2000 r--p 00014000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f78db9b2000-7f78db9b3000 rw-p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f78db9b3000-7f78db9b4000 r-xp 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f78db9b4000-7f78dbbb3000 ---p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f78dbbb3000-7f78dbbb4000 r--p 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f78dbbb4000-7f78dbbb5000 rw-p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f78dbbb5000-7f78dbbda000 r-xp 00000000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f78dbbda000-7f78dbdd9000 ---p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f78dbdd9000-7f78dbdda000 r--p 00024000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f78dbdda000-7f78dbddb000 rw-p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f78dbddb000-7f78dbdf2000 r-xp 00000000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f78dbdf2000-7f78dbff1000 ---p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f78dbff1000-7f78dbff2000 r--p 00016000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f78dbff2000-7f78dbff3000 rw-p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f78dbff3000-7f78dbff7000 rw-p 00000000 00:00 0 7f78dbff7000-7f78dc019000 r-xp 00000000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f78dc01c000-7f78dc01e000 rw-p 00000000 00:00 0 7f78dc01e000-7f78dc01f000 ---p 00000000 00:00 0 7f78dc01f000-7f78dc069000 rw-p 00000000 00:00 0 7f78dc069000-7f78dc06c000 rw-s 00000000 00:0b 34070 /[aio] (deleted) 7f78dc06c000-7f78dc06f000 rw-s 00000000 00:0b 34069 /[aio] (deleted) 7f78dc06f000-7f78dc072000 rw-s 00000000 00:0b 34068 /[aio] (deleted) 7f78dc072000-7f78dc075000 rw-s 00000000 00:0b 34067 /[aio] (deleted) 7f78dc075000-7f78dc078000 rw-s 00000000 00:0b 34066 /[aio] (deleted) 7f78dc078000-7f78dc07b000 rw-s 00000000 00:0b 34065 /[aio] (deleted) 7f78dc07b000-7f78dc07e000 rw-s 00000000 00:0b 34064 /[aio] (deleted) 7f78dc07e000-7f78dc081000 rw-s 00000000 00:0b 34063 /[aio] (deleted) 7f78dc081000-7f78dc20f000 rw-p 00000000 00:00 0 7f78dc20f000-7f78dc210000 rw-s 00000000 00:0b 34071 /[aio] (deleted) 7f78dc210000-7f78dc213000 rw-s 00000000 00:0b 34062 /[aio] (deleted) 7f78dc213000-7f78dc216000 rw-s 00000000 00:0b 34061 /[aio] (deleted) 7f78dc216000-7f78dc217000 rw-s 00000000 00:0b 34060 /[aio] (deleted) 7f78dc217000-7f78dc218000 rw-p 00000000 00:00 0 7f78dc218000-7f78dc219000 r--p 00021000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f78dc219000-7f78dc21a000 rw-p 00022000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f78dc21a000-7f78dc21b000 rw-p 00000000 00:00 0 7ffd0d30e000-7ffd0d32f000 rw-p 00000000 00:00 0 [stack] 7ffd0d3fa000-7ffd0d3fc000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Fatal signal 6 while backtracing 2020-01-31 18:00:27 0 [Note] mysqld: Aria engine: starting recovery tables to flush: 2 1 0 (0.0 seconds); 2020-01-31 18:00:27 0 [Note] mysqld: Aria engine: recovery done 2020-01-31 18:00:27 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:00:27 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:00:27 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:00:27 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:00:27 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:00:27 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:00:27 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:00:27 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:00:27 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:00:27 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135529723 2020-01-31 18:00:27 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000073', position 291012683 2020-01-31 18:00:27 0 [Note] InnoDB: Resizing redo log from 2*50331648 to 2*134217728 bytes; LSN=5135529741 2020-01-31 18:00:27 0 [Note] InnoDB: Starting to delete and rewrite log files. 2020-01-31 18:00:27 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 134217728 bytes 2020-01-31 18:00:27 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 134217728 bytes 2020-01-31 18:00:27 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2020-01-31 18:00:27 0 [Note] InnoDB: New log files created, LSN=5135529741 2020-01-31 18:00:27 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:00:27 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:00:27 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:00:27 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:00:27 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:00:27 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:00:27 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135529741; transaction id 5898129 2020-01-31 18:00:27 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:00:27 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:00:27 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:00:28 0 [Note] Starting crash recovery... 2020-01-31 18:00:28 0 [Note] Crash recovery finished. 2020-01-31 18:00:28 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:00:28 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:00:28 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:00:28 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000049' at position 393 2020-01-31 18:00:28 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000049' at position 393, relay log './dbsvc-528-relay-bin.000173' position: 696 2020-01-31 18:00:28 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:00:28 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 18:00:29 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000073, 291041118) 2020-01-31 18:00:30 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:00:30 2020-01-31 18:00:48 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:00:49 16 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 18:00:49 15 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 18:00:49 16 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 18:00:49 15 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 18:00:49 16 [ERROR] mysqld: Table './idtgroup/wp_posts' is marked as crashed and should be repaired 2020-01-31 18:00:50 16 [Warning] Checking table: './idtgroup/wp_posts' 2020-01-31 18:00:50 17 [ERROR] mysqld: Table './wpstart/wp_posts' is marked as crashed and should be repaired 2020-01-31 18:00:50 17 [Warning] Checking table: './wpstart/wp_posts' 2020-01-31 18:00:50 17 [ERROR] mysqld: Table './wpstart/wp_users' is marked as crashed and should be repaired 2020-01-31 18:00:50 17 [Warning] Checking table: './wpstart/wp_users' 2020-01-31 18:00:50 17 [ERROR] mysqld: Table './wpstart/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:00:50 17 [Warning] Checking table: './wpstart/wp_usermeta' 2020-01-31 18:00:50 16 [ERROR] mysqld: Table './idtgroup/wp_postmeta' is marked as crashed and should be repaired 2020-01-31 18:00:50 16 [Warning] Checking table: './idtgroup/wp_postmeta' 2020-01-31 18:00:51 20 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:00:51 20 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:00:51 20 [ERROR] mysqld: Table './wpwsutv/wp_nextend2_section_storage' is marked as crashed and should be repaired 2020-01-31 18:00:51 20 [Warning] Checking table: './wpwsutv/wp_nextend2_section_storage' 2020-01-31 18:00:52 22 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:00:52 22 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 18:00:52 22 [ERROR] mysqld: Table './wpwsutv/wp_posts' is marked as crashed and should be repaired 2020-01-31 18:00:52 22 [Warning] Checking table: './wpwsutv/wp_posts' 2020-01-31 18:00:52 22 [ERROR] mysqld: Table './wpwsutv/wp_postmeta' is marked as crashed and should be repaired 2020-01-31 18:00:52 22 [Warning] Checking table: './wpwsutv/wp_postmeta' 2020-01-31 18:00:52 20 [ERROR] mysqld: Table './wpwsutv/wp_term_taxonomy' is marked as crashed and should be repaired 2020-01-31 18:00:52 20 [Warning] Checking table: './wpwsutv/wp_term_taxonomy' 2020-01-31 18:00:52 20 [ERROR] mysqld: Table './wpwsutv/wp_term_relationships' is marked as crashed and should be repaired 2020-01-31 18:00:52 20 [Warning] Checking table: './wpwsutv/wp_term_relationships' 2020-01-31 18:00:57 32 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 18:00:57 32 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 18:00:57 32 [ERROR] mysqld: Table './wpcomm305/comm305_wfHits' is marked as crashed and should be repaired 2020-01-31 18:00:57 32 [Warning] Checking table: './wpcomm305/comm305_wfHits' 2020-01-31 18:00:58 32 [ERROR] mysqld: Table './wpcomm305/comm305_usermeta' is marked as crashed and should be repaired 2020-01-31 18:00:58 32 [Warning] Checking table: './wpcomm305/comm305_usermeta' 2020-01-31 18:00:58 32 [ERROR] mysqld: Table './wpcomm305/comm305_users' is marked as crashed and should be repaired 2020-01-31 18:00:58 32 [Warning] Checking table: './wpcomm305/comm305_users' 2020-01-31 18:01:13 41 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:01:13 41 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-01-31 18:01:22 45 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 18:01:22 45 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 18:01:23 45 [ERROR] mysqld: Table './ksarn/wp_posts' is marked as crashed and should be repaired 2020-01-31 18:01:23 45 [Warning] Checking table: './ksarn/wp_posts' 2020-01-31 18:01:23 45 [ERROR] mysqld: Table './ksarn/wp_postmeta' is marked as crashed and should be repaired 2020-01-31 18:01:23 45 [Warning] Checking table: './ksarn/wp_postmeta' 2020-01-31 18:01:24 47 [ERROR] mysqld: Table './ksarn/wp_term_taxonomy' is marked as crashed and should be repaired 2020-01-31 18:01:24 47 [Warning] Checking table: './ksarn/wp_term_taxonomy' 2020-01-31 18:01:24 47 [ERROR] mysqld: Table './ksarn/wp_term_relationships' is marked as crashed and should be repaired 2020-01-31 18:01:24 47 [Warning] Checking table: './ksarn/wp_term_relationships' 2020-01-31 18:01:25 50 [ERROR] mysqld: Table './ksarn/wp_bp_activity' is marked as crashed and should be repaired 2020-01-31 18:01:25 50 [Warning] Checking table: './ksarn/wp_bp_activity' 2020-01-31 18:01:25 50 [ERROR] mysqld: Table './ksarn/wp_bp_activity_meta' is marked as crashed and should be repaired 2020-01-31 18:01:25 50 [Warning] Checking table: './ksarn/wp_bp_activity_meta' 2020-01-31 18:01:25 50 [ERROR] mysqld: Table './ksarn/wp_eum_logs' is marked as crashed and should be repaired 2020-01-31 18:01:25 50 [Warning] Checking table: './ksarn/wp_eum_logs' 2020-01-31 18:01:25 50 [ERROR] mysqld: Table './ksarn/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 18:01:25 50 [Warning] Checking table: './ksarn/wp_gf_form_view' 2020-01-31 18:01:25 50 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity' is marked as crashed and should be repaired 2020-01-31 18:01:25 50 [Warning] Checking table: './ksarn/wp_learndash_user_activity' 2020-01-31 18:01:25 50 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity_meta' is marked as crashed and should be repaired 2020-01-31 18:01:25 50 [Warning] Checking table: './ksarn/wp_learndash_user_activity_meta' 2020-01-31 18:01:25 50 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:01:25 50 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 18:01:25 50 [ERROR] mysqld: Table './ksarn/wp_users' is marked as crashed and should be repaired 2020-01-31 18:01:25 50 [Warning] Checking table: './ksarn/wp_users' 2020-01-31 18:01:25 50 [ERROR] mysqld: Table './ksarn/wp_wp_pro_quiz_statistic' is marked as crashed and should be repaired 2020-01-31 18:01:25 50 [Warning] Checking table: './ksarn/wp_wp_pro_quiz_statistic' 2020-01-31 18:01:25 50 [ERROR] mysqld: Table './ksarn/wp_wp_pro_quiz_statistic_ref' is marked as crashed and should be repaired 2020-01-31 18:01:25 50 [Warning] Checking table: './ksarn/wp_wp_pro_quiz_statistic_ref' 2020-01-31 18:01:28 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000049' at position 393 2020-01-31 18:01:35 54 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:01:35 54 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:01:35 54 [ERROR] mysqld: Table './ojsdata/site' is marked as crashed and should be repaired 2020-01-31 18:01:35 54 [Warning] Checking table: './ojsdata/site' 2020-01-31 18:01:36 55 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:01:36 55 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 18:01:36 55 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 18:01:36 55 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 18:01:36 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f14f8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f15000afcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560d5781ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560d572b63af] sigaction.c:0(__restore_rt)[0x7f156aa6c630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x560d571e4151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x560d57322545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x560d570d8ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x560d570ced4b] /usr/sbin/mysqld(+0x6ca792)[0x560d570d9792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x560d571016da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560d5710365d] /usr/sbin/mysqld(+0x65e3f3)[0x560d5706d3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x560d5706ccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x560d57100b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560d5710365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x560d57103761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x560d57104377] /usr/sbin/mysqld(+0x58fd84)[0x560d56f9ed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x560d570abb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x560d570c3b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x560d570c3cd2] /usr/sbin/mysqld(+0x6b55e7)[0x560d570c45e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x560d570c4677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x560d570b356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x560d570b507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x560d57190afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560d57190bdd] pthread_create.c:0(start_thread)[0x7f156aa64ea5] /lib64/libc.so.6(clone+0x6d)[0x7f1568e058cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f14f800fe80): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 56 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:01:43 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:01:43 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:01:43 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:01:43 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:01:43 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:01:43 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:01:43 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:01:43 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:01:43 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:01:43 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135533977 2020-01-31 18:01:43 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000073', position 291012683 2020-01-31 18:01:43 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:01:43 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:01:43 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:01:43 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:01:43 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:01:43 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:01:43 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135533986; transaction id 5898159 2020-01-31 18:01:43 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:01:43 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:01:43 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:01:43 0 [Note] Starting crash recovery... 2020-01-31 18:01:43 0 [Note] Crash recovery finished. 2020-01-31 18:01:43 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:01:43 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:01:43 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:01:43 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000050' at position 346 2020-01-31 18:01:43 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000050' at position 346, relay log './dbsvc-528-relay-bin.000176' position: 649 2020-01-31 18:01:43 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:01:43 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000050' at position 346 2020-01-31 18:01:43 11 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 18:01:43 11 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 18:01:43 11 [Warning] wpcomm303.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:01:43 11 [Note] wpcomm303.wp_options: Table is fixed 2020-01-31 18:01:44 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:01:44 2020-01-31 18:01:54 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:01:54 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:01:54 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:01:54 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 18:01:54 11 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:01:54 11 [Warning] Checking table: './wpms/wp_usermeta' 2020-01-31 18:01:54 11 [Warning] wpms.wp_usermeta: 1 client is using or hasn't closed the table properly 2020-01-31 18:01:54 11 [Note] wpms.wp_usermeta: Table is fixed 2020-01-31 18:01:56 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:01:56 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:01:56 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 18:01:56 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 18:01:57 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:01:57 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 18:01:57 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 18:01:57 11 [Note] ojsdata.scheduled_tasks: Table is fixed 2020-01-31 18:01:57 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 18:01:57 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000050' at position 61649 2020-01-31 18:01:57 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 18:02:03 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000074, 763193) 2020-01-31 18:02:09 19 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:02:20 22 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:02:20 22 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:02:23 24 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 18:02:23 24 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 18:02:23 24 [ERROR] mysqld: Table './wpcomm306/wp_wfConfig' is marked as crashed and should be repaired 2020-01-31 18:02:23 24 [Warning] Checking table: './wpcomm306/wp_wfConfig' 2020-01-31 18:02:23 24 [ERROR] mysqld: Table './wpcomm306/wp_wfHits' is marked as crashed and should be repaired 2020-01-31 18:02:23 24 [Warning] Checking table: './wpcomm306/wp_wfHits' 2020-01-31 18:02:23 24 [ERROR] mysqld: Table './wpcomm306/wp_posts' is marked as crashed and should be repaired 2020-01-31 18:02:23 24 [Warning] Checking table: './wpcomm306/wp_posts' 2020-01-31 18:02:23 24 [ERROR] mysqld: Table './wpcomm306/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:02:23 24 [Warning] Checking table: './wpcomm306/wp_usermeta' 2020-01-31 18:02:25 26 [ERROR] mysqld: Table './wpcomm303/wp_wfConfig' is marked as crashed and should be repaired 2020-01-31 18:02:25 26 [Warning] Checking table: './wpcomm303/wp_wfConfig' 2020-01-31 18:02:25 26 [ERROR] mysqld: Table './wpcomm303/wp_wfHits' is marked as crashed and should be repaired 2020-01-31 18:02:25 26 [Warning] Checking table: './wpcomm303/wp_wfHits' 2020-01-31 18:02:25 26 [ERROR] mysqld: Table './wpcomm303/wp_posts' is marked as crashed and should be repaired 2020-01-31 18:02:25 26 [Warning] Checking table: './wpcomm303/wp_posts' 2020-01-31 18:02:26 26 [ERROR] mysqld: Table './wpcomm303/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:02:26 26 [Warning] Checking table: './wpcomm303/wp_usermeta' 2020-01-31 18:02:54 37 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 18:02:54 37 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 18:02:54 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f57f0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f5844b6fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x557b579def2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x557b574753af] sigaction.c:0(__restore_rt)[0x7f585fd53630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x557b573a3151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x557b574e1545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x557b57297ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x557b5728dd4b] /usr/sbin/mysqld(+0x6ca792)[0x557b57298792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x557b572c06da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557b572c265d] /usr/sbin/mysqld(+0x65e3f3)[0x557b5722c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x557b5722bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x557b572bfb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557b572c265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x557b572c2761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x557b572c3377] /usr/sbin/mysqld(+0x58fd84)[0x557b5715dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x557b5726ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x557b57282b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x557b57282cd2] /usr/sbin/mysqld(+0x6b55e7)[0x557b572835e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x557b57283677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x557b5727256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x557b5727407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x557b5734fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x557b5734fbdd] pthread_create.c:0(start_thread)[0x7f585fd4bea5] /lib64/libc.so.6(clone+0x6d)[0x7f585e0ec8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f57f000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 38 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:03:00 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:03:00 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:03:00 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:03:00 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:03:00 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:03:00 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:03:00 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:03:01 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:03:01 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:03:01 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135547800 2020-01-31 18:03:01 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000073', position 291012683 2020-01-31 18:03:01 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:03:01 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:03:01 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:03:01 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:03:01 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:03:01 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:03:01 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135547809; transaction id 5898222 2020-01-31 18:03:01 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:03:01 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:03:01 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:03:01 0 [Note] Starting crash recovery... 2020-01-31 18:03:01 0 [Note] Crash recovery finished. 2020-01-31 18:03:01 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:03:01 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:03:01 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:03:01 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000050' at position 61649 2020-01-31 18:03:01 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000050' at position 61649, relay log './dbsvc-528-relay-bin.000178' position: 40046 2020-01-31 18:03:01 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:03:01 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000050' at position 61649 2020-01-31 18:03:02 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:03:02 2020-01-31 18:03:05 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:03:05 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:03:05 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:03:05 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 18:03:08 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:03:08 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:03:08 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:03:08 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 18:03:10 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:03:10 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:03:10 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 18:03:10 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 18:03:10 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:03:10 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 18:03:10 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 18:03:10 11 [Note] ojsdata.scheduled_tasks: Table is fixed 2020-01-31 18:03:11 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 18:03:11 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000051' at position 28570 2020-01-31 18:03:11 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 18:03:17 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000075, 27830) 2020-01-31 18:03:29 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:03:30 15 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 18:03:30 15 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 18:03:30 16 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 18:03:30 16 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 18:03:30 15 [ERROR] mysqld: Table './pizzahut/wp_snippets' is marked as crashed and should be repaired 2020-01-31 18:03:30 15 [Warning] Checking table: './pizzahut/wp_snippets' 2020-01-31 18:03:30 16 [ERROR] mysqld: Table './itstraining/wp_comments' is marked as crashed and should be repaired 2020-01-31 18:03:30 16 [Warning] Checking table: './itstraining/wp_comments' 2020-01-31 18:03:30 15 [ERROR] mysqld: Table './pizzahut/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:03:30 15 [Warning] Checking table: './pizzahut/wp_usermeta' 2020-01-31 18:03:30 16 [ERROR] mysqld: Table './itstraining/wp_posts' is marked as crashed and should be repaired 2020-01-31 18:03:30 16 [Warning] Checking table: './itstraining/wp_posts' 2020-01-31 18:03:30 15 [ERROR] mysqld: Table './pizzahut/wp_users' is marked as crashed and should be repaired 2020-01-31 18:03:30 15 [Warning] Checking table: './pizzahut/wp_users' 2020-01-31 18:03:30 16 [ERROR] mysqld: Table './itstraining/wp_terms' is marked as crashed and should be repaired 2020-01-31 18:03:30 16 [Warning] Checking table: './itstraining/wp_terms' 2020-01-31 18:03:30 16 [ERROR] mysqld: Table './itstraining/wp_term_taxonomy' is marked as crashed and should be repaired 2020-01-31 18:03:30 16 [Warning] Checking table: './itstraining/wp_term_taxonomy' 2020-01-31 18:03:30 16 [ERROR] mysqld: Table './itstraining/wp_term_relationships' is marked as crashed and should be repaired 2020-01-31 18:03:30 16 [Warning] Checking table: './itstraining/wp_term_relationships' 2020-01-31 18:03:30 16 [ERROR] mysqld: Table './itstraining/wp_postmeta' is marked as crashed and should be repaired 2020-01-31 18:03:30 16 [Warning] Checking table: './itstraining/wp_postmeta' 2020-01-31 18:03:30 17 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 18:03:30 17 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 18:03:30 19 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 18:03:30 19 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 18:03:30 17 [ERROR] mysqld: Table './wpsem/wp_posts' is marked as crashed and should be repaired 2020-01-31 18:03:30 17 [Warning] Checking table: './wpsem/wp_posts' 2020-01-31 18:03:31 20 [ERROR] mysqld: Table './wpsem/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:03:31 20 [Warning] Checking table: './wpsem/wp_usermeta' 2020-01-31 18:03:31 20 [ERROR] mysqld: Table './wpsem/wp_postmeta' is marked as crashed and should be repaired 2020-01-31 18:03:31 20 [Warning] Checking table: './wpsem/wp_postmeta' 2020-01-31 18:03:32 22 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 18:03:32 22 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 18:03:32 22 [ERROR] mysqld: Table './openrecords/wp_term_taxonomy' is marked as crashed and should be repaired 2020-01-31 18:03:32 22 [Warning] Checking table: './openrecords/wp_term_taxonomy' 2020-01-31 18:03:33 24 [ERROR] mysqld: Table './openrecords/wp_gf_entry' is marked as crashed and should be repaired 2020-01-31 18:03:33 24 [Warning] Checking table: './openrecords/wp_gf_entry' 2020-01-31 18:03:33 24 [ERROR] mysqld: Table './openrecords/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 18:03:33 24 [Warning] Checking table: './openrecords/wp_gf_form_view' 2020-01-31 18:03:33 24 [ERROR] mysqld: Table './openrecords/wp_gf_entry_meta' is marked as crashed and should be repaired 2020-01-31 18:03:33 24 [Warning] Checking table: './openrecords/wp_gf_entry_meta' 2020-01-31 18:03:33 22 [ERROR] mysqld: Table './openrecords/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:03:33 22 [Warning] Checking table: './openrecords/wp_usermeta' 2020-01-31 18:03:33 22 [ERROR] mysqld: Table './openrecords/wp_users' is marked as crashed and should be repaired 2020-01-31 18:03:33 22 [Warning] Checking table: './openrecords/wp_users' 2020-01-31 18:03:55 36 [ERROR] mysqld: Table './wpms/wp_snippets' is marked as crashed and should be repaired 2020-01-31 18:03:55 36 [Warning] Checking table: './wpms/wp_snippets' 2020-01-31 18:03:55 36 [ERROR] mysqld: Table './wpms/wp_users' is marked as crashed and should be repaired 2020-01-31 18:03:55 36 [Warning] Checking table: './wpms/wp_users' 2020-01-31 18:03:55 36 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:03:55 36 [Warning] Checking table: './wpms/wp_usermeta' 2020-01-31 18:03:55 36 [ERROR] mysqld: Table './wpms/wp_posts' is marked as crashed and should be repaired 2020-01-31 18:03:55 36 [Warning] Checking table: './wpms/wp_posts' 2020-01-31 18:03:56 36 [ERROR] mysqld: Table './wpms/wp_term_taxonomy' is marked as crashed and should be repaired 2020-01-31 18:03:56 36 [Warning] Checking table: './wpms/wp_term_taxonomy' 2020-01-31 18:04:11 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000051' at position 28570 2020-01-31 18:04:32 51 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 18:04:32 51 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 18:04:33 51 [ERROR] mysqld: Table './wpcomm305/comm305_posts' is marked as crashed and should be repaired 2020-01-31 18:04:33 51 [Warning] Checking table: './wpcomm305/comm305_posts' 2020-01-31 18:04:33 51 [ERROR] mysqld: Table './wpcomm305/comm305_postmeta' is marked as crashed and should be repaired 2020-01-31 18:04:33 51 [Warning] Checking table: './wpcomm305/comm305_postmeta' 2020-01-31 18:04:33 51 [ERROR] mysqld: Table './wpcomm305/comm305_term_taxonomy' is marked as crashed and should be repaired 2020-01-31 18:04:33 51 [Warning] Checking table: './wpcomm305/comm305_term_taxonomy' 2020-01-31 18:04:33 51 [ERROR] mysqld: Table './wpcomm305/comm305_term_relationships' is marked as crashed and should be repaired 2020-01-31 18:04:33 51 [Warning] Checking table: './wpcomm305/comm305_term_relationships' 2020-01-31 18:04:33 51 [ERROR] mysqld: Table './wpcomm305/comm305_comments' is marked as crashed and should be repaired 2020-01-31 18:04:33 51 [Warning] Checking table: './wpcomm305/comm305_comments' 2020-01-31 18:05:34 87 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 18:05:34 87 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 18:05:34 87 [ERROR] mysqld: Table './ulrich/wp_posts' is marked as crashed and should be repaired 2020-01-31 18:05:34 87 [Warning] Checking table: './ulrich/wp_posts' 2020-01-31 18:05:34 87 [ERROR] mysqld: Table './ulrich/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:05:34 87 [Warning] Checking table: './ulrich/wp_usermeta' 2020-01-31 18:06:28 118 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 18:06:28 118 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 18:06:29 118 [ERROR] mysqld: Table './shockeripe/wp_posts' is marked as crashed and should be repaired 2020-01-31 18:06:29 118 [Warning] Checking table: './shockeripe/wp_posts' 2020-01-31 18:06:29 118 [ERROR] mysqld: Table './shockeripe/wp_terms' is marked as crashed and should be repaired 2020-01-31 18:06:29 118 [Warning] Checking table: './shockeripe/wp_terms' 2020-01-31 18:06:29 118 [ERROR] mysqld: Table './shockeripe/wp_term_taxonomy' is marked as crashed and should be repaired 2020-01-31 18:06:29 118 [Warning] Checking table: './shockeripe/wp_term_taxonomy' 2020-01-31 18:06:29 118 [ERROR] mysqld: Table './shockeripe/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:06:29 118 [Warning] Checking table: './shockeripe/wp_usermeta' 200131 18:06:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff6f4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff708fb1cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x564219970f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5642194073af] sigaction.c:0(__restore_rt)[0x7ff76bb87630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564219335151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564219473545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564219229ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56421921fd4b] /usr/sbin/mysqld(+0x6ca792)[0x56421922a792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5642192526da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56421925465d] /usr/sbin/mysqld(+0x65e3f3)[0x5642191be3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5642191bdcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x564219251b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56421925465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x564219254761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x564219255377] /usr/sbin/mysqld(+0x58fd84)[0x5642190efd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5642191fcb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564219214b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564219214cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5642192155e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564219215677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56421920456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56421920607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5642192e1afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5642192e1bdd] pthread_create.c:0(start_thread)[0x7ff76bb7fea5] /lib64/libc.so.6(clone+0x6d)[0x7ff769f208cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff6f400fe70): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 132 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:07:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:07:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:07:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:07:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:07:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:07:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:07:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:07:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:07:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:07:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135558186 2020-01-31 18:07:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000073', position 291012683 2020-01-31 18:07:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:07:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:07:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:07:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:07:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:07:03 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:07:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135558195; transaction id 5898259 2020-01-31 18:07:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:07:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:07:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:07:04 0 [Note] Starting crash recovery... 2020-01-31 18:07:04 0 [Note] Crash recovery finished. 2020-01-31 18:07:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:07:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:07:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:07:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000052' at position 346 2020-01-31 18:07:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000052' at position 346, relay log './dbsvc-528-relay-bin.000183' position: 649 2020-01-31 18:07:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:07:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000052' at position 346 2020-01-31 18:07:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:07:05 2020-01-31 18:07:10 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 18:07:10 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 18:07:10 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:07:10 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 18:07:21 11 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 18:07:21 11 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 18:07:21 11 [Warning] wpstart.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:07:21 11 [Note] wpstart.wp_options: Table is fixed 2020-01-31 18:07:30 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:07:33 20 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:07:33 20 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:07:56 28 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:07:56 28 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:07:58 30 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000076, 711711) 2020-01-31 18:09:31 77 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 18:09:31 77 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 18:09:31 77 [ERROR] mysqld: Table './itstraining/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:09:31 77 [Warning] Checking table: './itstraining/wp_usermeta' 2020-01-31 18:10:02 88 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 18:10:02 88 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 18:10:02 88 [ERROR] mysqld: Table './openrecords/wp_posts' is marked as crashed and should be repaired 2020-01-31 18:10:02 88 [Warning] Checking table: './openrecords/wp_posts' 2020-01-31 18:10:02 88 [ERROR] mysqld: Table './openrecords/wp_postmeta' is marked as crashed and should be repaired 2020-01-31 18:10:02 88 [Warning] Checking table: './openrecords/wp_postmeta' 2020-01-31 18:10:02 88 [ERROR] mysqld: Table './openrecords/wp_term_relationships' is marked as crashed and should be repaired 2020-01-31 18:10:02 88 [Warning] Checking table: './openrecords/wp_term_relationships' 2020-01-31 18:11:40 124 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 18:11:40 124 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 18:11:56 131 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:11:56 131 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:11:56 132 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:11:56 132 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:11:56 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f7ff8006c38 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f8068b7ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e828645f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e8280dc3af] sigaction.c:0(__restore_rt)[0x7f8085c7a630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55e82800a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55e828148545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55e827efeccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55e827ef4d4b] /usr/sbin/mysqld(+0x6ca792)[0x55e827eff792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55e827f276da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e827f2965d] /usr/sbin/mysqld(+0x65e3f3)[0x55e827e933f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55e827e92cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55e827f26b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e827f2965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55e827f29761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55e827f2a377] /usr/sbin/mysqld(+0x58fd84)[0x55e827dc4d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55e827ed1b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55e827ee9b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55e827ee9cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55e827eea5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55e827eea677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55e827ed956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55e827edb07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55e827fb6afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e827fb6bdd] pthread_create.c:0(start_thread)[0x7f8085c72ea5] /lib64/libc.so.6(clone+0x6d)[0x7f80840138cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f7ff8010ee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 133 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:12:02 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:12:02 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:12:02 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:12:02 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:12:02 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:12:02 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:12:02 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:12:02 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:12:02 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:12:02 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135592834 2020-01-31 18:12:02 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000073', position 291012683 2020-01-31 18:12:02 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:12:02 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:12:02 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:12:02 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:12:02 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:12:02 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:12:02 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135592843; transaction id 5898407 2020-01-31 18:12:02 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:12:02 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:12:02 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:12:02 0 [Note] Starting crash recovery... 2020-01-31 18:12:02 0 [Note] Crash recovery finished. 2020-01-31 18:12:02 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:12:02 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:12:02 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:12:02 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000052' at position 358723 2020-01-31 18:12:02 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000052' at position 358723, relay log './dbsvc-528-relay-bin.000185' position: 235774 2020-01-31 18:12:02 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:12:02 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000052' at position 358723 2020-01-31 18:12:03 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:12:03 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:12:03 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:12:03 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 18:12:04 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:12:04 2020-01-31 18:12:21 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:12:21 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:12:21 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:12:21 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 18:12:24 11 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 18:12:24 11 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 18:12:24 11 [Warning] wpcomm306.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:12:24 11 [Note] wpcomm306.wp_options: Table is fixed 2020-01-31 18:12:24 11 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:12:24 11 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 18:12:24 11 [Warning] wpwsutv.wp_usermeta: 1 client is using or hasn't closed the table properly 2020-01-31 18:12:24 11 [Note] wpwsutv.wp_usermeta: Table is fixed 2020-01-31 18:12:30 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:12:46 21 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 18:12:46 21 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 18:12:57 25 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000077, 249718) 2020-01-31 18:12:59 26 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 18:12:59 26 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 18:13:23 37 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 18:13:23 37 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 18:13:28 43 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 18:13:28 43 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 18:14:32 65 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 18:14:32 65 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 18:15:13 78 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 18:15:13 78 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 18:15:34 87 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 18:15:34 87 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 18:15:45 95 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 18:15:45 95 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 18:16:22 119 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:16:22 119 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:16:22 120 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:16:22 120 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:16:22 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=502 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f19a0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f195cffccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55de164fef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55de15f953af] sigaction.c:0(__restore_rt)[0x7f19c0034630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55de15ec3151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55de16001545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55de15db7ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55de15dadd4b] /usr/sbin/mysqld(+0x6ca792)[0x55de15db8792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55de15de06da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55de15de265d] /usr/sbin/mysqld(+0x65e3f3)[0x55de15d4c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55de15d4bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55de15ddfb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55de15de265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55de15de2761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55de15de3377] /usr/sbin/mysqld(+0x58fd84)[0x55de15c7dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55de15d8ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55de15da2b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55de15da2cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55de15da35e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55de15da3677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55de15d9256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55de15d9407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55de15e6fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55de15e6fbdd] pthread_create.c:0(start_thread)[0x7f19c002cea5] /lib64/libc.so.6(clone+0x6d)[0x7f19be3cd8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f19a000fe70): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 121 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:16:28 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:16:28 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:16:28 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:16:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:16:28 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:16:28 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:16:28 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:16:28 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:16:28 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:16:28 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135609323 2020-01-31 18:16:28 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000073', position 291012683 2020-01-31 18:16:28 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:16:28 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:16:28 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:16:28 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:16:28 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:16:28 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135609332; transaction id 5898473 2020-01-31 18:16:28 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:16:28 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:16:28 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:16:28 0 [Note] Starting crash recovery... 2020-01-31 18:16:28 0 [Note] Crash recovery finished. 2020-01-31 18:16:29 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:16:29 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:16:29 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:16:29 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000052' at position 448753 2020-01-31 18:16:29 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000052' at position 448753, relay log './dbsvc-528-relay-bin.000187' position: 59213 2020-01-31 18:16:29 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:16:29 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 18:16:30 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:16:30 2020-01-31 18:16:33 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000078, 813085) 2020-01-31 18:16:56 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:16:56 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:16:56 15 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:16:56 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:16:56 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:16:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9634000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f96914d4cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56531664df2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5653160e43af] sigaction.c:0(__restore_rt)[0x7f96ac69d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x565316012151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x565316150545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x565315f06ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x565315efcd4b] /usr/sbin/mysqld(+0x6ca792)[0x565315f07792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x565315f2f6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x565315f3165d] /usr/sbin/mysqld(+0x65e3f3)[0x565315e9b3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x565315e9acb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x565315f2eb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x565315f3165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x565315f31761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x565315f32377] /usr/sbin/mysqld(+0x58fd84)[0x565315dccd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x565315ed9b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x565315ef1b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x565315ef1cd2] /usr/sbin/mysqld(+0x6b55e7)[0x565315ef25e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x565315ef2677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x565315ee156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x565315ee307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x565315fbeafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x565315fbebdd] pthread_create.c:0(start_thread)[0x7f96ac695ea5] /lib64/libc.so.6(clone+0x6d)[0x7f96aaa368cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f963400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:17:02 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:17:02 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:17:02 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:17:02 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:17:02 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:17:02 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:17:02 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:17:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:17:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:17:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135614837 2020-01-31 18:17:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000073', position 291012683 2020-01-31 18:17:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:17:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:17:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:17:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:17:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:17:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135614846; transaction id 5898482 2020-01-31 18:17:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:17:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:17:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:17:03 0 [Note] Starting crash recovery... 2020-01-31 18:17:03 0 [Note] Crash recovery finished. 2020-01-31 18:17:03 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:17:03 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:17:03 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:17:03 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000052' at position 448753 2020-01-31 18:17:03 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000052' at position 448753, relay log './dbsvc-528-relay-bin.000188' position: 4 2020-01-31 18:17:03 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:17:03 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000052' at position 448753 2020-01-31 18:17:03 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:17:03 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:17:03 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 18:17:03 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:17:03 2020-01-31 18:17:03 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 18:17:03 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:17:03 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 18:17:03 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 18:17:03 11 [Note] ojsdata.scheduled_tasks: Table is fixed 2020-01-31 18:17:11 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:17:11 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:17:11 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:17:11 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 18:17:14 11 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 18:17:14 11 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 18:17:14 11 [Warning] wpms.wp_gf_form_view: 1 client is using or hasn't closed the table properly 2020-01-31 18:17:14 11 [Note] wpms.wp_gf_form_view: Table is fixed 2020-01-31 18:17:25 11 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 18:17:25 11 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 18:17:25 11 [Warning] wpcomm303.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:17:25 11 [Note] wpcomm303.wp_options: Table is fixed 2020-01-31 18:17:31 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:17:34 13 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 18:17:34 13 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 18:17:57 23 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000079, 1454) 2020-01-31 18:18:39 37 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 18:18:39 37 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 18:18:50 44 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:18:50 44 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:18:51 45 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:18:51 45 [Warning] Checking table: './wpwsutv/wp_usermeta' 200131 18:21:56 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f995c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f9919fb3cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55956ca38f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55956c4cf3af] sigaction.c:0(__restore_rt)[0x7f997b1f8630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55956c3fd151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55956c53b545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55956c2f1ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55956c2e7d4b] /usr/sbin/mysqld(+0x6ca792)[0x55956c2f2792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55956c31a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55956c31c65d] /usr/sbin/mysqld(+0x65e3f3)[0x55956c2863f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55956c285cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55956c319b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55956c31c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55956c31c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55956c31d377] /usr/sbin/mysqld(+0x58fd84)[0x55956c1b7d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55956c2c4b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55956c2dcb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55956c2dccd2] /usr/sbin/mysqld(+0x6b55e7)[0x55956c2dd5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55956c2dd677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55956c2cc56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55956c2ce07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55956c3a9afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55956c3a9bdd] pthread_create.c:0(start_thread)[0x7f997b1f0ea5] /lib64/libc.so.6(clone+0x6d)[0x7f99795918cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f995c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 130 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:22:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:22:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:22:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:22:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:22:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:22:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:22:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:22:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:22:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:22:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135638122 2020-01-31 18:22:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000080', position 782401 2020-01-31 18:22:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:22:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:22:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:22:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:22:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:22:03 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:22:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135638131; transaction id 5898600 2020-01-31 18:22:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:22:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:22:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:22:03 0 [Note] Starting crash recovery... 2020-01-31 18:22:03 0 [Note] Crash recovery finished. 2020-01-31 18:22:03 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:22:03 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:22:03 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:22:03 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000053' at position 186561 2020-01-31 18:22:03 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000053' at position 186561, relay log './dbsvc-528-relay-bin.000191' position: 121934 2020-01-31 18:22:03 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:22:03 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000053' at position 186561 2020-01-31 18:22:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:22:05 2020-01-31 18:22:09 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:22:09 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:22:09 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:22:09 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 18:22:32 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:22:35 18 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:22:35 18 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:22:36 19 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:22:36 19 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 18:22:57 37 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 18:22:57 37 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 18:22:58 39 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000080, 819238) 2020-01-31 18:23:00 41 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 18:23:00 41 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 18:23:15 50 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 18:23:15 50 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 18:23:23 56 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 18:23:23 56 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 18:23:24 58 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 18:23:24 58 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 18:23:28 64 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 18:23:28 64 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 18:23:45 73 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:23:45 73 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:23:45 74 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:23:45 74 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 18:23:45 74 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 18:23:45 74 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 18:23:45 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe964000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe9850e8cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d11d935f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55d11d3cc3af] sigaction.c:0(__restore_rt)[0x7fe9e622e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55d11d2fa151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55d11d438545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55d11d1eeccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55d11d1e4d4b] /usr/sbin/mysqld(+0x6ca792)[0x55d11d1ef792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55d11d2176da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d11d21965d] /usr/sbin/mysqld(+0x65e3f3)[0x55d11d1833f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55d11d182cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55d11d216b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d11d21965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55d11d219761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55d11d21a377] /usr/sbin/mysqld(+0x58fd84)[0x55d11d0b4d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55d11d1c1b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55d11d1d9b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55d11d1d9cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55d11d1da5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55d11d1da677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55d11d1c956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55d11d1cb07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55d11d2a6afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d11d2a6bdd] pthread_create.c:0(start_thread)[0x7fe9e6226ea5] /lib64/libc.so.6(clone+0x6d)[0x7fe9e45c78cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe96400feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 75 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:23:51 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:23:51 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:23:51 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:23:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:23:51 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:23:51 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:23:51 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:23:52 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:23:52 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:23:52 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135650137 2020-01-31 18:23:52 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000080', position 782401 2020-01-31 18:23:52 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:23:52 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:23:52 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:23:52 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:23:52 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:23:52 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:23:52 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135650146; transaction id 5898623 2020-01-31 18:23:52 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:23:52 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:23:52 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:23:52 0 [Note] Starting crash recovery... 2020-01-31 18:23:52 0 [Note] Crash recovery finished. 2020-01-31 18:23:52 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:23:52 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:23:52 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:23:52 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000053' at position 210420 2020-01-31 18:23:52 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000053' at position 210420, relay log './dbsvc-528-relay-bin.000193' position: 16095 2020-01-31 18:23:52 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:23:52 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000053' at position 210420 2020-01-31 18:23:53 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:23:53 2020-01-31 18:24:10 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:24:10 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:24:10 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:24:10 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 18:24:17 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:24:27 16 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 18:24:27 16 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 18:24:32 19 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 18:24:32 19 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 18:24:46 23 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000081, 393913) 2020-01-31 18:25:13 30 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 18:25:13 30 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 18:25:34 39 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 18:25:34 39 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 18:25:49 46 [ERROR] mysqld: Table './wpms/wp_gf_entry' is marked as crashed and should be repaired 2020-01-31 18:25:49 46 [Warning] Checking table: './wpms/wp_gf_entry' 2020-01-31 18:25:49 46 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 18:25:49 46 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 18:25:49 44 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:25:49 44 [Warning] Checking table: './wpms/wp_usermeta' 2020-01-31 18:25:49 44 [ERROR] mysqld: Table './wpms/wp_gf_entry_meta' is marked as crashed and should be repaired 2020-01-31 18:25:49 44 [Warning] Checking table: './wpms/wp_gf_entry_meta' 2020-01-31 18:25:49 44 [ERROR] mysqld: Table './wpms/wp_gf_entry_notes' is marked as crashed and should be repaired 2020-01-31 18:25:49 44 [Warning] Checking table: './wpms/wp_gf_entry_notes' 2020-01-31 18:26:04 52 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:26:04 52 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:26:06 53 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:26:06 53 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 18:26:09 57 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:26:09 57 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:26:09 58 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:26:09 58 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:26:09 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f6eb8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f6f140b1cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x555d1ad65f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x555d1a7fc3af] sigaction.c:0(__restore_rt)[0x7f6f2e3cd630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x555d1a72a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x555d1a868545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x555d1a61eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x555d1a614d4b] /usr/sbin/mysqld(+0x6ca792)[0x555d1a61f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x555d1a6476da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x555d1a64965d] /usr/sbin/mysqld(+0x65e3f3)[0x555d1a5b33f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x555d1a5b2cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x555d1a646b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x555d1a64965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x555d1a649761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x555d1a64a377] /usr/sbin/mysqld(+0x58fd84)[0x555d1a4e4d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x555d1a5f1b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x555d1a609b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x555d1a609cd2] /usr/sbin/mysqld(+0x6b55e7)[0x555d1a60a5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x555d1a60a677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x555d1a5f956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x555d1a5fb07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x555d1a6d6afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x555d1a6d6bdd] pthread_create.c:0(start_thread)[0x7f6f2e3c5ea5] /lib64/libc.so.6(clone+0x6d)[0x7f6f2c7668cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f6eb800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 59 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:26:15 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:26:15 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:26:15 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:26:15 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:26:15 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:26:15 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:26:15 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:26:15 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:26:15 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:26:15 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135658911 2020-01-31 18:26:15 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000080', position 782401 2020-01-31 18:26:15 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:26:15 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:26:15 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:26:15 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:26:15 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:26:15 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:26:15 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135658920; transaction id 5898650 2020-01-31 18:26:15 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:26:15 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:26:15 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:26:15 0 [Note] Starting crash recovery... 2020-01-31 18:26:15 0 [Note] Crash recovery finished. 2020-01-31 18:26:15 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:26:15 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:26:15 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:26:15 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000053' at position 234279 2020-01-31 18:26:15 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000053' at position 234279, relay log './dbsvc-528-relay-bin.000195' position: 16095 2020-01-31 18:26:15 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:26:15 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000053' at position 234279 2020-01-31 18:26:16 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:26:16 2020-01-31 18:26:32 11 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 18:26:32 11 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 18:26:32 11 [Warning] shockeripe.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:26:32 11 [Note] shockeripe.wp_options: Table is fixed 2020-01-31 18:26:35 11 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 18:26:35 11 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 18:26:35 11 [Warning] wpstart.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:26:35 11 [Note] wpstart.wp_options: Table is fixed 2020-01-31 18:26:43 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:26:56 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:26:56 15 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:26:56 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:26:56 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:26:56 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fa374000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fa3ec04fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c3884d1f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c387f683af] sigaction.c:0(__restore_rt)[0x7fa3f4e54630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c387e96151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c387fd4545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c387d8accf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c387d80d4b] /usr/sbin/mysqld(+0x6ca792)[0x55c387d8b792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c387db36da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c387db565d] /usr/sbin/mysqld(+0x65e3f3)[0x55c387d1f3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c387d1ecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c387db2b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c387db565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c387db5761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c387db6377] /usr/sbin/mysqld(+0x58fd84)[0x55c387c50d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c387d5db6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c387d75b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c387d75cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c387d765e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c387d76677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c387d6556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c387d6707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c387e42afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c387e42bdd] pthread_create.c:0(start_thread)[0x7fa3f4e4cea5] /lib64/libc.so.6(clone+0x6d)[0x7fa3f31ed8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fa37400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:27:02 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:27:02 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:27:02 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:27:02 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:27:02 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:27:02 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:27:02 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:27:02 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:27:02 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:27:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135672513 2020-01-31 18:27:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000080', position 782401 2020-01-31 18:27:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:27:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:27:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:27:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:27:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:27:03 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:27:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135672522; transaction id 5898717 2020-01-31 18:27:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:27:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:27:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:27:03 0 [Note] Starting crash recovery... 2020-01-31 18:27:03 0 [Note] Crash recovery finished. 2020-01-31 18:27:03 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:27:03 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:27:03 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:27:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000053' at position 409214 2020-01-31 18:27:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000053' at position 409214, relay log './dbsvc-528-relay-bin.000197' position: 115184 2020-01-31 18:27:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:27:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000053' at position 409214 2020-01-31 18:27:04 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:27:04 2020-01-31 18:27:10 17 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000082, 203858) 2020-01-31 18:27:25 11 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 18:27:25 11 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 18:27:25 11 [Warning] wpcomm303.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:27:25 11 [Note] wpcomm303.wp_options: Table is fixed 2020-01-31 18:27:27 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:27:27 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:27:27 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:27:27 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 18:27:28 18 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:27:31 19 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 18:27:31 19 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 18:27:35 21 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:27:35 21 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:27:35 22 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:27:35 22 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:27:36 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f51d4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f51d909acf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563d7be93f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x563d7b92a3af] sigaction.c:0(__restore_rt)[0x7f51f4279630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x563d7b858151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x563d7b996545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x563d7b74cccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x563d7b742d4b] /usr/sbin/mysqld(+0x6ca792)[0x563d7b74d792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x563d7b7756da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563d7b77765d] /usr/sbin/mysqld(+0x65e3f3)[0x563d7b6e13f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x563d7b6e0cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x563d7b774b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563d7b77765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x563d7b777761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x563d7b778377] /usr/sbin/mysqld(+0x58fd84)[0x563d7b612d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x563d7b71fb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x563d7b737b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x563d7b737cd2] /usr/sbin/mysqld(+0x6b55e7)[0x563d7b7385e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x563d7b738677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x563d7b72756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x563d7b72907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x563d7b804afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x563d7b804bdd] pthread_create.c:0(start_thread)[0x7f51f4271ea5] /lib64/libc.so.6(clone+0x6d)[0x7f51f26128cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f51d400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 23 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:27:42 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:27:42 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:27:42 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:27:42 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:27:42 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:27:42 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:27:42 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:27:42 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:27:42 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:27:42 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135683933 2020-01-31 18:27:42 0 [Note] InnoDB: Starting final batch to recover 2 pages from redo log. 2020-01-31 18:27:43 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000080', position 782401 2020-01-31 18:27:43 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:27:43 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:27:43 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:27:43 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:27:43 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:27:43 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:27:43 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135685941; transaction id 5898768 2020-01-31 18:27:43 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:27:43 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:27:43 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:27:43 0 [Note] Starting crash recovery... 2020-01-31 18:27:43 0 [Note] Crash recovery finished. 2020-01-31 18:27:43 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:27:43 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:27:43 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:27:43 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000053' at position 454999 2020-01-31 18:27:43 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000053' at position 454999, relay log './dbsvc-528-relay-bin.000199' position: 30330 2020-01-31 18:27:43 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:27:43 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000053' at position 454999 2020-01-31 18:27:44 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:27:44 2020-01-31 18:27:53 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:27:53 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:27:53 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:27:53 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 18:28:08 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:28:11 13 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:28:11 13 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:28:37 20 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000084, 2638) 2020-01-31 18:28:42 22 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 18:28:42 22 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 18:29:06 31 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:29:06 31 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:29:06 32 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:29:06 32 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 18:29:06 32 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 18:29:06 32 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 18:29:06 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9e78000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f9e7c77dcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5583bd588f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5583bd01f3af] sigaction.c:0(__restore_rt)[0x7f9e999a3630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5583bcf4d151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5583bd08b545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5583bce41ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5583bce37d4b] /usr/sbin/mysqld(+0x6ca792)[0x5583bce42792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5583bce6a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5583bce6c65d] /usr/sbin/mysqld(+0x65e3f3)[0x5583bcdd63f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5583bcdd5cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5583bce69b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5583bce6c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5583bce6c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5583bce6d377] /usr/sbin/mysqld(+0x58fd84)[0x5583bcd07d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5583bce14b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5583bce2cb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5583bce2ccd2] /usr/sbin/mysqld(+0x6b55e7)[0x5583bce2d5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5583bce2d677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5583bce1c56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5583bce1e07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5583bcef9afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5583bcef9bdd] pthread_create.c:0(start_thread)[0x7f9e9999bea5] /lib64/libc.so.6(clone+0x6d)[0x7f9e97d3c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f9e7800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 33 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:29:12 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:29:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:29:12 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:29:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:29:12 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:29:12 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:29:12 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:29:12 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:29:12 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:29:12 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135693376 2020-01-31 18:29:12 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000080', position 782401 2020-01-31 18:29:12 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:29:12 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:29:12 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:29:12 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:29:12 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:29:12 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:29:12 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135693385; transaction id 5898787 2020-01-31 18:29:12 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:29:12 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:29:12 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:29:12 0 [Note] Starting crash recovery... 2020-01-31 18:29:12 0 [Note] Crash recovery finished. 2020-01-31 18:29:12 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:29:12 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:29:12 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:29:12 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000053' at position 456004 2020-01-31 18:29:12 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000053' at position 456004, relay log './dbsvc-528-relay-bin.000201' position: 1267 2020-01-31 18:29:12 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:29:12 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000053' at position 456004 2020-01-31 18:29:13 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:29:13 2020-01-31 18:29:16 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:29:16 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:29:16 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:29:16 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 18:29:18 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:29:18 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:29:18 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 18:29:18 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 18:29:18 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:29:18 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 18:29:18 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 18:29:18 11 [Note] ojsdata.scheduled_tasks: Table is fixed 2020-01-31 18:29:19 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 18:29:19 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000053' at position 458281 2020-01-31 18:29:19 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 18:29:25 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000085, 25465) 2020-01-31 18:29:39 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:29:40 16 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 18:29:40 16 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 18:29:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fd6400073a8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fd65cffccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561ed6ed3f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x561ed696a3af] sigaction.c:0(__restore_rt)[0x7fd6bfbeb630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x561ed6898151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x561ed69d6545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x561ed678cccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x561ed6782d4b] /usr/sbin/mysqld(+0x6ca792)[0x561ed678d792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x561ed67b56da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x561ed67b765d] /usr/sbin/mysqld(+0x65e3f3)[0x561ed67213f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x561ed6720cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x561ed67b4b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x561ed67b765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x561ed67b7761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x561ed67b8377] /usr/sbin/mysqld(+0x58fd84)[0x561ed6652d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x561ed675fb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x561ed6777b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x561ed6777cd2] /usr/sbin/mysqld(+0x6b55e7)[0x561ed67785e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x561ed6778677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x561ed676756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x561ed676907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x561ed6844afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x561ed6844bdd] pthread_create.c:0(start_thread)[0x7fd6bfbe3ea5] /lib64/libc.so.6(clone+0x6d)[0x7fd6bdf848cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fd640011650): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:29:46 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:29:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:29:46 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:29:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:29:46 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:29:46 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:29:46 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:29:46 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:29:46 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:29:46 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135700635 2020-01-31 18:29:46 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000080', position 782401 2020-01-31 18:29:46 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:29:46 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:29:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:29:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:29:46 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:29:46 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135700644; transaction id 5898811 2020-01-31 18:29:46 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:29:46 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:29:46 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:29:46 0 [Note] Starting crash recovery... 2020-01-31 18:29:46 0 [Note] Crash recovery finished. 2020-01-31 18:29:46 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:29:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:29:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:29:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000053' at position 458281 2020-01-31 18:29:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000053' at position 458281, relay log './dbsvc-528-relay-bin.000203' position: 2019 2020-01-31 18:29:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:29:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000053' at position 458281 2020-01-31 18:29:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:29:48 2020-01-31 18:29:54 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:29:54 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:29:54 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 18:29:54 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 18:29:55 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:29:55 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 18:29:55 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 18:29:55 11 [Note] ojsdata.scheduled_tasks: Table is fixed 2020-01-31 18:29:55 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 18:29:55 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000054' at position 3167 2020-01-31 18:29:55 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 18:30:01 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000086, 1869) 2020-01-31 18:30:14 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:30:15 14 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:30:15 14 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:30:15 15 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:30:15 15 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:30:55 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000054' at position 3167 2020-01-31 18:30:55 29 [ERROR] mysqld: Table './idtgroup/wp_comments' is marked as crashed and should be repaired 2020-01-31 18:30:55 29 [Warning] Checking table: './idtgroup/wp_comments' 2020-01-31 18:31:14 39 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 18:31:14 39 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 18:31:40 52 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 18:31:40 52 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 18:31:45 56 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 18:31:45 56 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 18:31:53 59 [ERROR] mysqld: Table './wpms/wp_term_relationships' is marked as crashed and should be repaired 2020-01-31 18:31:53 59 [Warning] Checking table: './wpms/wp_term_relationships' 2020-01-31 18:31:53 59 [ERROR] mysqld: Table './wpms/wp_postmeta' is marked as crashed and should be repaired 2020-01-31 18:31:53 59 [Warning] Checking table: './wpms/wp_postmeta' 200131 18:31:56 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=502 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fcc64000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fccb4b33cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c3b64c9f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c3b5f603af] sigaction.c:0(__restore_rt)[0x7fccd1d06630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c3b5e8e151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c3b5fcc545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c3b5d82ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c3b5d78d4b] /usr/sbin/mysqld(+0x6ca792)[0x55c3b5d83792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c3b5dab6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c3b5dad65d] /usr/sbin/mysqld(+0x65e3f3)[0x55c3b5d173f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c3b5d16cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c3b5daab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c3b5dad65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c3b5dad761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c3b5dae377] /usr/sbin/mysqld(+0x58fd84)[0x55c3b5c48d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c3b5d55b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c3b5d6db16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c3b5d6dcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c3b5d6e5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c3b5d6e677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c3b5d5d56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c3b5d5f07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c3b5e3aafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c3b5e3abdd] pthread_create.c:0(start_thread)[0x7fccd1cfeea5] /lib64/libc.so.6(clone+0x6d)[0x7fccd009f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fcc6400ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 64 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:32:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:32:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:32:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:32:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:32:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:32:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:32:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:32:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:32:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:32:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135707443 2020-01-31 18:32:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000080', position 782401 2020-01-31 18:32:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:32:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:32:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:32:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:32:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:32:03 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:32:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135707452; transaction id 5898838 2020-01-31 18:32:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:32:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:32:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:32:03 0 [Note] Starting crash recovery... 2020-01-31 18:32:03 0 [Note] Crash recovery finished. 2020-01-31 18:32:03 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:32:03 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:32:03 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:32:03 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 346 2020-01-31 18:32:03 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000055' at position 346, relay log './dbsvc-528-relay-bin.000208' position: 649 2020-01-31 18:32:03 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:32:03 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 346 2020-01-31 18:32:04 11 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 18:32:04 11 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 18:32:04 11 [Warning] wpstart.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:32:04 11 [Note] wpstart.wp_options: Table is fixed 2020-01-31 18:32:04 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:32:04 2020-01-31 18:32:15 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:32:15 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:32:15 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:32:15 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 18:32:18 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:32:18 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:32:18 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:32:18 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 18:32:29 19 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:32:40 23 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 18:32:40 23 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 18:32:57 27 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000087, 163535) 2020-01-31 18:32:59 28 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 18:32:59 28 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 18:33:00 30 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 18:33:00 30 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 18:33:15 41 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 18:33:15 41 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 18:33:21 45 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 18:33:21 45 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 18:33:28 56 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 18:33:28 56 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 18:33:52 78 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 18:33:52 78 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 18:35:34 119 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 18:35:34 119 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 18:36:34 147 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 18:36:34 147 [Warning] Checking table: './shockeripe/wp_options' 200131 18:36:56 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=8 max_threads=502 thread_count=17 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f71ec000ac8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f725418acf0 thread_stack 0x49000 *** buffer overflow detected ***: /usr/sbin/mysqld terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7f726d41b567] /lib64/libc.so.6(+0x1166e2)[0x7f726d4196e2] /lib64/libc.so.6(+0x1184c7)[0x7f726d41b4c7] /usr/sbin/mysqld(my_addr_resolve+0xda)[0x564f9d7f294a] /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x564f9d7db0c2] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x564f9d2713af] /lib64/libpthread.so.0(+0xf630)[0x7f726f068630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564f9d19f151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564f9d2dd545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564f9d093ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x564f9d089d4b] /usr/sbin/mysqld(+0x6ca792)[0x564f9d094792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x564f9d0bc6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564f9d0be65d] /usr/sbin/mysqld(+0x65e3f3)[0x564f9d0283f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x564f9d027cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x564f9d0bbb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564f9d0be65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x564f9d0be761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x564f9d0bf377] /usr/sbin/mysqld(+0x58fd84)[0x564f9cf59d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x564f9d066b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564f9d07eb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564f9d07ecd2] /usr/sbin/mysqld(+0x6b55e7)[0x564f9d07f5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564f9d07f677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x564f9d06e56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x564f9d07007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x564f9d14bafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x564f9d14bbdd] /lib64/libpthread.so.0(+0x7ea5)[0x7f726f060ea5] /lib64/libc.so.6(clone+0x6d)[0x7f726d4018cd] ======= Memory map: ======== 564f9c9ca000-564f9de18000 r-xp 00000000 fd:00 2101454 /usr/sbin/mysqld 564f9e017000-564f9e143000 r--p 0144d000 fd:00 2101454 /usr/sbin/mysqld 564f9e143000-564f9e1fd000 rw-p 01579000 fd:00 2101454 /usr/sbin/mysqld 564f9e1fd000-564f9eaac000 rw-p 00000000 00:00 0 564fa00e3000-564fa4687000 rw-p 00000000 00:00 0 [heap] 7f71c8000000-7f71c8403000 rw-p 00000000 00:00 0 7f71c8403000-7f71cc000000 ---p 00000000 00:00 0 7f71cdffc000-7f71cdffd000 ---p 00000000 00:00 0 7f71cdffd000-7f71ce7fd000 rw-p 00000000 00:00 0 7f71ce7fd000-7f71ce7fe000 ---p 00000000 00:00 0 7f71ce7fe000-7f71ceffe000 rw-p 00000000 00:00 0 7f71ceffe000-7f71cefff000 ---p 00000000 00:00 0 7f71cefff000-7f71cf7ff000 rw-p 00000000 00:00 0 7f71cf7ff000-7f71cf800000 ---p 00000000 00:00 0 7f71cf800000-7f71d0000000 rw-p 00000000 00:00 0 7f71d0000000-7f71d0021000 rw-p 00000000 00:00 0 7f71d0021000-7f71d4000000 ---p 00000000 00:00 0 7f71d4000000-7f71d4021000 rw-p 00000000 00:00 0 7f71d4021000-7f71d8000000 ---p 00000000 00:00 0 7f71d8000000-7f71d8026000 rw-p 00000000 00:00 0 7f71d8026000-7f71dc000000 ---p 00000000 00:00 0 7f71dc000000-7f71dc02b000 rw-p 00000000 00:00 0 7f71dc02b000-7f71e0000000 ---p 00000000 00:00 0 7f71e0000000-7f71e0021000 rw-p 00000000 00:00 0 7f71e0021000-7f71e4000000 ---p 00000000 00:00 0 7f71e4000000-7f71e415c000 rw-p 00000000 00:00 0 7f71e415c000-7f71e8000000 ---p 00000000 00:00 0 7f71e8000000-7f71e8021000 rw-p 00000000 00:00 0 7f71e8021000-7f71ec000000 ---p 00000000 00:00 0 7f71ec000000-7f71edbe4000 rw-p 00000000 00:00 0 7f71edbe4000-7f71f0000000 ---p 00000000 00:00 0 7f71f0000000-7f71f036c000 rw-p 00000000 00:00 0 7f71f036c000-7f71f4000000 ---p 00000000 00:00 0 7f71f4000000-7f71f4021000 rw-p 00000000 00:00 0 7f71f4021000-7f71f8000000 ---p 00000000 00:00 0 7f71f8000000-7f71f84c8000 rw-p 00000000 00:00 0 7f71f84c8000-7f71fc000000 ---p 00000000 00:00 0 7f71fc1ca000-7f71fc1dc000 r-xp 00000000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f71fc1dc000-7f71fc3db000 ---p 00012000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f71fc3db000-7f71fc3de000 r--p 00011000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f71fc3de000-7f71fc3df000 rw-p 00014000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f71fc3df000-7f71fc3e5000 r-xp 00000000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f71fc3e5000-7f71fc5e4000 ---p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f71fc5e4000-7f71fc5e5000 r--p 00005000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f71fc5e5000-7f71fc5e6000 rw-p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f71fc5e6000-7f71fc5f2000 r-xp 00000000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f71fc5f2000-7f71fc7f1000 ---p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f71fc7f1000-7f71fc7f2000 r--p 0000b000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f71fc7f2000-7f71fc7f3000 rw-p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f71fc7f3000-7f71fc7f9000 rw-p 00000000 00:00 0 7f71fc7f9000-7f71fc7fa000 ---p 00000000 00:00 0 7f71fc7fa000-7f71fcffa000 rw-p 00000000 00:00 0 7f71fcffa000-7f71fcffb000 ---p 00000000 00:00 0 7f71fcffb000-7f71fd7fb000 rw-p 00000000 00:00 0 7f71fd7fb000-7f71fd7fc000 ---p 00000000 00:00 0 7f71fd7fc000-7f71fdffc000 rw-p 00000000 00:00 0 7f71fdffc000-7f71fdffd000 ---p 00000000 00:00 0 7f71fdffd000-7f71fe7fd000 rw-p 00000000 00:00 0 7f71fe7fd000-7f71fe7fe000 ---p 00000000 00:00 0 7f71fe7fe000-7f71feffe000 rw-p 00000000 00:00 0 7f71feffe000-7f71fefff000 ---p 00000000 00:00 0 7f71fefff000-7f71ff7ff000 rw-p 00000000 00:00 0 7f71ff7ff000-7f71ff800000 ---p 00000000 00:00 0 7f71ff800000-7f7200000000 rw-p 00000000 00:00 0 7f7200000000-7f720060d000 rw-p 00000000 00:00 0 7f720060d000-7f7204000000 ---p 00000000 00:00 0 7f7204151000-7f7204152000 ---p 00000000 00:00 0 7f7204152000-7f7204952000 rw-p 00000000 00:00 0 7f7204952000-7f7204953000 ---p 00000000 00:00 0 7f7204953000-7f7205358000 rw-p 00000000 00:00 0 7f7205358000-7f7205359000 ---p 00000000 00:00 0 7f7205359000-7f7205b59000 rw-p 00000000 00:00 0 7f7205b59000-7f7205b5a000 ---p 00000000 00:00 0 7f7205b5a000-7f720635a000 rw-p 00000000 00:00 0 7f720635a000-7f720635b000 ---p 00000000 00:00 0 7f720635b000-7f7206b5b000 rw-p 00000000 00:00 0 7f7206b5b000-7f7206b5c000 ---p 00000000 00:00 0 7f7206b5c000-7f720735c000 rw-p 00000000 00:00 0 7f720735c000-7f720735d000 ---p 00000000 00:00 0 7f720735d000-7f7207b5d000 rw-p 00000000 00:00 0 7f7207b5d000-7f7207b5e000 ---p 00000000 00:00 0 7f7207b5e000-7f720835e000 rw-p 00000000 00:00 0 7f720835e000-7f720835f000 ---p 00000000 00:00 0 7f720835f000-7f7208b5f000 rw-p 00000000 00:00 0 7f7208b5f000-7f7208b60000 ---p 00000000 00:00 0 7f7208b60000-7f7209360000 rw-p 00000000 00:00 0 7f7209360000-7f7209361000 ---p 00000000 00:00 0 7f7209361000-7f7209b61000 rw-p 00000000 00:00 0 7f7209b61000-7f7209b62000 ---p 00000000 00:00 0 7f7209b62000-7f720a362000 rw-p 00000000 00:00 0 7f720a362000-7f720a363000 ---p 00000000 00:00 0 7f720a363000-7f720ab63000 rw-p 00000000 00:00 0 7f720ab63000-7f720ab64000 ---p 00000000 00:00 0 7f720ab64000-7f720b364000 rw-p 00000000 00:00 0 7f720b364000-7f720b365000 ---p 00000000 00:00 0 7f720b365000-7f720bb65000 rw-p 00000000 00:00 0 7f720bb65000-7f720bb66000 ---p 00000000 00:00 0 7f720bb66000-7f720cb01000 rw-p 00000000 00:00 0 7f720cbe1000-7f720cbe2000 ---p 00000000 00:00 0 7f720cbe2000-7f720d3e2000 rw-p 00000000 00:00 0 7f720d3e2000-7f720d3e3000 ---p 00000000 00:00 0 7f720d3e3000-7f720dbe3000 rw-p 00000000 00:00 0 7f720dd02000-7f720dd03000 ---p 00000000 00:00 0 7f720dd03000-7f720dd4d000 rw-p 00000000 00:00 0 7f720dd4d000-7f720dd4e000 ---p 00000000 00:00 0 7f720dd4e000-7f720dd98000 rw-p 00000000 00:00 0 7f720dd98000-7f720dd99000 ---p 00000000 00:00 0 7f720dd99000-7f720dde3000 rw-p 00000000 00:00 0 7f720dde3000-7f720fde3000 rw-p 00000000 00:00 0 7f720fde3000-7f7210000000 rw-p 00000000 00:00 0 7f7210000000-7f7250000000 rw-p 00000000 00:00 0 7f7250000000-7f72505d0000 rw-p 00000000 00:00 0 7f72505d0000-7f7254000000 ---p 00000000 00:00 0 7f7254015000-7f7254016000 ---p 00000000 00:00 0 7f7254016000-7f7254060000 rw-p 00000000 00:00 0 7f7254060000-7f7254061000 ---p 00000000 00:00 0 7f7254061000-7f72540ab000 rw-p 00000000 00:00 0 7f72540ab000-7f72540ac000 ---p 00000000 00:00 0 7f72540ac000-7f72540f6000 rw-p 00000000 00:00 0 7f72540f6000-7f72540f7000 ---p 00000000 00:00 0 7f72540f7000-7f7254141000 rw-p 00000000 00:00 0 7f7254141000-7f7254142000 ---p 00000000 00:00 0 7f7254142000-7f725418c000 rw-p 00000000 00:00 0 7f725418c000-7f725418d000 ---p 00000000 00:00 0 7f725418d000-7f72554a3000 rw-p 00000000 00:00 0 7f72554a3000-7f72554a4000 ---p 00000000 00:00 0 7f72554a4000-7f7264000000 rw-p 00000000 00:00 0 7f7264000000-7f72643e5000 rw-p 00000000 00:00 0 7f72643e5000-7f7268000000 ---p 00000000 00:00 0 7f7268030000-7f7268031000 ---p 00000000 00:00 0 7f7268031000-7f726807b000 rw-p 00000000 00:00 0 7f726807b000-7f726807c000 ---p 00000000 00:00 0 7f726807c000-7f72680c6000 rw-p 00000000 00:00 0 7f72680c6000-7f72680c7000 ---p 00000000 00:00 0 7f72680c7000-7f7269fa0000 rw-p 00000000 00:00 0 7f7269fa0000-7f7269fa1000 ---p 00000000 00:00 0 7f7269fa1000-7f726a7a1000 rw-p 00000000 00:00 0 7f726a7a1000-7f726a7a4000 r-xp 00000000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f726a7a4000-7f726a9a3000 ---p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f726a9a3000-7f726a9a4000 r--p 00002000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f726a9a4000-7f726a9a5000 rw-p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f726a9a5000-7f726a9b3000 r-xp 00000000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f726a9b3000-7f726abb3000 ---p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f726abb3000-7f726abb4000 r--p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f726abb4000-7f726abb5000 rw-p 0000f000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f726abb5000-7f726abc4000 r-xp 00000000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f726abc4000-7f726adc3000 ---p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f726adc3000-7f726adc4000 r--p 0000e000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f726adc4000-7f726adc5000 rw-p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f726adc5000-7f726addc000 r-xp 00000000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f726addc000-7f726afdb000 ---p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f726afdb000-7f726afdc000 r--p 00016000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f726afdc000-7f726afdd000 rw-p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f726afdd000-7f726b03d000 r-xp 00000000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f726b03d000-7f726b23d000 ---p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f726b23d000-7f726b23e000 r--p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f726b23e000-7f726b23f000 rw-p 00061000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f726b23f000-7f726b243000 r-xp 00000000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f726b243000-7f726b442000 ---p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f726b442000-7f726b443000 r--p 00003000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f726b443000-7f726b444000 rw-p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f726b444000-7f726b475000 r-xp 00000000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f726b475000-7f726b674000 ---p 00031000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f726b674000-7f726b676000 r--p 00030000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f726b676000-7f726b677000 rw-p 00032000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f726b677000-7f726b67a000 r-xp 00000000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f726b67a000-7f726b879000 ---p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f726b879000-7f726b87a000 r--p 00002000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f726b87a000-7f726b87b000 rw-p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f726b87b000-7f726b954000 r-xp 00000000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f726b954000-7f726bb53000 ---p 000d9000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f726bb53000-7f726bb61000 r--p 000d8000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f726bb61000-7f726bb64000 rw-p 000e6000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f726bb64000-7f726bbae000 r-xp 00000000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f726bbae000-7f726bdae000 ---p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f726bdae000-7f726bdaf000 r--p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f726bdaf000-7f726bdb1000 rw-p 0004b000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f726bdb1000-7f726bdc6000 r-xp 00000000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f726bdc6000-7f726bfc5000 ---p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f726bfc5000-7f726bfc6000 r--p 00014000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f726bfc6000-7f726bfc7000 rw-p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f726bfc7000-7f726c015000 r-xp 00000000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f726c015000-7f726c215000 ---p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f726c215000-7f726c217000 r--p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f726c217000-7f726c218000 rw-p 00050000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f726c218000-7f726c22e000 r-xp 00000000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f726c22e000-7f726c42d000 ---p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f726c42d000-7f726c42e000 r--p 00015000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f726c42e000-7f726c42f000 rw-p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f726c42f000-7f726c431000 rw-p 00000000 00:00 0 7f726c431000-7f726c435000 r-xp 00000000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f726c435000-7f726c634000 ---p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f726c634000-7f726c635000 r--p 00003000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f726c635000-7f726c636000 rw-p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f726c636000-7f726c6b3000 r-xp 00000000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f726c6b3000-7f726c8b2000 ---p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f726c8b2000-7f726c8b3000 r--p 0007c000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f726c8b3000-7f726c8b6000 rw-p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f726c8b6000-7f726c8b7000 rw-p 00000000 00:00 0 7f726c8b7000-7f726c8cb000 r-xp 00000000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f726c8cb000-7f726caca000 ---p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f726caca000-7f726cacb000 r--p 00013000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f726cacb000-7f726cacc000 rw-p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f726cacc000-7f726caf0000 r-xp 00000000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f726caf0000-7f726ccef000 ---p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f726ccef000-7f726ccf0000 r--p 00023000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f726ccf0000-7f726ccf1000 rw-p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f726ccf1000-7f726ccf3000 rw-p 00000000 00:00 0 7f726ccf3000-7f726ccfa000 r-xp 00000000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f726ccfa000-7f726cef9000 ---p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f726cef9000-7f726cefa000 r--p 00006000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f726cefa000-7f726cefb000 rw-p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f726cefb000-7f726ceff000 r-xp 00000000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f726ceff000-7f726d0fe000 ---p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f726d0fe000-7f726d0ff000 r--p 00003000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f726d0ff000-7f726d100000 rw-p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f726d100000-7f726d102000 r-xp 00000000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f726d102000-7f726d301000 ---p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f726d301000-7f726d302000 r--p 00001000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f726d302000-7f726d303000 rw-p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f726d303000-7f726d4c6000 r-xp 00000000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f726d4c6000-7f726d6c6000 ---p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f726d6c6000-7f726d6ca000 r--p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f726d6ca000-7f726d6cc000 rw-p 001c7000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f726d6cc000-7f726d6d1000 rw-p 00000000 00:00 0 7f726d6d1000-7f726d7d2000 r-xp 00000000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f726d7d2000-7f726d9d1000 ---p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f726d9d1000-7f726d9d2000 r--p 00100000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f726d9d2000-7f726d9d3000 rw-p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f726d9d3000-7f726dabc000 r-xp 00000000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f726dabc000-7f726dcbb000 ---p 000e9000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f726dcbb000-7f726dcc3000 r--p 000e8000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f726dcc3000-7f726dcc5000 rw-p 000f0000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f726dcc5000-7f726dcda000 rw-p 00000000 00:00 0 7f726dcda000-7f726dcdc000 r-xp 00000000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f726dcdc000-7f726dedc000 ---p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f726dedc000-7f726dedd000 r--p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f726dedd000-7f726dede000 rw-p 00003000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f726dede000-7f726e114000 r-xp 00000000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f726e114000-7f726e314000 ---p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f726e314000-7f726e330000 r--p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f726e330000-7f726e33d000 rw-p 00252000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f726e33d000-7f726e341000 rw-p 00000000 00:00 0 7f726e341000-7f726e3a8000 r-xp 00000000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f726e3a8000-7f726e5a8000 ---p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f726e5a8000-7f726e5ac000 r--p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f726e5ac000-7f726e5b3000 rw-p 0006b000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f726e5b3000-7f726e5e2000 r-xp 00000000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f726e5e2000-7f726e7e2000 ---p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f726e7e2000-7f726e7e3000 r--p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f726e7e3000-7f726e7e4000 rw-p 00030000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f726e7e4000-7f726e7ec000 r-xp 00000000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f726e7ec000-7f726e9eb000 ---p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f726e9eb000-7f726e9ec000 r--p 00007000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f726e9ec000-7f726e9ed000 rw-p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f726e9ed000-7f726ea1b000 rw-p 00000000 00:00 0 7f726ea1b000-7f726ea30000 r-xp 00000000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f726ea30000-7f726ec2f000 ---p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f726ec2f000-7f726ec30000 r--p 00014000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f726ec30000-7f726ec31000 rw-p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f726ec31000-7f726ec32000 r-xp 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f726ec32000-7f726ee31000 ---p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f726ee31000-7f726ee32000 r--p 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f726ee32000-7f726ee33000 rw-p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f726ee33000-7f726ee58000 r-xp 00000000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f726ee58000-7f726f057000 ---p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f726f057000-7f726f058000 r--p 00024000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f726f058000-7f726f059000 rw-p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f726f059000-7f726f070000 r-xp 00000000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f726f070000-7f726f26f000 ---p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f726f26f000-7f726f270000 r--p 00016000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f726f270000-7f726f271000 rw-p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f726f271000-7f726f275000 rw-p 00000000 00:00 0 7f726f275000-7f726f297000 r-xp 00000000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f726f29b000-7f726f29c000 ---p 00000000 00:00 0 7f726f29c000-7f726f2e7000 rw-p 00000000 00:00 0 7f726f2e7000-7f726f2ea000 rw-s 00000000 00:0b 50315269 /[aio] (deleted) 7f726f2ea000-7f726f2ed000 rw-s 00000000 00:0b 50315268 /[aio] (deleted) 7f726f2ed000-7f726f2f0000 rw-s 00000000 00:0b 50315267 /[aio] (deleted) 7f726f2f0000-7f726f2f3000 rw-s 00000000 00:0b 50315266 /[aio] (deleted) 7f726f2f3000-7f726f2f6000 rw-s 00000000 00:0b 50315265 /[aio] (deleted) 7f726f2f6000-7f726f2f9000 rw-s 00000000 00:0b 50314240 /[aio] (deleted) 7f726f2f9000-7f726f2fc000 rw-s 00000000 00:0b 50314239 /[aio] (deleted) 7f726f2fc000-7f726f2ff000 rw-s 00000000 00:0b 50314238 /[aio] (deleted) 7f726f2ff000-7f726f48d000 rw-p 00000000 00:00 0 7f726f48d000-7f726f48e000 rw-s 00000000 00:0b 50315270 /[aio] (deleted) 7f726f48e000-7f726f491000 rw-s 00000000 00:0b 50314237 /[aio] (deleted) 7f726f491000-7f726f494000 rw-s 00000000 00:0b 50314236 /[aio] (deleted) 7f726f494000-7f726f495000 rw-s 00000000 00:0b 50314235 /[aio] (deleted) 7f726f495000-7f726f496000 rw-p 00000000 00:00 0 7f726f496000-7f726f497000 r--p 00021000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f726f497000-7f726f498000 rw-p 00022000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f726f498000-7f726f499000 rw-p 00000000 00:00 0 7fff26312000-7fff26333000 rw-p 00000000 00:00 0 [stack] 7fff263a3000-7fff263a5000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Fatal signal 6 while backtracing 2020-01-31 18:37:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:37:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:37:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:37:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:37:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:37:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:37:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:37:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:37:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:37:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135731914 2020-01-31 18:37:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000088', position 1103 2020-01-31 18:37:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:37:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:37:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:37:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:37:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:37:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:37:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135731923; transaction id 5898929 2020-01-31 18:37:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:37:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:37:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:37:04 0 [Note] Starting crash recovery... 2020-01-31 18:37:04 0 [Note] Crash recovery finished. 2020-01-31 18:37:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:37:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:37:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:37:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 235755 2020-01-31 18:37:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000055' at position 235755, relay log './dbsvc-528-relay-bin.000210' position: 154653 2020-01-31 18:37:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:37:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 235755 2020-01-31 18:37:06 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:37:06 2020-01-31 18:37:30 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:37:59 21 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000088, 734759) 2020-01-31 18:39:20 69 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:39:20 69 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:39:20 70 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:39:20 70 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:39:20 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=8 max_threads=502 thread_count=17 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fddd4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fddc4066cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5590c0f19f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5590c09b03af] sigaction.c:0(__restore_rt)[0x7fddde1bd630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5590c08de151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5590c0a1c545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5590c07d2ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5590c07c8d4b] /usr/sbin/mysqld(+0x6ca792)[0x5590c07d3792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5590c07fb6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5590c07fd65d] /usr/sbin/mysqld(+0x65e3f3)[0x5590c07673f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5590c0766cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5590c07fab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5590c07fd65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5590c07fd761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5590c07fe377] /usr/sbin/mysqld(+0x58fd84)[0x5590c0698d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5590c07a5b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5590c07bdb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5590c07bdcd2] /usr/sbin/mysqld(+0x6b55e7)[0x5590c07be5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5590c07be677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5590c07ad56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5590c07af07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5590c088aafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5590c088abdd] pthread_create.c:0(start_thread)[0x7fddde1b5ea5] /lib64/libc.so.6(clone+0x6d)[0x7fdddc5568cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fddd400fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 72 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:39:27 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:39:27 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:39:27 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:39:27 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:39:27 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:39:27 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:39:27 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:39:27 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:39:27 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:39:27 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135747255 2020-01-31 18:39:27 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000088', position 1103 2020-01-31 18:39:27 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:39:27 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:39:27 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:39:27 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:39:27 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:39:27 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:39:27 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135747264; transaction id 5898969 2020-01-31 18:39:27 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:39:27 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:39:27 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:39:27 0 [Note] Starting crash recovery... 2020-01-31 18:39:27 0 [Note] Crash recovery finished. 2020-01-31 18:39:27 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:39:27 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:39:27 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:39:27 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 259942 2020-01-31 18:39:27 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000055' at position 259942, relay log './dbsvc-528-relay-bin.000212' position: 16237 2020-01-31 18:39:27 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:39:27 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 259942 2020-01-31 18:39:27 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:39:27 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:39:27 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:39:27 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 18:39:28 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:39:28 2020-01-31 18:39:56 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:40:21 18 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000089, 311396) 2020-01-31 18:40:22 19 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:40:22 19 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:40:37 36 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 18:40:39 37 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 18:40:39 37 [Warning] Checking table: './ksarn/wp_options' 200131 18:41:12 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fbdb80009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fbe0818acf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55810e116f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55810dbad3af] sigaction.c:0(__restore_rt)[0x7fbe3309f630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55810dadb151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55810dc19545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55810d9cfccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55810d9c5d4b] /usr/sbin/mysqld(+0x6ca792)[0x55810d9d0792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55810d9f86da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55810d9fa65d] /usr/sbin/mysqld(+0x65e3f3)[0x55810d9643f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55810d963cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55810d9f7b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55810d9fa65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55810d9fa761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55810d9fb377] /usr/sbin/mysqld(+0x58fd84)[0x55810d895d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55810d9a2b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55810d9bab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55810d9bacd2] /usr/sbin/mysqld(+0x6b55e7)[0x55810d9bb5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55810d9bb677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55810d9aa56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55810d9ac07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55810da87afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55810da87bdd] pthread_create.c:0(start_thread)[0x7fbe33097ea5] /lib64/libc.so.6(clone+0x6d)[0x7fbe314388cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fbdb800fd90): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 56 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:41:18 0 [Note] mysqld: Aria engine: starting recovery recovered pages: 0% 10% 21% 32% 44% 56% 68% 82% 97% 100% (0.0 seconds); tables to flush: 9 8 7 6 5 4 3 2 1 0 (0.0 seconds); 2020-01-31 18:41:18 0 [Note] mysqld: Aria engine: recovery done 2020-01-31 18:41:18 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:41:18 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:41:18 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:41:18 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:41:18 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:41:18 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:41:18 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:41:18 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:41:18 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:41:18 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135755585 2020-01-31 18:41:18 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000088', position 1103 2020-01-31 18:41:18 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:41:18 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:41:18 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:41:18 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:41:18 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:41:18 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:41:18 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135755594; transaction id 5898996 2020-01-31 18:41:18 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:41:18 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/plugin' 2020-01-31 18:41:18 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:41:18 0 [Note] Starting crash recovery... 2020-01-31 18:41:18 0 [Note] Crash recovery finished. 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/servers' 2020-01-31 18:41:18 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/db' 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/proxies_priv' 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/roles_mapping' 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/global_priv' 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/time_zone_leap_second' 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/time_zone_name' 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/time_zone' 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/time_zone_transition_type' 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/time_zone_transition' 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/tables_priv' 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/columns_priv' 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/procs_priv' 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/func' 2020-01-31 18:41:18 0 [Note] Zerofilling moved table: './mysql/event' 2020-01-31 18:41:18 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:41:18 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:41:18 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 281776 2020-01-31 18:41:18 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:41:18 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000055' at position 281776, relay log './dbsvc-528-relay-bin.000214' position: 14851 2020-01-31 18:41:18 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 281776 2020-01-31 18:41:20 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:41:20 2020-01-31 18:41:46 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:41:56 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:41:56 16 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:41:56 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:41:56 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:41:56 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f067c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f06e0078cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55f59554af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55f594fe13af] sigaction.c:0(__restore_rt)[0x7f06fa556630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55f594f0f151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55f59504d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55f594e03ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55f594df9d4b] /usr/sbin/mysqld(+0x6ca792)[0x55f594e04792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55f594e2c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55f594e2e65d] /usr/sbin/mysqld(+0x65e3f3)[0x55f594d983f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55f594d97cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55f594e2bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55f594e2e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55f594e2e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55f594e2f377] /usr/sbin/mysqld(+0x58fd84)[0x55f594cc9d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55f594dd6b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55f594deeb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55f594deecd2] /usr/sbin/mysqld(+0x6b55e7)[0x55f594def5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55f594def677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55f594dde56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55f594de007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55f594ebbafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55f594ebbbdd] pthread_create.c:0(start_thread)[0x7f06fa54eea5] /lib64/libc.so.6(clone+0x6d)[0x7f06f88ef8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f067c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:42:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:42:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:42:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:42:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:42:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:42:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:42:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:42:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:42:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:42:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135774880 2020-01-31 18:42:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000088', position 1103 2020-01-31 18:42:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:42:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:42:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:42:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:42:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:42:03 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:42:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135774889; transaction id 5899087 2020-01-31 18:42:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:42:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:42:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:42:03 0 [Note] Starting crash recovery... 2020-01-31 18:42:03 0 [Note] Crash recovery finished. 2020-01-31 18:42:03 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:42:03 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:42:03 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:42:03 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 403167 2020-01-31 18:42:03 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000055' at position 403167, relay log './dbsvc-528-relay-bin.000216' position: 79509 2020-01-31 18:42:03 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:42:03 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 403167 2020-01-31 18:42:04 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:42:04 2020-01-31 18:42:12 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000090, 357416) 2020-01-31 18:42:13 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:42:13 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:42:13 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:42:13 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 18:42:31 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:43:15 29 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 18:43:15 29 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 18:43:24 34 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 18:43:24 34 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 18:43:24 35 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 18:43:24 35 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 18:44:32 70 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 18:44:32 70 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 18:44:57 81 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:44:57 81 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:44:57 82 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:44:57 82 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:44:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f481c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f488494fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x556033bdef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5560336753af] sigaction.c:0(__restore_rt)[0x7f48a1a73630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5560335a3151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5560336e1545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x556033497ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55603348dd4b] /usr/sbin/mysqld(+0x6ca792)[0x556033498792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5560334c06da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5560334c265d] /usr/sbin/mysqld(+0x65e3f3)[0x55603342c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55603342bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5560334bfb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5560334c265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5560334c2761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5560334c3377] /usr/sbin/mysqld(+0x58fd84)[0x55603335dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55603346ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x556033482b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x556033482cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5560334835e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x556033483677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55603347256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55603347407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55603354fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55603354fbdd] pthread_create.c:0(start_thread)[0x7f48a1a6bea5] /lib64/libc.so.6(clone+0x6d)[0x7f489fe0c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f481c012150): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 83 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:45:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:45:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:45:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:45:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:45:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:45:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:45:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:45:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:45:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:45:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135789973 2020-01-31 18:45:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000088', position 1103 2020-01-31 18:45:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:45:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:45:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:45:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:45:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:45:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:45:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135789982; transaction id 5899122 2020-01-31 18:45:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:45:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:45:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:45:04 0 [Note] Starting crash recovery... 2020-01-31 18:45:04 0 [Note] Crash recovery finished. 2020-01-31 18:45:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:45:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:45:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:45:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 428031 2020-01-31 18:45:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000055' at position 428031, relay log './dbsvc-528-relay-bin.000218' position: 16803 2020-01-31 18:45:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:45:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 428031 2020-01-31 18:45:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:45:05 2020-01-31 18:45:05 11 [ERROR] mysqld: Table './openrecords/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 18:45:05 11 [Warning] Checking table: './openrecords/wp_gf_form_view' 2020-01-31 18:45:05 11 [Warning] openrecords.wp_gf_form_view: 1 client is using or hasn't closed the table properly 2020-01-31 18:45:05 11 [Note] openrecords.wp_gf_form_view: Table is fixed 2020-01-31 18:45:22 11 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 18:45:22 11 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 18:45:22 11 [Warning] wpsem.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:45:22 11 [Note] wpsem.wp_options: Table is fixed 2020-01-31 18:45:29 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:45:29 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:45:29 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:45:29 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 18:45:32 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:45:39 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:45:39 16 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:45:40 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:45:40 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:45:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fba5c006ab8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fbac0138cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56466e68af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56466e1213af] sigaction.c:0(__restore_rt)[0x7fbadc4ca630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56466e04f151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56466e18d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56466df43ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56466df39d4b] /usr/sbin/mysqld(+0x6ca792)[0x56466df44792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56466df6c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56466df6e65d] /usr/sbin/mysqld(+0x65e3f3)[0x56466ded83f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56466ded7cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56466df6bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56466df6e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56466df6e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56466df6f377] /usr/sbin/mysqld(+0x58fd84)[0x56466de09d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56466df16b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56466df2eb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56466df2ecd2] /usr/sbin/mysqld(+0x6b55e7)[0x56466df2f5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56466df2f677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56466df1e56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56466df2007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56466dffbafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56466dffbbdd] pthread_create.c:0(start_thread)[0x7fbadc4c2ea5] /lib64/libc.so.6(clone+0x6d)[0x7fbada8638cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fba5c010d60): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:45:46 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:45:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:45:46 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:45:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:45:46 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:45:46 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:45:46 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:45:46 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:45:46 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:45:46 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135800245 2020-01-31 18:45:46 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000088', position 1103 2020-01-31 18:45:46 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:45:46 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:45:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:45:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:45:46 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:45:46 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:45:46 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135800254; transaction id 5899158 2020-01-31 18:45:46 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:45:46 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:45:46 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:45:46 0 [Note] Starting crash recovery... 2020-01-31 18:45:46 0 [Note] Crash recovery finished. 2020-01-31 18:45:46 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:45:46 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:45:46 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:45:46 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 434220 2020-01-31 18:45:46 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000055' at position 434220, relay log './dbsvc-528-relay-bin.000220' position: 4567 2020-01-31 18:45:46 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:45:46 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 434220 2020-01-31 18:45:47 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:45:47 2020-01-31 18:45:48 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:45:48 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:45:48 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:45:48 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 18:45:57 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 18:45:57 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 18:45:57 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:45:57 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 18:45:58 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000092, 246004) 2020-01-31 18:46:12 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:46:52 30 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 18:46:52 30 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 18:46:56 33 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:46:56 33 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:46:56 34 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:46:56 34 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:46:56 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fb3ec000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fb405ae3cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56355192af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5635513c13af] sigaction.c:0(__restore_rt)[0x7fb466c6e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5635512ef151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56355142d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5635511e3ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5635511d9d4b] /usr/sbin/mysqld(+0x6ca792)[0x5635511e4792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56355120c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56355120e65d] /usr/sbin/mysqld(+0x65e3f3)[0x5635511783f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x563551177cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56355120bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56355120e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56355120e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56355120f377] /usr/sbin/mysqld(+0x58fd84)[0x5635510a9d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5635511b6b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5635511ceb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5635511cecd2] /usr/sbin/mysqld(+0x6b55e7)[0x5635511cf5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5635511cf677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5635511be56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5635511c007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56355129bafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56355129bbdd] pthread_create.c:0(start_thread)[0x7fb466c66ea5] /lib64/libc.so.6(clone+0x6d)[0x7fb4650078cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fb3ec010a30): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 35 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:47:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:47:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:47:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:47:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:47:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:47:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:47:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:47:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:47:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:47:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135814514 2020-01-31 18:47:03 0 [Note] InnoDB: Starting final batch to recover 12 pages from redo log. 2020-01-31 18:47:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000094', position 131406 2020-01-31 18:47:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:47:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:47:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:47:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:47:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:47:03 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:47:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135839789; transaction id 5899245 2020-01-31 18:47:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:47:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:47:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:47:04 0 [Note] Starting crash recovery... 2020-01-31 18:47:04 0 [Note] Crash recovery finished. 2020-01-31 18:47:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:47:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:47:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:47:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 627034 2020-01-31 18:47:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:47:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 627034 2020-01-31 18:47:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000055' at position 627034, relay log './dbsvc-528-relay-bin.000222' position: 126238 2020-01-31 18:47:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:47:05 2020-01-31 18:47:10 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:47:10 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:47:10 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:47:10 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 18:47:32 19 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:47:57 28 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000094, 132514) 2020-01-31 18:48:27 34 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:48:27 34 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:48:47 40 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 18:48:47 40 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 18:48:48 41 [ERROR] mysqld: Table './openrecords/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 18:48:48 41 [Warning] Checking table: './openrecords/wp_gf_form_view' 2020-01-31 18:49:05 51 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 18:49:05 51 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 18:50:07 83 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:50:07 83 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 18:51:40 114 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 18:51:40 114 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 18:51:56 136 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:51:56 136 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:51:56 137 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:51:56 137 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:51:56 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f7a94000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f7aa9b2ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5563a45cff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5563a40663af] sigaction.c:0(__restore_rt)[0x7f7b0ad0c630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5563a3f94151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5563a40d2545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5563a3e88ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5563a3e7ed4b] /usr/sbin/mysqld(+0x6ca792)[0x5563a3e89792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5563a3eb16da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5563a3eb365d] /usr/sbin/mysqld(+0x65e3f3)[0x5563a3e1d3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5563a3e1ccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5563a3eb0b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5563a3eb365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5563a3eb3761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5563a3eb4377] /usr/sbin/mysqld(+0x58fd84)[0x5563a3d4ed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5563a3e5bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5563a3e73b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5563a3e73cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5563a3e745e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5563a3e74677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5563a3e6356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5563a3e6507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5563a3f40afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5563a3f40bdd] pthread_create.c:0(start_thread)[0x7f7b0ad04ea5] /lib64/libc.so.6(clone+0x6d)[0x7f7b090a58cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f7a9400ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 138 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:52:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:52:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:52:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:52:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:52:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:52:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:52:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:52:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:52:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:52:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135855701 2020-01-31 18:52:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000094', position 131406 2020-01-31 18:52:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:52:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:52:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:52:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:52:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:52:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135855710; transaction id 5899296 2020-01-31 18:52:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:52:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:52:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:52:03 0 [Note] Starting crash recovery... 2020-01-31 18:52:03 0 [Note] Crash recovery finished. 2020-01-31 18:52:03 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:52:03 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:52:03 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:52:03 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 673408 2020-01-31 18:52:03 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000055' at position 673408, relay log './dbsvc-528-relay-bin.000224' position: 30750 2020-01-31 18:52:03 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:52:03 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 673408 2020-01-31 18:52:04 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:52:04 2020-01-31 18:52:28 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:52:28 14 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:52:28 14 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:52:29 14 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:52:29 14 [Warning] Checking table: './wpms/wp_usermeta' 2020-01-31 18:52:29 16 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:52:29 16 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:52:30 17 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:52:30 17 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 18:52:31 18 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 18:52:31 18 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 18:52:33 21 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 18:52:33 21 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 18:52:57 29 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000095, 718930) 2020-01-31 18:53:01 31 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 18:53:01 31 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 18:53:15 38 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 18:53:15 38 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 18:53:23 42 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 18:53:23 42 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 18:53:28 49 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 18:53:28 49 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 18:53:45 58 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:53:45 58 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:53:45 59 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:53:45 59 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:53:45 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f1994000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f19f0078cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55dd177e9f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55dd172803af] sigaction.c:0(__restore_rt)[0x7f1a0a5f0630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55dd171ae151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55dd172ec545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55dd170a2ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55dd17098d4b] /usr/sbin/mysqld(+0x6ca792)[0x55dd170a3792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55dd170cb6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55dd170cd65d] /usr/sbin/mysqld(+0x65e3f3)[0x55dd170373f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55dd17036cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55dd170cab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55dd170cd65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55dd170cd761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55dd170ce377] /usr/sbin/mysqld(+0x58fd84)[0x55dd16f68d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55dd17075b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55dd1708db16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55dd1708dcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55dd1708e5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55dd1708e677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55dd1707d56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55dd1707f07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55dd1715aafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55dd1715abdd] pthread_create.c:0(start_thread)[0x7f1a0a5e8ea5] /lib64/libc.so.6(clone+0x6d)[0x7f1a089898cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f199400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 60 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:53:51 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:53:51 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:53:51 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:53:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:53:51 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:53:51 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:53:51 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:53:51 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:53:51 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:53:51 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135860597 2020-01-31 18:53:51 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000094', position 131406 2020-01-31 18:53:51 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:53:51 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:53:51 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:53:51 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:53:51 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:53:51 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:53:51 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135860606; transaction id 5899307 2020-01-31 18:53:51 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:53:51 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:53:51 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:53:51 0 [Note] Starting crash recovery... 2020-01-31 18:53:51 0 [Note] Crash recovery finished. 2020-01-31 18:53:51 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:53:51 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:53:51 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:53:51 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 673408 2020-01-31 18:53:51 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000055' at position 673408, relay log './dbsvc-528-relay-bin.000226' position: 559 2020-01-31 18:53:51 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:53:51 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 673408 2020-01-31 18:53:51 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 18:53:51 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 18:53:51 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:53:51 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 18:53:52 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:53:52 2020-01-31 18:54:00 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:54:00 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:54:00 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:54:00 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 18:54:18 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:54:19 11 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 18:54:19 11 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 18:54:19 11 [Warning] wpstart.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:54:19 11 [Note] wpstart.wp_options: Table is fixed 2020-01-31 18:54:27 17 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 18:54:27 17 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 18:54:28 20 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:54:28 20 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:54:30 22 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 18:54:30 22 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 18:54:37 27 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:54:37 27 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:54:37 28 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:54:37 28 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:54:37 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fb9b8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fba1cbbacf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x559630443f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55962feda3af] sigaction.c:0(__restore_rt)[0x7fba37d2a630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55962fe08151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55962ff46545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55962fcfcccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55962fcf2d4b] /usr/sbin/mysqld(+0x6ca792)[0x55962fcfd792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55962fd256da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55962fd2765d] /usr/sbin/mysqld(+0x65e3f3)[0x55962fc913f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55962fc90cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55962fd24b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55962fd2765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55962fd27761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55962fd28377] /usr/sbin/mysqld(+0x58fd84)[0x55962fbc2d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55962fccfb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55962fce7b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55962fce7cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55962fce85e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55962fce8677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55962fcd756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55962fcd907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55962fdb4afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55962fdb4bdd] pthread_create.c:0(start_thread)[0x7fba37d22ea5] /lib64/libc.so.6(clone+0x6d)[0x7fba360c38cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fb9b800feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 29 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:54:43 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:54:43 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:54:43 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:54:43 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:54:43 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:54:43 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:54:43 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:54:43 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:54:43 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:54:43 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135874824 2020-01-31 18:54:43 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000094', position 131406 2020-01-31 18:54:43 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:54:43 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:54:43 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:54:43 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:54:43 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:54:43 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:54:43 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135874833; transaction id 5899369 2020-01-31 18:54:43 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:54:43 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:54:43 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:54:43 0 [Note] Starting crash recovery... 2020-01-31 18:54:43 0 [Note] Crash recovery finished. 2020-01-31 18:54:43 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:54:43 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:54:43 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:54:43 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 807438 2020-01-31 18:54:43 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:54:43 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000055' at position 807438, relay log './dbsvc-528-relay-bin.000228' position: 88274 2020-01-31 18:54:43 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 807438 2020-01-31 18:54:44 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:54:44 2020-01-31 18:54:46 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000096, 198399) 2020-01-31 18:55:09 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:55:13 14 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:55:13 14 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:55:35 24 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 18:55:35 24 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 18:55:47 30 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:55:47 30 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:56:12 40 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 18:56:12 40 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 18:56:13 41 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 18:56:13 41 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 18:56:22 46 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 18:56:22 46 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 18:56:34 53 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 18:56:34 53 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 18:56:56 62 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 18:56:56 62 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 18:56:56 63 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 18:56:56 63 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 18:56:56 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f23ac0009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f23c9de1cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5557b2992f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5557b24293af] sigaction.c:0(__restore_rt)[0x7f242afd7630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5557b2357151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5557b2495545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5557b224bccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5557b2241d4b] /usr/sbin/mysqld(+0x6ca792)[0x5557b224c792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5557b22746da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5557b227665d] /usr/sbin/mysqld(+0x65e3f3)[0x5557b21e03f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5557b21dfcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5557b2273b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5557b227665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5557b2276761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5557b2277377] /usr/sbin/mysqld(+0x58fd84)[0x5557b2111d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5557b221eb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5557b2236b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5557b2236cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5557b22375e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5557b2237677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5557b222656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5557b222807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5557b2303afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5557b2303bdd] pthread_create.c:0(start_thread)[0x7f242afcfea5] /lib64/libc.so.6(clone+0x6d)[0x7f24293708cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f23ac00fe20): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 64 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 18:57:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 18:57:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 18:57:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 18:57:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 18:57:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 18:57:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 18:57:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 18:57:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 18:57:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 18:57:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135881006 2020-01-31 18:57:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000094', position 131406 2020-01-31 18:57:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 18:57:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 18:57:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 18:57:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 18:57:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 18:57:03 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 18:57:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135881015; transaction id 5899380 2020-01-31 18:57:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 18:57:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 18:57:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 18:57:03 0 [Note] Starting crash recovery... 2020-01-31 18:57:03 0 [Note] Crash recovery finished. 2020-01-31 18:57:03 0 [Note] Server socket created on IP: '::'. 2020-01-31 18:57:03 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 18:57:03 0 [Note] Added new Master_info '' to hash table 2020-01-31 18:57:03 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 807438 2020-01-31 18:57:03 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000055' at position 807438, relay log './dbsvc-528-relay-bin.000230' position: 559 2020-01-31 18:57:03 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 18:57:03 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 807438 2020-01-31 18:57:04 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 18:57:04 2020-01-31 18:57:10 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 18:57:10 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 18:57:10 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:57:10 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 18:57:26 11 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 18:57:26 11 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 18:57:26 11 [Warning] wpcomm303.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 18:57:26 11 [Note] wpcomm303.wp_options: Table is fixed 2020-01-31 18:57:29 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 18:57:37 15 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 18:57:37 15 [Warning] Checking table: './wpms/wp_options' 2020-01-31 18:57:57 21 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000098, 157876) 2020-01-31 18:58:20 27 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 18:58:20 27 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 18:59:29 54 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 18:59:29 54 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 18:59:32 57 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 18:59:32 57 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 18:59:41 63 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 18:59:41 63 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 18:59:53 71 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 18:59:53 71 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 19:01:44 118 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:01:44 118 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:01:44 119 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:01:44 119 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:01:45 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0924000e78 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f09ac05bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c27eb8ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c27e6263af] sigaction.c:0(__restore_rt)[0x7f09c957c630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c27e554151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c27e692545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c27e448ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c27e43ed4b] /usr/sbin/mysqld(+0x6ca792)[0x55c27e449792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c27e4716da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c27e47365d] /usr/sbin/mysqld(+0x65e3f3)[0x55c27e3dd3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c27e3dccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c27e470b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c27e47365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c27e473761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c27e474377] /usr/sbin/mysqld(+0x58fd84)[0x55c27e30ed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c27e41bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c27e433b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c27e433cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c27e4345e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c27e434677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c27e42356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c27e42507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c27e500afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c27e500bdd] pthread_create.c:0(start_thread)[0x7f09c9574ea5] /lib64/libc.so.6(clone+0x6d)[0x7f09c79158cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f092400f140): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 121 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:01:50 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:01:50 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:01:50 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:01:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:01:50 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:01:50 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:01:50 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:01:50 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:01:50 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:01:50 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135916632 2020-01-31 19:01:50 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000099', position 813509 2020-01-31 19:01:51 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:01:51 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:01:51 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:01:51 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:01:51 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:01:51 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:01:51 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135916641; transaction id 5899459 2020-01-31 19:01:51 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:01:51 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:01:51 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:01:51 0 [Note] Starting crash recovery... 2020-01-31 19:01:51 0 [Note] Crash recovery finished. 2020-01-31 19:01:51 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:01:51 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:01:51 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:01:51 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 854342 2020-01-31 19:01:51 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000055' at position 854342, relay log './dbsvc-528-relay-bin.000232' position: 31113 2020-01-31 19:01:51 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:01:51 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 854342 2020-01-31 19:01:52 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:01:52 2020-01-31 19:01:56 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:01:56 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:01:56 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 19:01:56 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 19:01:56 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:01:56 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 19:01:56 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 19:01:56 11 [Note] ojsdata.scheduled_tasks: Table is fixed 2020-01-31 19:01:57 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 19:01:57 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000055' at position 855450 2020-01-31 19:01:57 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 19:02:04 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000099, 845866) 2020-01-31 19:02:20 18 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:02:23 20 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 19:02:23 20 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 19:02:25 22 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 19:02:25 22 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 19:02:27 23 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:02:27 23 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:02:28 24 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 19:02:28 24 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 19:02:56 38 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 19:02:56 38 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-01-31 19:02:57 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000055' at position 855450 2020-01-31 19:03:00 40 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 19:03:00 40 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 19:03:05 42 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:03:05 42 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:03:06 44 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 19:03:06 44 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 19:03:11 49 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 19:03:11 49 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 19:03:14 51 [Note] Zerofilling moved table: './mysql/proc' 2020-01-31 19:03:26 62 [ERROR] mysqld: Table './openrecords/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 19:03:26 62 [Warning] Checking table: './openrecords/wp_gf_form_view' 2020-01-31 19:03:27 66 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 19:03:27 66 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 19:03:28 68 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 19:03:28 68 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 19:04:33 98 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 19:04:33 98 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 19:05:34 116 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 19:05:34 116 [Warning] Checking table: './ulrich/wp_options' 200131 19:06:13 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f85c4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f863156acf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b5c3511f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55b5c2fa83af] sigaction.c:0(__restore_rt)[0x7f864c6db630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55b5c2ed6151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55b5c3014545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55b5c2dcaccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55b5c2dc0d4b] /usr/sbin/mysqld(+0x6ca792)[0x55b5c2dcb792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55b5c2df36da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b5c2df565d] /usr/sbin/mysqld(+0x65e3f3)[0x55b5c2d5f3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55b5c2d5ecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55b5c2df2b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b5c2df565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55b5c2df5761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55b5c2df6377] /usr/sbin/mysqld(+0x58fd84)[0x55b5c2c90d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55b5c2d9db6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55b5c2db5b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55b5c2db5cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55b5c2db65e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55b5c2db6677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55b5c2da556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55b5c2da707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55b5c2e82afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55b5c2e82bdd] pthread_create.c:0(start_thread)[0x7f864c6d3ea5] /lib64/libc.so.6(clone+0x6d)[0x7f864aa748cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f85c400feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 136 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:06:20 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:06:20 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:06:20 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:06:20 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:06:20 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:06:20 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:06:20 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:06:20 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:06:20 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:06:20 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135925391 2020-01-31 19:06:20 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000099', position 813509 2020-01-31 19:06:20 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:06:20 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:06:20 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:06:20 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:06:20 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:06:20 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:06:21 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135925400; transaction id 5899483 2020-01-31 19:06:21 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:06:21 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:06:21 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:06:21 0 [Note] Starting crash recovery... 2020-01-31 19:06:21 0 [Note] Crash recovery finished. 2020-01-31 19:06:21 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:06:21 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:06:21 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:06:21 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000056' at position 346 2020-01-31 19:06:21 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000056' at position 346, relay log './dbsvc-528-relay-bin.000236' position: 649 2020-01-31 19:06:21 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:06:21 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 19:06:22 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:06:22 2020-01-31 19:06:24 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000100, 966648) 2020-01-31 19:06:45 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:06:47 14 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:06:47 14 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:06:48 15 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:06:48 15 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:06:48 16 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 19:06:48 16 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 19:06:50 19 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 19:06:50 19 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 19:06:50 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:06:50 20 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:06:50 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:06:50 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:06:50 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fd428000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fd447d01cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b2ace76f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55b2ac90d3af] sigaction.c:0(__restore_rt)[0x7fd4aa0a6630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55b2ac83b151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55b2ac979545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55b2ac72fccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55b2ac725d4b] /usr/sbin/mysqld(+0x6ca792)[0x55b2ac730792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55b2ac7586da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b2ac75a65d] /usr/sbin/mysqld(+0x65e3f3)[0x55b2ac6c43f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55b2ac6c3cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55b2ac757b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b2ac75a65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55b2ac75a761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55b2ac75b377] /usr/sbin/mysqld(+0x58fd84)[0x55b2ac5f5d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55b2ac702b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55b2ac71ab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55b2ac71acd2] /usr/sbin/mysqld(+0x6b55e7)[0x55b2ac71b5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55b2ac71b677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55b2ac70a56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55b2ac70c07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55b2ac7e7afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55b2ac7e7bdd] pthread_create.c:0(start_thread)[0x7fd4aa09eea5] /lib64/libc.so.6(clone+0x6d)[0x7fd4a843f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fd42800fe70): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:06:57 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:06:57 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:06:57 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:06:57 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:06:57 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:06:57 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:06:57 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:06:57 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:06:57 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:06:57 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135929179 2020-01-31 19:06:57 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000099', position 813509 2020-01-31 19:06:57 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:06:57 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:06:57 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:06:57 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:06:57 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:06:57 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:06:57 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135929188; transaction id 5899492 2020-01-31 19:06:57 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:06:57 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:06:57 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:06:57 0 [Note] Starting crash recovery... 2020-01-31 19:06:57 0 [Note] Crash recovery finished. 2020-01-31 19:06:57 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:06:57 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:06:57 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:06:57 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000056' at position 346 2020-01-31 19:06:57 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000056' at position 346, relay log './dbsvc-528-relay-bin.000237' position: 4 2020-01-31 19:06:57 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:06:57 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000056' at position 346 2020-01-31 19:06:57 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:06:57 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:06:57 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 19:06:57 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 19:06:57 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:06:57 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 19:06:57 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 19:06:57 11 [Note] ojsdata.scheduled_tasks: Table is fixed 2020-01-31 19:06:57 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 19:06:57 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000057' at position 1454 2020-01-31 19:06:57 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 19:06:58 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:06:58 2020-01-31 19:07:03 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000101, 29603) 2020-01-31 19:07:26 18 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:07:26 20 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 19:07:26 20 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 19:07:26 21 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 19:07:26 21 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 19:07:31 23 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 19:07:31 23 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 19:07:31 24 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:07:31 24 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:07:33 26 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 19:07:33 26 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 19:07:57 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000057' at position 1454 2020-01-31 19:08:33 55 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:08:33 55 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:08:34 55 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 19:08:34 55 [Warning] Checking table: './wpms/wp_usermeta' 2020-01-31 19:10:11 95 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 19:10:11 95 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 19:10:21 99 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 19:10:21 99 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 19:11:15 124 [ERROR] mysqld: Table './openrecords/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 19:11:15 124 [Warning] Checking table: './openrecords/wp_gf_form_view' 200131 19:11:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fcb88000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fcb47fb3cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x562ff2f5cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x562ff29f33af] sigaction.c:0(__restore_rt)[0x7fcba927d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x562ff2921151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x562ff2a5f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x562ff2815ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x562ff280bd4b] /usr/sbin/mysqld(+0x6ca792)[0x562ff2816792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x562ff283e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x562ff284065d] /usr/sbin/mysqld(+0x65e3f3)[0x562ff27aa3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x562ff27a9cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x562ff283db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x562ff284065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x562ff2840761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x562ff2841377] /usr/sbin/mysqld(+0x58fd84)[0x562ff26dbd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x562ff27e8b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x562ff2800b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x562ff2800cd2] /usr/sbin/mysqld(+0x6b55e7)[0x562ff28015e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x562ff2801677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x562ff27f056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x562ff27f207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x562ff28cdafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x562ff28cdbdd] pthread_create.c:0(start_thread)[0x7fcba9275ea5] /lib64/libc.so.6(clone+0x6d)[0x7fcba76168cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fcb8800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 146 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:12:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:12:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:12:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:12:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:12:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:12:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:12:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:12:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:12:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:12:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135935904 2020-01-31 19:12:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000099', position 813509 2020-01-31 19:12:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:12:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:12:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:12:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:12:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:12:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:12:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135935913; transaction id 5899518 2020-01-31 19:12:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:12:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:12:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:12:04 0 [Note] Starting crash recovery... 2020-01-31 19:12:04 0 [Note] Crash recovery finished. 2020-01-31 19:12:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:12:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:12:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:12:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000058' at position 346 2020-01-31 19:12:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 346, relay log './dbsvc-528-relay-bin.000242' position: 649 2020-01-31 19:12:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:12:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000058' at position 346 2020-01-31 19:12:06 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:12:06 2020-01-31 19:12:31 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:12:58 18 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000102, 429571) 2020-01-31 19:13:00 19 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 19:13:00 19 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 19:13:04 22 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:13:04 22 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:13:05 23 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 19:13:05 23 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 19:13:15 28 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 19:13:15 28 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 19:13:21 32 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 19:13:21 32 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 19:13:28 40 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 19:13:28 40 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 19:14:32 71 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 19:14:32 71 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 19:14:35 73 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:14:35 73 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:14:36 73 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 19:14:36 73 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 19:15:34 103 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 19:15:34 103 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 19:16:33 125 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 19:16:33 125 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 19:16:57 141 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:16:57 141 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:16:58 142 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:16:58 142 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:16:58 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe618000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe62cffccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5652e307df2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5652e2b143af] sigaction.c:0(__restore_rt)[0x7fe690218630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5652e2a42151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5652e2b80545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5652e2936ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5652e292cd4b] /usr/sbin/mysqld(+0x6ca792)[0x5652e2937792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5652e295f6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5652e296165d] /usr/sbin/mysqld(+0x65e3f3)[0x5652e28cb3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5652e28cacb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5652e295eb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5652e296165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5652e2961761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5652e2962377] /usr/sbin/mysqld(+0x58fd84)[0x5652e27fcd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5652e2909b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5652e2921b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5652e2921cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5652e29225e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5652e2922677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5652e291156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5652e291307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5652e29eeafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5652e29eebdd] pthread_create.c:0(start_thread)[0x7fe690210ea5] /lib64/libc.so.6(clone+0x6d)[0x7fe68e5b18cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe61800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 143 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:17:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:17:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:17:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:17:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:17:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:17:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:17:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:17:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:17:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:17:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135947054 2020-01-31 19:17:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000099', position 813509 2020-01-31 19:17:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:17:05 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:17:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:17:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:17:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:17:05 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:17:05 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135947063; transaction id 5899552 2020-01-31 19:17:05 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:17:05 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:17:05 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:17:05 0 [Note] Starting crash recovery... 2020-01-31 19:17:05 0 [Note] Crash recovery finished. 2020-01-31 19:17:05 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:17:05 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:17:05 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:17:05 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000058' at position 41268 2020-01-31 19:17:05 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 41268, relay log './dbsvc-528-relay-bin.000244' position: 27255 2020-01-31 19:17:05 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:17:05 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000058' at position 41268 2020-01-31 19:17:06 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:17:06 2020-01-31 19:17:16 11 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 19:17:16 11 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 19:17:16 11 [Warning] wpstart.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 19:17:16 11 [Note] wpstart.wp_options: Table is fixed 2020-01-31 19:17:25 11 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 19:17:25 11 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 19:17:25 11 [Warning] wpcomm303.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 19:17:25 11 [Note] wpcomm303.wp_options: Table is fixed 2020-01-31 19:17:32 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:17:45 20 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:17:45 20 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:17:46 21 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 19:17:46 21 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 19:17:59 25 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 19:17:59 25 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 19:17:59 26 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000103, 572159) 2020-01-31 19:18:59 45 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 19:18:59 45 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 19:19:44 76 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:19:44 76 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:19:50 81 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 19:19:50 81 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 19:21:57 142 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:21:57 142 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:21:57 143 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:21:57 143 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:21:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f4960000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4964064cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5563371bff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x556336c563af] sigaction.c:0(__restore_rt)[0x7f4973d41630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x556336b84151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x556336cc2545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x556336a78ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x556336a6ed4b] /usr/sbin/mysqld(+0x6ca792)[0x556336a79792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x556336aa16da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556336aa365d] /usr/sbin/mysqld(+0x65e3f3)[0x556336a0d3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x556336a0ccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x556336aa0b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556336aa365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x556336aa3761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x556336aa4377] /usr/sbin/mysqld(+0x58fd84)[0x55633693ed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x556336a4bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x556336a63b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x556336a63cd2] /usr/sbin/mysqld(+0x6b55e7)[0x556336a645e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x556336a64677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x556336a5356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x556336a5507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x556336b30afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x556336b30bdd] pthread_create.c:0(start_thread)[0x7f4973d39ea5] /lib64/libc.so.6(clone+0x6d)[0x7f49720da8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f496000fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 144 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:22:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:22:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:22:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:22:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:22:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:22:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:22:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:22:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:22:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:22:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135976762 2020-01-31 19:22:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000099', position 813509 2020-01-31 19:22:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:22:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:22:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:22:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:22:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:22:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:22:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135976771; transaction id 5899635 2020-01-31 19:22:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:22:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:22:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:22:04 0 [Note] Starting crash recovery... 2020-01-31 19:22:04 0 [Note] Crash recovery finished. 2020-01-31 19:22:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:22:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:22:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:22:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000058' at position 244683 2020-01-31 19:22:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 244683, relay log './dbsvc-528-relay-bin.000246' position: 133594 2020-01-31 19:22:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:22:05 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000058' at position 244683 2020-01-31 19:22:06 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:22:06 2020-01-31 19:22:33 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:22:33 13 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:22:33 13 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:22:34 14 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 19:22:34 14 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 19:22:34 15 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 19:22:34 15 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 19:22:35 16 [ERROR] mysqld: Table './openrecords/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 19:22:35 16 [Warning] Checking table: './openrecords/wp_gf_form_view' 2020-01-31 19:22:59 26 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000104, 745028) 2020-01-31 19:23:00 27 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 19:23:00 27 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 19:23:15 35 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 19:23:15 35 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 19:23:24 40 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 19:23:24 40 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 19:23:28 45 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 19:23:28 45 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 19:24:27 66 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 19:24:27 66 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 19:24:32 70 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 19:24:32 70 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 19:24:37 72 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:24:37 72 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:25:34 94 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 19:25:34 94 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 19:25:42 97 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 19:25:42 97 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 19:25:48 102 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 19:25:48 102 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 19:26:08 110 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:26:08 110 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:26:16 116 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:26:16 116 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 19:26:16 116 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 19:26:16 116 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 19:26:16 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=9 max_threads=502 thread_count=18 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x555d1ad75848 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2bd806ccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x555d16aacf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x555d165433af] sigaction.c:0(__restore_rt)[0x7f2c4220a630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x555d16471151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x555d165af545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x555d16365ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x555d1635bd4b] /usr/sbin/mysqld(+0x6ca792)[0x555d16366792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x555d1638e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x555d1639065d] /usr/sbin/mysqld(+0x65e3f3)[0x555d162fa3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x555d162f9cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x555d1638db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x555d1639065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x555d16390761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x555d16391377] /usr/sbin/mysqld(+0x58fd84)[0x555d1622bd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x555d16338b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x555d16350b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x555d16350cd2] /usr/sbin/mysqld(+0x6b55e7)[0x555d163515e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x555d16351677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x555d1634056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x555d1634207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x555d1641dafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x555d1641dbdd] pthread_create.c:0(start_thread)[0x7f2c42202ea5] /lib64/libc.so.6(clone+0x6d)[0x7f2c405a38cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x555d1ad84350): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 117 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:26:23 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:26:23 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:26:23 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:26:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:26:23 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:26:23 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:26:23 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:26:23 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:26:23 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:26:23 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135983510 2020-01-31 19:26:23 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000099', position 813509 2020-01-31 19:26:23 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:26:23 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:26:23 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:26:23 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:26:23 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:26:23 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:26:23 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135983519; transaction id 5899646 2020-01-31 19:26:23 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:26:23 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:26:23 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:26:23 0 [Note] Starting crash recovery... 2020-01-31 19:26:23 0 [Note] Crash recovery finished. 2020-01-31 19:26:23 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:26:23 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:26:23 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:26:23 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000058' at position 244683 2020-01-31 19:26:23 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 244683, relay log './dbsvc-528-relay-bin.000248' position: 559 2020-01-31 19:26:23 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:26:23 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000058' at position 244683 2020-01-31 19:26:24 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:26:24 2020-01-31 19:26:33 11 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 19:26:33 11 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 19:26:33 11 [Warning] shockeripe.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 19:26:33 11 [Note] shockeripe.wp_options: Table is fixed 2020-01-31 19:26:49 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:26:57 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:26:57 14 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:26:57 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:26:57 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:26:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f75b0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f7614ae8cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5561e0138f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5561dfbcf3af] sigaction.c:0(__restore_rt)[0x7f7631c60630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5561dfafd151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5561dfc3b545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5561df9f1ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5561df9e7d4b] /usr/sbin/mysqld(+0x6ca792)[0x5561df9f2792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5561dfa1a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5561dfa1c65d] /usr/sbin/mysqld(+0x65e3f3)[0x5561df9863f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5561df985cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5561dfa19b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5561dfa1c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5561dfa1c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5561dfa1d377] /usr/sbin/mysqld(+0x58fd84)[0x5561df8b7d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5561df9c4b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5561df9dcb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5561df9dccd2] /usr/sbin/mysqld(+0x6b55e7)[0x5561df9dd5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5561df9dd677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5561df9cc56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5561df9ce07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5561dfaa9afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5561dfaa9bdd] pthread_create.c:0(start_thread)[0x7f7631c58ea5] /lib64/libc.so.6(clone+0x6d)[0x7f762fff98cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f75b000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:27:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:27:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:27:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:27:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:27:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:27:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:27:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:27:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:27:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:27:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5135989904 2020-01-31 19:27:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000099', position 813509 2020-01-31 19:27:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:27:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:27:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:27:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:27:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:27:03 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:27:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5135989913; transaction id 5899665 2020-01-31 19:27:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:27:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:27:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:27:03 0 [Note] Starting crash recovery... 2020-01-31 19:27:03 0 [Note] Crash recovery finished. 2020-01-31 19:27:03 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:27:03 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:27:03 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:27:03 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000058' at position 250585 2020-01-31 19:27:03 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 250585, relay log './dbsvc-528-relay-bin.000250' position: 4429 2020-01-31 19:27:03 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000058' at position 250585 2020-01-31 19:27:03 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:27:03 11 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 19:27:03 11 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 19:27:03 11 [Warning] itstraining.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 19:27:03 11 [Note] itstraining.wp_options: Table is fixed 2020-01-31 19:27:04 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:27:04 2020-01-31 19:27:17 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000105, 637714) 2020-01-31 19:27:20 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:27:20 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:27:20 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 19:27:20 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 19:27:26 11 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 19:27:26 11 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 19:27:26 11 [Warning] wpcomm303.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 19:27:26 11 [Note] wpcomm303.wp_options: Table is fixed 2020-01-31 19:27:29 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:27:40 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:27:40 16 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:27:40 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:27:40 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:27:41 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7febdc000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fec3990ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55de2a427f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55de29ebe3af] sigaction.c:0(__restore_rt)[0x7fec54add630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55de29dec151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55de29f2a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55de29ce0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55de29cd6d4b] /usr/sbin/mysqld(+0x6ca792)[0x55de29ce1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55de29d096da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55de29d0b65d] /usr/sbin/mysqld(+0x65e3f3)[0x55de29c753f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55de29c74cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55de29d08b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55de29d0b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55de29d0b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55de29d0c377] /usr/sbin/mysqld(+0x58fd84)[0x55de29ba6d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55de29cb3b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55de29ccbb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55de29ccbcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55de29ccc5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55de29ccc677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55de29cbb56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55de29cbd07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55de29d98afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55de29d98bdd] pthread_create.c:0(start_thread)[0x7fec54ad5ea5] /lib64/libc.so.6(clone+0x6d)[0x7fec52e768cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7febdc00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:27:46 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:27:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:27:46 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:27:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:27:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:27:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:27:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:27:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:27:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:27:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136003021 2020-01-31 19:27:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000099', position 813509 2020-01-31 19:27:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:27:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:27:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:27:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:27:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:27:47 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:27:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136003030; transaction id 5899722 2020-01-31 19:27:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:27:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:27:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:27:47 0 [Note] Starting crash recovery... 2020-01-31 19:27:47 0 [Note] Crash recovery finished. 2020-01-31 19:27:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:27:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:27:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:27:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000058' at position 318117 2020-01-31 19:27:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 318117, relay log './dbsvc-528-relay-bin.000252' position: 44646 2020-01-31 19:27:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:27:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000058' at position 318117 2020-01-31 19:27:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:27:48 2020-01-31 19:28:14 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:28:41 18 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000107, 2663) 2020-01-31 19:28:55 24 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:28:55 24 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:28:57 25 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 19:28:57 25 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 19:29:02 29 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:29:02 29 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:29:11 33 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 19:29:11 33 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 19:30:14 55 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:30:14 55 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:30:14 56 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:30:14 56 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:30:14 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f6bb4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f6c0c904cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x558f9bd4ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x558f9b7e53af] sigaction.c:0(__restore_rt)[0x7f6c29b41630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x558f9b713151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x558f9b851545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x558f9b607ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x558f9b5fdd4b] /usr/sbin/mysqld(+0x6ca792)[0x558f9b608792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x558f9b6306da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x558f9b63265d] /usr/sbin/mysqld(+0x65e3f3)[0x558f9b59c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x558f9b59bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x558f9b62fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x558f9b63265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x558f9b632761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x558f9b633377] /usr/sbin/mysqld(+0x58fd84)[0x558f9b4cdd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x558f9b5dab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x558f9b5f2b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x558f9b5f2cd2] /usr/sbin/mysqld(+0x6b55e7)[0x558f9b5f35e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x558f9b5f3677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x558f9b5e256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x558f9b5e407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x558f9b6bfafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x558f9b6bfbdd] pthread_create.c:0(start_thread)[0x7f6c29b39ea5] /lib64/libc.so.6(clone+0x6d)[0x7f6c27eda8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f6bb400ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 58 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:30:21 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:30:21 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:30:21 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:30:21 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:30:21 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:30:21 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:30:21 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:30:21 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:30:21 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:30:21 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136008208 2020-01-31 19:30:21 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000099', position 813509 2020-01-31 19:30:21 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:30:21 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:30:21 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:30:21 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:30:21 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:30:21 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:30:21 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136008217; transaction id 5899733 2020-01-31 19:30:21 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:30:21 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:30:21 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:30:21 0 [Note] Starting crash recovery... 2020-01-31 19:30:21 0 [Note] Crash recovery finished. 2020-01-31 19:30:21 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:30:21 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:30:21 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:30:21 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000058' at position 318117 2020-01-31 19:30:21 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 318117, relay log './dbsvc-528-relay-bin.000254' position: 559 2020-01-31 19:30:21 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:30:21 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000058' at position 318117 2020-01-31 19:30:22 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:30:22 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:30:22 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 19:30:22 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 19:30:22 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:30:22 2020-01-31 19:30:39 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:30:39 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:30:39 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 19:30:39 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 19:30:50 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:31:08 19 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 19:31:08 19 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 19:31:15 22 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000108, 51525) 2020-01-31 19:31:40 32 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 19:31:40 32 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 19:31:57 37 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:31:57 37 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:31:57 38 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:31:57 38 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:31:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f88a8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f88bcffccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x557d6ca8af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x557d6c5213af] sigaction.c:0(__restore_rt)[0x7f891f409630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x557d6c44f151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x557d6c58d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x557d6c343ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x557d6c339d4b] /usr/sbin/mysqld(+0x6ca792)[0x557d6c344792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x557d6c36c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557d6c36e65d] /usr/sbin/mysqld(+0x65e3f3)[0x557d6c2d83f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x557d6c2d7cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x557d6c36bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557d6c36e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x557d6c36e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x557d6c36f377] /usr/sbin/mysqld(+0x58fd84)[0x557d6c209d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x557d6c316b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x557d6c32eb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x557d6c32ecd2] /usr/sbin/mysqld(+0x6b55e7)[0x557d6c32f5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x557d6c32f677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x557d6c31e56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x557d6c32007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x557d6c3fbafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x557d6c3fbbdd] pthread_create.c:0(start_thread)[0x7f891f401ea5] /lib64/libc.so.6(clone+0x6d)[0x7f891d7a28cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f88a800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 39 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:32:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:32:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:32:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:32:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:32:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:32:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:32:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:32:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:32:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:32:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136019381 2020-01-31 19:32:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000099', position 813509 2020-01-31 19:32:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:32:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:32:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:32:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:32:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:32:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136019390; transaction id 5899776 2020-01-31 19:32:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:32:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:32:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:32:03 0 [Note] Starting crash recovery... 2020-01-31 19:32:03 0 [Note] Crash recovery finished. 2020-01-31 19:32:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:32:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:32:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:32:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000058' at position 367908 2020-01-31 19:32:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 367908, relay log './dbsvc-528-relay-bin.000256' position: 33087 2020-01-31 19:32:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:32:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000058' at position 367908 2020-01-31 19:32:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:32:05 2020-01-31 19:32:21 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 19:32:21 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 19:32:21 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 19:32:21 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 19:32:30 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:32:58 28 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000109, 34521) 2020-01-31 19:33:01 30 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 19:33:01 30 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 19:33:04 32 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:33:04 32 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:33:05 33 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 19:33:05 33 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 19:33:15 38 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 19:33:15 38 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 19:33:22 42 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:33:22 42 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:33:24 46 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 19:33:24 46 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 19:33:28 53 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 19:33:28 53 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 19:33:35 62 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:33:35 62 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:33:35 63 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:33:35 63 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:33:36 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2a24000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2a80ec8cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x558ba448ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x558ba3f263af] sigaction.c:0(__restore_rt)[0x7f2a9c0cf630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x558ba3e54151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x558ba3f92545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x558ba3d48ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x558ba3d3ed4b] /usr/sbin/mysqld(+0x6ca792)[0x558ba3d49792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x558ba3d716da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x558ba3d7365d] /usr/sbin/mysqld(+0x65e3f3)[0x558ba3cdd3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x558ba3cdccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x558ba3d70b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x558ba3d7365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x558ba3d73761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x558ba3d74377] /usr/sbin/mysqld(+0x58fd84)[0x558ba3c0ed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x558ba3d1bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x558ba3d33b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x558ba3d33cd2] /usr/sbin/mysqld(+0x6b55e7)[0x558ba3d345e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x558ba3d34677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x558ba3d2356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x558ba3d2507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x558ba3e00afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x558ba3e00bdd] pthread_create.c:0(start_thread)[0x7f2a9c0c7ea5] /lib64/libc.so.6(clone+0x6d)[0x7f2a9a4688cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f2a2400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 65 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:33:42 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:33:42 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:33:42 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:33:42 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:33:42 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:33:42 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:33:42 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:33:42 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:33:42 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:33:42 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136031169 2020-01-31 19:33:42 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000099', position 813509 2020-01-31 19:33:42 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:33:42 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:33:42 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:33:42 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:33:42 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:33:42 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:33:42 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136031178; transaction id 5899813 2020-01-31 19:33:42 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:33:42 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:33:42 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:33:42 0 [Note] Starting crash recovery... 2020-01-31 19:33:42 0 [Note] Crash recovery finished. 2020-01-31 19:33:42 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:33:42 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:33:42 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:33:42 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000058' at position 398450 2020-01-31 19:33:42 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 398450, relay log './dbsvc-528-relay-bin.000258' position: 20490 2020-01-31 19:33:42 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:33:42 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000058' at position 398450 2020-01-31 19:33:42 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 19:33:42 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 19:33:42 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 19:33:42 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 19:33:43 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:33:43 2020-01-31 19:33:57 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:33:57 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:33:57 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 19:33:57 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 19:34:10 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:34:27 21 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 19:34:27 21 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 19:34:32 27 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 19:34:32 27 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 19:34:36 31 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000110, 548559) 2020-01-31 19:34:40 33 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 19:34:40 33 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 19:35:34 59 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 19:35:34 59 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 19:35:47 65 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:35:47 65 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:36:57 92 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:36:57 92 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:36:57 93 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:36:57 93 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:36:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0708000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0758b7ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x559f43dfbf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x559f438923af] sigaction.c:0(__restore_rt)[0x7f0775d92630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x559f437c0151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x559f438fe545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x559f436b4ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x559f436aad4b] /usr/sbin/mysqld(+0x6ca792)[0x559f436b5792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x559f436dd6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559f436df65d] /usr/sbin/mysqld(+0x65e3f3)[0x559f436493f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x559f43648cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x559f436dcb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559f436df65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x559f436df761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x559f436e0377] /usr/sbin/mysqld(+0x58fd84)[0x559f4357ad84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x559f43687b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x559f4369fb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x559f4369fcd2] /usr/sbin/mysqld(+0x6b55e7)[0x559f436a05e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x559f436a0677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x559f4368f56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x559f4369107c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x559f4376cafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x559f4376cbdd] pthread_create.c:0(start_thread)[0x7f0775d8aea5] /lib64/libc.so.6(clone+0x6d)[0x7f077412b8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f070800feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 94 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:37:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:37:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:37:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:37:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:37:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:37:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:37:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:37:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:37:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:37:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136047915 2020-01-31 19:37:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000099', position 813509 2020-01-31 19:37:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:37:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:37:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:37:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:37:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:37:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:37:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136047924; transaction id 5899866 2020-01-31 19:37:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:37:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:37:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:37:04 0 [Note] Starting crash recovery... 2020-01-31 19:37:04 0 [Note] Crash recovery finished. 2020-01-31 19:37:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:37:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:37:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:37:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000058' at position 531871 2020-01-31 19:37:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 531871, relay log './dbsvc-528-relay-bin.000260' position: 87532 2020-01-31 19:37:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:37:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000058' at position 531871 2020-01-31 19:37:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:37:05 2020-01-31 19:37:10 11 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 19:37:10 11 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 19:37:10 11 [Warning] shockeripe.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 19:37:10 11 [Note] shockeripe.wp_options: Table is fixed 2020-01-31 19:37:10 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:37:10 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:37:10 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 19:37:10 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 19:37:31 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:37:52 31 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 19:37:52 31 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 19:37:56 37 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 19:37:56 37 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 19:37:58 39 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000111, 609604) 2020-01-31 19:38:32 49 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 19:38:32 49 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 19:39:25 69 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 19:39:25 69 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 19:41:14 105 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 19:41:14 105 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 19:41:40 119 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 19:41:40 119 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 19:41:57 129 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:41:57 129 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:41:57 130 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:41:57 130 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:41:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f1f14000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f1f79b2bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e7c70f3f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e7c6b8a3af] sigaction.c:0(__restore_rt)[0x7f1f94d1e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55e7c6ab8151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55e7c6bf6545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55e7c69acccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55e7c69a2d4b] /usr/sbin/mysqld(+0x6ca792)[0x55e7c69ad792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55e7c69d56da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e7c69d765d] /usr/sbin/mysqld(+0x65e3f3)[0x55e7c69413f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55e7c6940cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55e7c69d4b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e7c69d765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55e7c69d7761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55e7c69d8377] /usr/sbin/mysqld(+0x58fd84)[0x55e7c6872d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55e7c697fb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55e7c6997b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55e7c6997cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55e7c69985e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55e7c6998677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55e7c698756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55e7c698907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55e7c6a64afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e7c6a64bdd] pthread_create.c:0(start_thread)[0x7f1f94d16ea5] /lib64/libc.so.6(clone+0x6d)[0x7f1f930b78cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f1f1400feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 131 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:42:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:42:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:42:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:42:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:42:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:42:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:42:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:42:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:42:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:42:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136068246 2020-01-31 19:42:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000112', position 139241 2020-01-31 19:42:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:42:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:42:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:42:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:42:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:42:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:42:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136068255; transaction id 5899909 2020-01-31 19:42:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:42:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:42:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:42:04 0 [Note] Starting crash recovery... 2020-01-31 19:42:04 0 [Note] Crash recovery finished. 2020-01-31 19:42:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:42:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:42:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:42:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 19:42:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000262' position: 2728 2020-01-31 19:42:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:42:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 19:42:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:42:05 2020-01-31 19:42:32 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:42:33 18 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:42:33 18 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:42:33 19 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:42:33 19 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:42:33 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f79cc000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f79e5305cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x559198355f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x559197dec3af] sigaction.c:0(__restore_rt)[0x7f7a464d4630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x559197d1a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x559197e58545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x559197c0eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x559197c04d4b] /usr/sbin/mysqld(+0x6ca792)[0x559197c0f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x559197c376da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559197c3965d] /usr/sbin/mysqld(+0x65e3f3)[0x559197ba33f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x559197ba2cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x559197c36b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559197c3965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x559197c39761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x559197c3a377] /usr/sbin/mysqld(+0x58fd84)[0x559197ad4d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x559197be1b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x559197bf9b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x559197bf9cd2] /usr/sbin/mysqld(+0x6b55e7)[0x559197bfa5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x559197bfa677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x559197be956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x559197beb07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x559197cc6afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x559197cc6bdd] pthread_create.c:0(start_thread)[0x7f7a464ccea5] /lib64/libc.so.6(clone+0x6d)[0x7f7a4486d8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f79cc00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 20 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:42:39 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:42:39 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:42:39 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:42:39 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:42:39 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:42:39 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:42:39 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:42:39 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:42:39 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:42:39 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136072392 2020-01-31 19:42:39 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000112', position 139241 2020-01-31 19:42:39 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:42:39 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:42:39 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:42:39 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:42:39 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:42:39 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:42:39 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136072401; transaction id 5899920 2020-01-31 19:42:39 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:42:39 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:42:39 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:42:39 0 [Note] Starting crash recovery... 2020-01-31 19:42:39 0 [Note] Crash recovery finished. 2020-01-31 19:42:39 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:42:39 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:42:39 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:42:39 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 19:42:39 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000264' position: 559 2020-01-31 19:42:39 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:42:39 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 19:42:39 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:42:39 2020-01-31 19:42:45 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000112, 458623) 2020-01-31 19:43:07 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:43:07 14 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:43:07 14 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:43:08 16 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 19:43:08 16 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 19:43:08 15 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 19:43:08 15 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 19:43:15 20 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 19:43:15 20 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 19:43:15 21 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 19:43:15 21 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 19:43:15 22 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 19:43:15 22 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 19:43:23 29 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 19:43:23 29 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 19:43:24 31 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:43:24 31 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:43:29 37 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 19:43:29 37 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 19:43:39 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 19:43:39 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:43:39 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:43:39 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 19:43:39 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 19:43:39 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 19:43:39 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 19:43:39 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 19:43:39 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 19:43:39 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 19:44:28 57 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 19:44:28 57 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 19:45:34 83 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 19:45:34 83 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 19:46:02 102 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:46:02 102 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:46:02 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f60900009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f60f0069cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x555ac6ee0f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x555ac69773af] sigaction.c:0(__restore_rt)[0x7f610d397630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x555ac68a5151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x555ac69e3545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x555ac6799ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x555ac678fd4b] /usr/sbin/mysqld(+0x6ca792)[0x555ac679a792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x555ac67c26da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x555ac67c465d] /usr/sbin/mysqld(+0x65e3f3)[0x555ac672e3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x555ac672dcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x555ac67c1b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x555ac67c465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x555ac67c4761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x555ac67c5377] /usr/sbin/mysqld(+0x58fd84)[0x555ac665fd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x555ac676cb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x555ac6784b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x555ac6784cd2] /usr/sbin/mysqld(+0x6b55e7)[0x555ac67855e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x555ac6785677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x555ac677456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x555ac677607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x555ac6851afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x555ac6851bdd] pthread_create.c:0(start_thread)[0x7f610d38fea5] /lib64/libc.so.6(clone+0x6d)[0x7f610b7308cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f6090010950): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 103 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:46:09 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:46:09 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:46:09 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:46:09 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:46:09 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:46:09 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:46:09 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:46:09 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:46:09 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:46:09 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136075277 2020-01-31 19:46:09 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000112', position 139241 2020-01-31 19:46:09 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:46:09 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:46:09 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:46:09 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:46:09 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:46:09 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:46:09 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136075286; transaction id 5899940 2020-01-31 19:46:09 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:46:09 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:46:09 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:46:09 0 [Note] Starting crash recovery... 2020-01-31 19:46:09 0 [Note] Crash recovery finished. 2020-01-31 19:46:09 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:46:09 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:46:09 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:46:09 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000059' at position 346 2020-01-31 19:46:09 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 19:46:09 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000059' at position 346 2020-01-31 19:46:09 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:46:09 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:46:09 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:46:09 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 19:46:09 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 19:46:09 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 19:46:09 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 19:46:09 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 19:46:09 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 19:46:09 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 19:46:10 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:46:10 2020-01-31 19:46:37 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:46:48 16 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 19:46:48 16 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 19:46:57 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:46:57 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:46:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f60ac006bf8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f611117bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55bc42a25f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55bc424bc3af] sigaction.c:0(__restore_rt)[0x7f612c293630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55bc423ea151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55bc42528545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55bc422deccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55bc422d4d4b] /usr/sbin/mysqld(+0x6ca792)[0x55bc422df792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55bc423076da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55bc4230965d] /usr/sbin/mysqld(+0x65e3f3)[0x55bc422733f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55bc42272cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55bc42306b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55bc4230965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55bc42309761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55bc4230a377] /usr/sbin/mysqld(+0x58fd84)[0x55bc421a4d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55bc422b1b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55bc422c9b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55bc422c9cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55bc422ca5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55bc422ca677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55bc422b956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55bc422bb07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55bc42396afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55bc42396bdd] pthread_create.c:0(start_thread)[0x7f612c28bea5] /lib64/libc.so.6(clone+0x6d)[0x7f612a62c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f60ac010ea0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:47:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:47:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:47:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:47:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:47:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:47:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:47:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:47:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:47:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:47:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136078123 2020-01-31 19:47:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000112', position 139241 2020-01-31 19:47:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:47:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:47:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:47:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:47:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:47:03 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:47:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136078132; transaction id 5899951 2020-01-31 19:47:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:47:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:47:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:47:03 0 [Note] Starting crash recovery... 2020-01-31 19:47:03 0 [Note] Crash recovery finished. 2020-01-31 19:47:03 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:47:03 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:47:03 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:47:03 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000059' at position 21933 2020-01-31 19:47:03 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 19:47:03 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:47:03 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:47:03 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:47:03 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 19:47:03 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 19:47:03 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000059' at position 21933 2020-01-31 19:47:03 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 19:47:03 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 19:47:03 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 19:47:03 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 19:47:03 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 19:47:03 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:47:03 2020-01-31 19:47:04 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000114, 387397) 2020-01-31 19:47:33 18 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:47:47 22 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 19:47:47 22 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 19:47:51 26 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:47:51 26 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:48:35 54 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 19:48:35 54 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 19:48:41 57 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:48:41 57 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:48:43 59 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 19:48:43 59 [Warning] Checking table: './wpms/wp_usermeta' 2020-01-31 19:50:46 104 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 19:50:46 104 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 19:51:05 113 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 19:51:05 113 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 19:51:40 129 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 19:51:40 129 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 19:51:58 135 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:51:58 135 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:51:58 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc270000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc2cdbc1cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5639166abf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5639161423af] sigaction.c:0(__restore_rt)[0x7fc2e8d9a630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x563916070151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5639161ae545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x563915f64ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x563915f5ad4b] /usr/sbin/mysqld(+0x6ca792)[0x563915f65792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x563915f8d6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563915f8f65d] /usr/sbin/mysqld(+0x65e3f3)[0x563915ef93f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x563915ef8cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x563915f8cb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563915f8f65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x563915f8f761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x563915f90377] /usr/sbin/mysqld(+0x58fd84)[0x563915e2ad84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x563915f37b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x563915f4fb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x563915f4fcd2] /usr/sbin/mysqld(+0x6b55e7)[0x563915f505e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x563915f50677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x563915f3f56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x563915f4107c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56391601cafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56391601cbdd] pthread_create.c:0(start_thread)[0x7fc2e8d92ea5] /lib64/libc.so.6(clone+0x6d)[0x7fc2e71338cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc27000ff10): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 136 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:52:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:52:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:52:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:52:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:52:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:52:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:52:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:52:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:52:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:52:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136081786 2020-01-31 19:52:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000112', position 139241 2020-01-31 19:52:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:52:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:52:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:52:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:52:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:52:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136081795; transaction id 5899962 2020-01-31 19:52:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:52:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:52:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:52:04 0 [Note] Starting crash recovery... 2020-01-31 19:52:04 0 [Note] Crash recovery finished. 2020-01-31 19:52:05 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:52:05 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:52:05 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:52:05 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000059' at position 122024 2020-01-31 19:52:05 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 19:52:05 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:52:05 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:52:05 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:52:05 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 19:52:05 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 19:52:05 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 19:52:05 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 19:52:05 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 19:52:05 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 19:52:05 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 19:52:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:52:05 2020-01-31 19:52:06 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000059' at position 122024 2020-01-31 19:52:33 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:52:59 25 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000116, 1123298) 2020-01-31 19:52:59 26 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:52:59 26 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:53:00 28 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 19:53:00 28 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 19:53:15 47 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 19:53:15 47 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 19:53:24 52 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 19:53:24 52 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 19:53:24 53 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 19:53:24 53 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 19:53:28 57 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 19:53:28 57 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 19:53:29 60 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:53:29 60 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:53:37 63 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 19:53:37 63 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 19:53:42 69 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:53:42 69 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:53:42 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe79c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe7f4ca0cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560afe243f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560afdcda3af] sigaction.c:0(__restore_rt)[0x7fe8110bd630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x560afdc08151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x560afdd46545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x560afdafcccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x560afdaf2d4b] /usr/sbin/mysqld(+0x6ca792)[0x560afdafd792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x560afdb256da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560afdb2765d] /usr/sbin/mysqld(+0x65e3f3)[0x560afda913f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x560afda90cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x560afdb24b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560afdb2765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x560afdb27761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x560afdb28377] /usr/sbin/mysqld(+0x58fd84)[0x560afd9c2d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x560afdacfb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x560afdae7b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x560afdae7cd2] /usr/sbin/mysqld(+0x6b55e7)[0x560afdae85e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x560afdae8677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x560afdad756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x560afdad907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x560afdbb4afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560afdbb4bdd] pthread_create.c:0(start_thread)[0x7fe8110b5ea5] /lib64/libc.so.6(clone+0x6d)[0x7fe80f4568cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe79c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 70 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:53:49 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:53:49 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:53:49 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:53:49 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:53:49 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:53:49 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:53:49 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:53:49 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:53:49 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:53:49 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136084786 2020-01-31 19:53:49 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000112', position 139241 2020-01-31 19:53:49 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:53:49 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:53:49 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:53:49 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:53:49 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:53:49 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136084795; transaction id 5899973 2020-01-31 19:53:49 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:53:49 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:53:49 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:53:49 0 [Note] Starting crash recovery... 2020-01-31 19:53:49 0 [Note] Crash recovery finished. 2020-01-31 19:53:49 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:53:49 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:53:49 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:53:49 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000059' at position 267308 2020-01-31 19:53:49 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 19:53:49 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:53:49 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:53:49 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000059' at position 267308 2020-01-31 19:53:49 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:53:49 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 19:53:49 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 19:53:49 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 19:53:49 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 19:53:49 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 19:53:49 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 19:53:49 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 19:53:50 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:53:50 2020-01-31 19:54:14 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:54:14 14 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 19:54:14 14 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 19:54:27 18 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 19:54:27 18 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 19:54:48 25 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000117, 190401) 2020-01-31 19:54:54 28 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:54:54 28 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:55:00 30 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 19:55:00 30 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 19:55:14 40 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:55:14 40 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:55:14 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ffad8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ffaebf1ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5650a4455f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5650a3eec3af] sigaction.c:0(__restore_rt)[0x7ffb4e324630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5650a3e1a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5650a3f58545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5650a3d0eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5650a3d04d4b] /usr/sbin/mysqld(+0x6ca792)[0x5650a3d0f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5650a3d376da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5650a3d3965d] /usr/sbin/mysqld(+0x65e3f3)[0x5650a3ca33f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5650a3ca2cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5650a3d36b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5650a3d3965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5650a3d39761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5650a3d3a377] /usr/sbin/mysqld(+0x58fd84)[0x5650a3bd4d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5650a3ce1b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5650a3cf9b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5650a3cf9cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5650a3cfa5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5650a3cfa677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5650a3ce956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5650a3ceb07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5650a3dc6afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5650a3dc6bdd] pthread_create.c:0(start_thread)[0x7ffb4e31cea5] /lib64/libc.so.6(clone+0x6d)[0x7ffb4c6bd8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ffad800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 41 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:55:20 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:55:20 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:55:20 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:55:20 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:55:20 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:55:20 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:55:20 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:55:20 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:55:20 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:55:20 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136088077 2020-01-31 19:55:20 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000118', position 749 2020-01-31 19:55:20 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:55:20 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:55:20 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:55:20 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:55:20 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:55:20 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:55:20 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136088086; transaction id 5899988 2020-01-31 19:55:20 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:55:20 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:55:20 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:55:20 0 [Note] Starting crash recovery... 2020-01-31 19:55:20 0 [Note] Crash recovery finished. 2020-01-31 19:55:20 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:55:20 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:55:20 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:55:20 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000059' at position 267308 2020-01-31 19:55:20 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 19:55:20 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:55:20 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:55:20 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:55:20 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 19:55:20 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 19:55:20 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 19:55:20 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 19:55:20 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 19:55:20 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 19:55:20 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 19:55:20 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:55:20 2020-01-31 19:55:21 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000059' at position 267308 2020-01-31 19:55:43 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 19:55:43 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000059' at position 303809 2020-01-31 19:55:43 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 19:55:49 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:55:49 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000118, 7859) 2020-01-31 19:55:50 14 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:55:50 14 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:56:09 25 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:56:09 25 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:56:10 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fd0a4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fd0bade9cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56302d954f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56302d3eb3af] sigaction.c:0(__restore_rt)[0x7fd11d154630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56302d319151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56302d457545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56302d20dccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56302d203d4b] /usr/sbin/mysqld(+0x6ca792)[0x56302d20e792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56302d2366da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56302d23865d] /usr/sbin/mysqld(+0x65e3f3)[0x56302d1a23f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56302d1a1cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56302d235b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56302d23865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56302d238761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56302d239377] /usr/sbin/mysqld(+0x58fd84)[0x56302d0d3d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56302d1e0b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56302d1f8b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56302d1f8cd2] /usr/sbin/mysqld(+0x6b55e7)[0x56302d1f95e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56302d1f9677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56302d1e856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56302d1ea07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56302d2c5afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56302d2c5bdd] pthread_create.c:0(start_thread)[0x7fd11d14cea5] /lib64/libc.so.6(clone+0x6d)[0x7fd11b4ed8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fd0a400fe70): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 26 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:56:16 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:56:16 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:56:16 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:56:16 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:56:16 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:56:16 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:56:16 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:56:16 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:56:16 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:56:17 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136090924 2020-01-31 19:56:17 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000118', position 749 2020-01-31 19:56:17 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:56:17 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:56:17 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:56:17 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:56:17 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:56:17 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:56:17 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136090933; transaction id 5899999 2020-01-31 19:56:17 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:56:17 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:56:17 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:56:17 0 [Note] Starting crash recovery... 2020-01-31 19:56:17 0 [Note] Crash recovery finished. 2020-01-31 19:56:17 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:56:17 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:56:17 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:56:17 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000059' at position 303809 2020-01-31 19:56:17 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 19:56:17 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:56:17 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:56:17 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000059' at position 303809 2020-01-31 19:56:17 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:56:17 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 19:56:17 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 19:56:17 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 19:56:17 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 19:56:17 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 19:56:17 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 19:56:17 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 19:56:18 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:56:18 2020-01-31 19:56:44 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:56:48 13 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 19:56:48 13 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 19:56:50 15 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:56:50 15 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:56:57 19 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 19:56:57 19 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 19:56:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9e6c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f9ed00c3cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563b78b45f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x563b785dc3af] sigaction.c:0(__restore_rt)[0x7f9eea772630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x563b7850a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x563b78648545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x563b783feccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x563b783f4d4b] /usr/sbin/mysqld(+0x6ca792)[0x563b783ff792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x563b784276da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563b7842965d] /usr/sbin/mysqld(+0x65e3f3)[0x563b783933f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x563b78392cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x563b78426b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563b7842965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x563b78429761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x563b7842a377] /usr/sbin/mysqld(+0x58fd84)[0x563b782c4d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x563b783d1b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x563b783e9b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x563b783e9cd2] /usr/sbin/mysqld(+0x6b55e7)[0x563b783ea5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x563b783ea677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x563b783d956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x563b783db07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x563b784b6afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x563b784b6bdd] pthread_create.c:0(start_thread)[0x7f9eea76aea5] /lib64/libc.so.6(clone+0x6d)[0x7f9ee8b0b8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f9e6c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 20 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 19:57:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 19:57:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 19:57:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 19:57:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 19:57:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 19:57:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 19:57:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 19:57:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 19:57:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 19:57:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136093787 2020-01-31 19:57:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000118', position 749 2020-01-31 19:57:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 19:57:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 19:57:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 19:57:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 19:57:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 19:57:03 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 19:57:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136093796; transaction id 5900010 2020-01-31 19:57:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 19:57:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 19:57:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 19:57:04 0 [Note] Starting crash recovery... 2020-01-31 19:57:04 0 [Note] Crash recovery finished. 2020-01-31 19:57:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 19:57:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 19:57:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 19:57:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000060' at position 120143 2020-01-31 19:57:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 19:57:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 19:57:04 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 19:57:04 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 19:57:04 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 19:57:04 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 19:57:04 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 19:57:04 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 19:57:04 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 19:57:04 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 19:57:04 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 19:57:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000060' at position 120143 2020-01-31 19:57:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 19:57:05 2020-01-31 19:57:10 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000119, 52025) 2020-01-31 19:57:29 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 19:57:30 14 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 19:57:30 14 [Warning] Checking table: './wpms/wp_options' 2020-01-31 19:57:48 30 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 19:57:48 30 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 19:57:57 34 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 19:57:57 34 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 19:57:58 35 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 19:57:58 35 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 19:58:13 40 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 19:58:13 40 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 19:59:38 74 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 19:59:38 74 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 20:01:41 124 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 20:01:41 124 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 20:01:58 134 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:01:58 134 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:01:58 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fceec000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fcf003cfcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5653c868ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5653c81263af] sigaction.c:0(__restore_rt)[0x7fcf6cdb3630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5653c8054151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5653c8192545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5653c7f48ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5653c7f3ed4b] /usr/sbin/mysqld(+0x6ca792)[0x5653c7f49792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5653c7f716da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5653c7f7365d] /usr/sbin/mysqld(+0x65e3f3)[0x5653c7edd3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5653c7edccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5653c7f70b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5653c7f7365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5653c7f73761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5653c7f74377] /usr/sbin/mysqld(+0x58fd84)[0x5653c7e0ed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5653c7f1bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5653c7f33b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5653c7f33cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5653c7f345e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5653c7f34677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5653c7f2356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5653c7f2507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5653c8000afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5653c8000bdd] pthread_create.c:0(start_thread)[0x7fcf6cdabea5] /lib64/libc.so.6(clone+0x6d)[0x7fcf6b14c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fceec00fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 135 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:02:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:02:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:02:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:02:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:02:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:02:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:02:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:02:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:02:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:02:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136097921 2020-01-31 20:02:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 20:02:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:02:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:02:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:02:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:02:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:02:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:02:05 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136097930; transaction id 5900027 2020-01-31 20:02:05 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:02:05 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:02:05 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:02:05 0 [Note] Starting crash recovery... 2020-01-31 20:02:05 0 [Note] Crash recovery finished. 2020-01-31 20:02:05 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:02:05 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:02:05 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:02:05 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000060' at position 419268 2020-01-31 20:02:05 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:02:05 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:02:05 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:02:05 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000060' at position 419268 2020-01-31 20:02:05 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:02:05 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:02:05 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:02:05 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:02:05 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:02:05 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:02:05 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:02:05 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:02:06 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:02:06 2020-01-31 20:02:30 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:02:32 19 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:02:32 19 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 20:02:33 19 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 20:02:33 19 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 20:02:33 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f14e8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f1549211cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x556ea4be7f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x556ea467e3af] sigaction.c:0(__restore_rt)[0x7f1564328630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x556ea45ac151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x556ea46ea545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x556ea44a0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x556ea4496d4b] /usr/sbin/mysqld(+0x6ca792)[0x556ea44a1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x556ea44c96da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556ea44cb65d] /usr/sbin/mysqld(+0x65e3f3)[0x556ea44353f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x556ea4434cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x556ea44c8b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556ea44cb65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x556ea44cb761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x556ea44cc377] /usr/sbin/mysqld(+0x58fd84)[0x556ea4366d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x556ea4473b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x556ea448bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x556ea448bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x556ea448c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x556ea448c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x556ea447b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x556ea447d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x556ea4558afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x556ea4558bdd] pthread_create.c:0(start_thread)[0x7f1564320ea5] /lib64/libc.so.6(clone+0x6d)[0x7f15626c18cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f14e800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 20 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:02:39 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:02:39 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:02:39 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:02:39 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:02:39 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:02:39 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:02:39 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:02:39 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:02:39 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:02:39 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136100766 2020-01-31 20:02:39 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 20:02:39 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:02:39 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:02:39 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:02:39 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:02:39 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:02:39 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:02:39 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136100775; transaction id 5900039 2020-01-31 20:02:39 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:02:39 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:02:39 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:02:39 0 [Note] Starting crash recovery... 2020-01-31 20:02:39 0 [Note] Crash recovery finished. 2020-01-31 20:02:39 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:02:39 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:02:39 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:02:39 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000060' at position 419268 2020-01-31 20:02:39 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:02:39 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:02:39 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000060' at position 419268 2020-01-31 20:02:39 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:02:39 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:02:39 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:02:39 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:02:39 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:02:39 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:02:39 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:02:39 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:02:39 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:02:40 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:02:40 2020-01-31 20:02:59 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000121, 852990) 2020-01-31 20:03:05 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:03:07 15 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:03:07 15 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:03:15 18 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 20:03:15 18 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 20:03:24 24 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 20:03:24 24 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 20:03:24 25 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 20:03:24 25 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 20:03:28 31 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 20:03:28 31 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 20:03:33 34 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:03:33 34 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:03:34 34 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:03:34 34 [Warning] Checking table: './wpms/wp_usermeta' 2020-01-31 20:04:27 71 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 20:04:27 71 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 20:04:32 75 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 20:04:32 75 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 20:05:34 99 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 20:05:34 99 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 20:06:02 115 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 20:06:02 115 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 20:06:13 119 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 20:06:13 119 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 20:06:28 125 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:06:28 125 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:06:28 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f8f74000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f8fcc0f4cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55564eba8f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55564e63f3af] sigaction.c:0(__restore_rt)[0x7f8fe8ea6630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55564e56d151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55564e6ab545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55564e461ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55564e457d4b] /usr/sbin/mysqld(+0x6ca792)[0x55564e462792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55564e48a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55564e48c65d] /usr/sbin/mysqld(+0x65e3f3)[0x55564e3f63f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55564e3f5cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55564e489b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55564e48c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55564e48c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55564e48d377] /usr/sbin/mysqld(+0x58fd84)[0x55564e327d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55564e434b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55564e44cb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55564e44ccd2] /usr/sbin/mysqld(+0x6b55e7)[0x55564e44d5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55564e44d677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55564e43c56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55564e43e07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55564e519afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55564e519bdd] pthread_create.c:0(start_thread)[0x7f8fe8e9eea5] /lib64/libc.so.6(clone+0x6d)[0x7f8fe723f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f8f7400fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 126 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:06:35 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:06:35 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:06:35 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:06:35 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:06:35 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:06:35 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:06:35 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:06:35 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:06:35 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:06:35 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136104420 2020-01-31 20:06:35 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 20:06:35 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:06:35 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:06:35 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:06:35 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:06:35 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:06:35 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:06:35 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136104429; transaction id 5900050 2020-01-31 20:06:35 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:06:35 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:06:35 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:06:35 0 [Note] Starting crash recovery... 2020-01-31 20:06:35 0 [Note] Crash recovery finished. 2020-01-31 20:06:35 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:06:35 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:06:35 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:06:35 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000060' at position 478261 2020-01-31 20:06:35 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:06:35 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:06:35 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:06:35 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000060' at position 478261 2020-01-31 20:06:35 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:06:35 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:06:35 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:06:35 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:06:35 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:06:35 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:06:35 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:06:35 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:06:36 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:06:36 2020-01-31 20:06:58 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 20:06:58 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000060' at position 485209 2020-01-31 20:06:59 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 20:07:01 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:07:04 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000123, 959072) 2020-01-31 20:07:22 18 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:07:22 18 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:07:23 19 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:07:23 19 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 20:07:23 20 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 20:07:23 20 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 20:07:25 22 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 20:07:25 22 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 20:07:34 30 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:07:34 30 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:07:38 35 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 20:07:38 35 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 20:07:59 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000060' at position 485209 2020-01-31 20:08:07 53 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 20:08:07 53 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 20:08:41 64 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:08:41 64 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 20:08:41 64 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 20:08:41 64 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 20:08:41 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f716c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f71c00accf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561aa9e12f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x561aa98a93af] sigaction.c:0(__restore_rt)[0x7f71da9da630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x561aa97d7151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x561aa9915545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x561aa96cbccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x561aa96c1d4b] /usr/sbin/mysqld(+0x6ca792)[0x561aa96cc792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x561aa96f46da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x561aa96f665d] /usr/sbin/mysqld(+0x65e3f3)[0x561aa96603f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x561aa965fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x561aa96f3b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x561aa96f665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x561aa96f6761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x561aa96f7377] /usr/sbin/mysqld(+0x58fd84)[0x561aa9591d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x561aa969eb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x561aa96b6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x561aa96b6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x561aa96b75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x561aa96b7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x561aa96a656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x561aa96a807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x561aa9783afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x561aa9783bdd] pthread_create.c:0(start_thread)[0x7f71da9d2ea5] /lib64/libc.so.6(clone+0x6d)[0x7f71d8d738cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f716c00ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 66 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:08:48 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:08:48 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:08:48 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:08:48 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:08:48 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:08:48 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:08:48 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:08:48 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:08:48 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:08:48 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136108205 2020-01-31 20:08:48 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 20:08:48 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:08:48 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:08:48 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:08:48 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:08:48 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:08:48 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:08:48 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136108214; transaction id 5900067 2020-01-31 20:08:48 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:08:48 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:08:48 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:08:48 0 [Note] Starting crash recovery... 2020-01-31 20:08:48 0 [Note] Crash recovery finished. 2020-01-31 20:08:48 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:08:48 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:08:48 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:08:48 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000061' at position 346 2020-01-31 20:08:48 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:08:48 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:08:48 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:08:48 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000061' at position 346 2020-01-31 20:08:48 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:08:48 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:08:48 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:08:48 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:08:48 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:08:48 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:08:48 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:08:48 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:08:50 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:08:50 2020-01-31 20:09:17 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:09:34 19 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:09:34 19 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:09:45 23 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000124, 96366) 2020-01-31 20:09:58 27 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:09:58 27 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:09:59 28 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:09:59 28 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 20:10:05 33 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 20:10:05 33 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 20:10:52 56 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:10:52 56 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 20:10:52 56 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 20:10:52 56 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 20:10:52 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff478006be8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff4e9130cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e51e208f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e51dc9f3af] sigaction.c:0(__restore_rt)[0x7ff5041e7630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55e51dbcd151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55e51dd0b545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55e51dac1ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55e51dab7d4b] /usr/sbin/mysqld(+0x6ca792)[0x55e51dac2792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55e51daea6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e51daec65d] /usr/sbin/mysqld(+0x65e3f3)[0x55e51da563f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55e51da55cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55e51dae9b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e51daec65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55e51daec761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55e51daed377] /usr/sbin/mysqld(+0x58fd84)[0x55e51d987d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55e51da94b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55e51daacb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55e51daaccd2] /usr/sbin/mysqld(+0x6b55e7)[0x55e51daad5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55e51daad677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55e51da9c56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55e51da9e07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55e51db79afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e51db79bdd] pthread_create.c:0(start_thread)[0x7ff5041dfea5] /lib64/libc.so.6(clone+0x6d)[0x7ff5025808cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff478010e90): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 57 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:10:59 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:10:59 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:10:59 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:10:59 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:10:59 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:10:59 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:10:59 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:10:59 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:10:59 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:10:59 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136111196 2020-01-31 20:10:59 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 20:10:59 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:10:59 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:10:59 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:10:59 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:10:59 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:10:59 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:10:59 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136111205; transaction id 5900078 2020-01-31 20:10:59 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:10:59 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:10:59 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:10:59 0 [Note] Starting crash recovery... 2020-01-31 20:10:59 0 [Note] Crash recovery finished. 2020-01-31 20:10:59 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:10:59 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:10:59 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:10:59 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000061' at position 66997 2020-01-31 20:10:59 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:10:59 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:10:59 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:10:59 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000061' at position 66997 2020-01-31 20:10:59 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:10:59 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:10:59 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:10:59 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:10:59 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:10:59 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:10:59 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:10:59 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:11:00 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:11:00 2020-01-31 20:11:27 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:11:34 14 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:11:34 14 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:11:43 23 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:11:43 23 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:11:44 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f462c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4688bc9cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563014d6af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5630148013af] sigaction.c:0(__restore_rt)[0x7f46a5d08630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56301472f151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56301486d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x563014623ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x563014619d4b] /usr/sbin/mysqld(+0x6ca792)[0x563014624792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56301464c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56301464e65d] /usr/sbin/mysqld(+0x65e3f3)[0x5630145b83f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5630145b7cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56301464bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56301464e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56301464e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56301464f377] /usr/sbin/mysqld(+0x58fd84)[0x5630144e9d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5630145f6b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56301460eb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56301460ecd2] /usr/sbin/mysqld(+0x6b55e7)[0x56301460f5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56301460f677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5630145fe56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56301460007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5630146dbafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5630146dbbdd] pthread_create.c:0(start_thread)[0x7f46a5d00ea5] /lib64/libc.so.6(clone+0x6d)[0x7f46a40a18cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f462c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 24 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:11:50 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:11:50 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:11:50 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:11:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:11:50 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:11:50 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:11:50 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:11:50 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:11:50 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:11:50 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136114109 2020-01-31 20:11:50 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 20:11:50 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:11:50 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:11:50 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:11:50 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:11:50 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:11:50 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:11:50 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136114118; transaction id 5900089 2020-01-31 20:11:50 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:11:50 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:11:50 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:11:50 0 [Note] Starting crash recovery... 2020-01-31 20:11:50 0 [Note] Crash recovery finished. 2020-01-31 20:11:50 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:11:50 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:11:50 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:11:50 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000061' at position 97117 2020-01-31 20:11:50 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:11:50 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:11:50 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:11:50 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:11:50 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:11:50 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:11:50 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:11:50 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:11:50 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:11:50 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:11:50 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:11:50 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:11:50 2020-01-31 20:11:50 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000061' at position 97117 2020-01-31 20:11:53 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000125, 203516) 2020-01-31 20:11:59 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 20:11:59 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000061' at position 99230 2020-01-31 20:11:59 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 20:12:04 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000127, 346) 2020-01-31 20:12:17 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:12:23 16 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:12:23 16 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:12:24 17 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:12:24 17 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 20:12:25 20 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 20:12:25 20 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 20:12:54 36 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 20:12:54 36 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 20:13:04 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000061' at position 99230 2020-01-31 20:13:06 43 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 20:13:06 43 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 20:13:15 47 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 20:13:15 47 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 20:13:23 52 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 20:13:23 52 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 20:13:24 54 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 20:13:24 54 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 20:13:28 58 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 20:13:28 58 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 20:13:34 62 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:13:34 62 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:15:34 109 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 20:15:34 109 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 20:15:38 116 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:15:38 116 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 20:15:38 116 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 20:15:38 116 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 20:15:38 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff998000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff99c58bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a447b63f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a4475fa3af] sigaction.c:0(__restore_rt)[0x7ffa08f98630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a447528151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a447666545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a44741cccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a447412d4b] /usr/sbin/mysqld(+0x6ca792)[0x55a44741d792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a4474456da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a44744765d] /usr/sbin/mysqld(+0x65e3f3)[0x55a4473b13f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a4473b0cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a447444b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a44744765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a447447761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a447448377] /usr/sbin/mysqld(+0x58fd84)[0x55a4472e2d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a4473efb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a447407b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a447407cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a4474085e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a447408677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a4473f756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a4473f907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a4474d4afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a4474d4bdd] pthread_create.c:0(start_thread)[0x7ffa08f90ea5] /lib64/libc.so.6(clone+0x6d)[0x7ffa073318cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff99800fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 117 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:15:45 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:15:45 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:15:45 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:15:45 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:15:45 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:15:45 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:15:45 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:15:45 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:15:45 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:15:45 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136119404 2020-01-31 20:15:45 0 [Note] InnoDB: Starting final batch to recover 4 pages from redo log. 2020-01-31 20:15:45 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000127', position 65531 2020-01-31 20:15:45 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:15:45 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:15:45 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:15:45 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:15:45 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:15:45 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:15:46 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136120117; transaction id 5900110 2020-01-31 20:15:46 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:15:46 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:15:46 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:15:46 0 [Note] Starting crash recovery... 2020-01-31 20:15:46 0 [Note] Crash recovery finished. 2020-01-31 20:15:46 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:15:46 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:15:46 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:15:46 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000062' at position 346 2020-01-31 20:15:46 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:15:46 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:15:46 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:15:46 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000062' at position 346 2020-01-31 20:15:46 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:15:46 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:15:46 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:15:46 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:15:46 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:15:46 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:15:46 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:15:46 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:15:47 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:15:47 2020-01-31 20:16:13 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:16:39 21 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000127, 537035) 2020-01-31 20:16:42 22 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 20:16:42 22 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 20:16:57 30 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:16:57 30 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:16:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f7724000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f7742de9cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x564640962f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5646403f93af] sigaction.c:0(__restore_rt)[0x7f77a51cc630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564640327151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564640465545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56464021bccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x564640211d4b] /usr/sbin/mysqld(+0x6ca792)[0x56464021c792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5646402446da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56464024665d] /usr/sbin/mysqld(+0x65e3f3)[0x5646401b03f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5646401afcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x564640243b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56464024665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x564640246761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x564640247377] /usr/sbin/mysqld(+0x58fd84)[0x5646400e1d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5646401eeb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564640206b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564640206cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5646402075e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564640207677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5646401f656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5646401f807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5646402d3afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5646402d3bdd] pthread_create.c:0(start_thread)[0x7f77a51c4ea5] /lib64/libc.so.6(clone+0x6d)[0x7f77a35658cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f772400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 31 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:17:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:17:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:17:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:17:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:17:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:17:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:17:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:17:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:17:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:17:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136123099 2020-01-31 20:17:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000127', position 65531 2020-01-31 20:17:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:17:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:17:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:17:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:17:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:17:03 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:17:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136123108; transaction id 5900121 2020-01-31 20:17:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:17:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:17:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:17:03 0 [Note] Starting crash recovery... 2020-01-31 20:17:03 0 [Note] Crash recovery finished. 2020-01-31 20:17:03 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:17:03 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:17:03 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:17:03 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000062' at position 128006 2020-01-31 20:17:03 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:17:03 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:17:03 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:17:03 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000062' at position 128006 2020-01-31 20:17:03 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:17:03 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:17:03 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:17:03 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:17:03 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:17:03 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:17:03 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:17:03 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:17:04 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:17:04 2020-01-31 20:17:28 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:17:29 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 20:17:29 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000062' at position 290623 2020-01-31 20:17:29 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 200131 20:17:29 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=1 max_threads=502 thread_count=9 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f5f74006bf8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f5fdd398cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b5eea34f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55b5ee4cb3af] sigaction.c:0(__restore_rt)[0x7f5ff8445630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55b5ee3f9151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55b5ee537545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55b5ee2edccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55b5ee2e3d4b] /usr/sbin/mysqld(+0x6ca792)[0x55b5ee2ee792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55b5ee3166da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b5ee31865d] /usr/sbin/mysqld(+0x65e3f3)[0x55b5ee2823f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55b5ee281cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55b5ee315b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b5ee31865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55b5ee318761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55b5ee319377] /usr/sbin/mysqld(+0x58fd84)[0x55b5ee1b3d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55b5ee2c0b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55b5ee2d8b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55b5ee2d8cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55b5ee2d95e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55b5ee2d9677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55b5ee2c856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55b5ee2ca07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55b5ee3a5afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55b5ee3a5bdd] pthread_create.c:0(start_thread)[0x7f5ff843dea5] /lib64/libc.so.6(clone+0x6d)[0x7f5ff67de8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f5f74011000): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 13 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:17:35 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:17:35 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:17:35 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:17:35 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:17:35 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:17:35 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:17:35 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:17:35 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:17:35 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:17:35 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136126002 2020-01-31 20:17:35 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000127', position 65531 2020-01-31 20:17:35 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:17:35 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:17:35 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:17:35 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:17:35 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:17:35 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:17:36 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136126011; transaction id 5900132 2020-01-31 20:17:36 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:17:36 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:17:36 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:17:36 0 [Note] Starting crash recovery... 2020-01-31 20:17:36 0 [Note] Crash recovery finished. 2020-01-31 20:17:36 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:17:36 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:17:36 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:17:36 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000062' at position 290623 2020-01-31 20:17:36 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:17:36 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:17:36 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:17:36 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:17:36 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:17:36 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:17:36 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000062' at position 290623 2020-01-31 20:17:36 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:17:36 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:17:36 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:17:36 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:17:36 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:17:37 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:17:37 2020-01-31 20:18:03 18 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:18:08 20 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:18:08 20 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:18:09 20 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:18:09 20 [Warning] Checking table: './wpms/wp_usermeta' 2020-01-31 20:18:10 21 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 20:18:10 21 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 20:18:13 22 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 20:18:13 22 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 20:18:36 32 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000128, 30351) 2020-01-31 20:18:54 40 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:18:54 40 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:19:43 73 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:19:43 73 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 20:19:43 73 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 20:19:43 73 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 20:19:43 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7feb20000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7feb8413fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a0c89c8f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a0c845f3af] sigaction.c:0(__restore_rt)[0x7feb9f13f630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a0c838d151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a0c84cb545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a0c8281ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a0c8277d4b] /usr/sbin/mysqld(+0x6ca792)[0x55a0c8282792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a0c82aa6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a0c82ac65d] /usr/sbin/mysqld(+0x65e3f3)[0x55a0c82163f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a0c8215cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a0c82a9b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a0c82ac65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a0c82ac761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a0c82ad377] /usr/sbin/mysqld(+0x58fd84)[0x55a0c8147d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a0c8254b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a0c826cb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a0c826ccd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a0c826d5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a0c826d677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a0c825c56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a0c825e07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a0c8339afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a0c8339bdd] pthread_create.c:0(start_thread)[0x7feb9f137ea5] /lib64/libc.so.6(clone+0x6d)[0x7feb9d4d88cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7feb2000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 74 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:19:49 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:19:49 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:19:49 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:19:49 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:19:49 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:19:49 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:19:49 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:19:49 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:19:49 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:19:49 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136128816 2020-01-31 20:19:49 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000127', position 65531 2020-01-31 20:19:49 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:19:49 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:19:49 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:19:49 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:19:49 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:19:49 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136128825; transaction id 5900143 2020-01-31 20:19:49 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:19:49 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:19:49 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:19:49 0 [Note] Starting crash recovery... 2020-01-31 20:19:49 0 [Note] Crash recovery finished. 2020-01-31 20:19:49 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:19:49 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:19:49 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:19:49 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000063' at position 346 2020-01-31 20:19:49 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:19:49 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:19:49 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:19:49 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000063' at position 346 2020-01-31 20:19:49 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:19:49 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:19:49 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:19:49 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:19:49 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:19:49 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:19:49 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:19:49 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:19:49 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:19:49 2020-01-31 20:20:19 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:20:20 13 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:20:20 13 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:20:22 16 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 20:20:22 16 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 20:20:40 35 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 20:20:40 35 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 20:20:44 37 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000130, 492001) 2020-01-31 20:20:52 41 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:20:52 41 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:21:13 53 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 20:21:13 53 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 20:21:35 62 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 20:21:35 62 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 20:21:44 72 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 20:21:44 72 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 20:21:57 79 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:21:57 79 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:21:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f87fc000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f8868054cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5618c9c91f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5618c97283af] sigaction.c:0(__restore_rt)[0x7f8871029630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5618c9656151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5618c9794545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5618c954accf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5618c9540d4b] /usr/sbin/mysqld(+0x6ca792)[0x5618c954b792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5618c95736da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5618c957565d] /usr/sbin/mysqld(+0x65e3f3)[0x5618c94df3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5618c94decb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5618c9572b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5618c957565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5618c9575761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5618c9576377] /usr/sbin/mysqld(+0x58fd84)[0x5618c9410d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5618c951db6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5618c9535b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5618c9535cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5618c95365e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5618c9536677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5618c952556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5618c952707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5618c9602afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5618c9602bdd] pthread_create.c:0(start_thread)[0x7f8871021ea5] /lib64/libc.so.6(clone+0x6d)[0x7f886f3c28cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f87fc00feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 80 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:22:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:22:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:22:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:22:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:22:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:22:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:22:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:22:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:22:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:22:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136132862 2020-01-31 20:22:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000131', position 308075 2020-01-31 20:22:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:22:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:22:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:22:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:22:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:22:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:22:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136132871; transaction id 5900166 2020-01-31 20:22:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:22:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:22:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:22:04 0 [Note] Starting crash recovery... 2020-01-31 20:22:04 0 [Note] Crash recovery finished. 2020-01-31 20:22:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:22:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:22:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:22:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000063' at position 454587 2020-01-31 20:22:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:22:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:22:04 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:22:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000063' at position 454587 2020-01-31 20:22:04 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:22:04 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:22:04 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:22:04 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:22:04 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:22:04 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:22:04 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:22:04 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:22:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:22:05 2020-01-31 20:22:29 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:22:47 23 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:22:47 23 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:22:47 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f7d1c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f7d7c092cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d155446f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55d154edd3af] sigaction.c:0(__restore_rt)[0x7f7d96da8630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55d154e0b151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55d154f49545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55d154cffccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55d154cf5d4b] /usr/sbin/mysqld(+0x6ca792)[0x55d154d00792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55d154d286da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d154d2a65d] /usr/sbin/mysqld(+0x65e3f3)[0x55d154c943f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55d154c93cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55d154d27b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d154d2a65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55d154d2a761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55d154d2b377] /usr/sbin/mysqld(+0x58fd84)[0x55d154bc5d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55d154cd2b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55d154ceab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55d154ceacd2] /usr/sbin/mysqld(+0x6b55e7)[0x55d154ceb5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55d154ceb677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55d154cda56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55d154cdc07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55d154db7afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d154db7bdd] pthread_create.c:0(start_thread)[0x7f7d96da0ea5] /lib64/libc.so.6(clone+0x6d)[0x7f7d951418cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f7d1c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 24 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:22:53 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:22:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:22:53 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:22:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:22:53 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:22:53 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:22:53 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:22:53 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:22:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:22:54 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136135697 2020-01-31 20:22:54 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000131', position 308075 2020-01-31 20:22:54 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:22:54 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:22:54 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:22:54 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:22:54 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:22:54 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:22:55 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136135706; transaction id 5900177 2020-01-31 20:22:55 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:22:55 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:22:55 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:22:55 0 [Note] Starting crash recovery... 2020-01-31 20:22:55 0 [Note] Crash recovery finished. 2020-01-31 20:22:55 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:22:55 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:22:55 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:22:55 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000063' at position 685957 2020-01-31 20:22:55 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:22:55 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:22:55 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000063' at position 685957 2020-01-31 20:22:55 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:22:55 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:22:55 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:22:55 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:22:55 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:22:55 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:22:55 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:22:55 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:22:55 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:22:56 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:22:56 2020-01-31 20:22:58 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000131, 523085) 2020-01-31 20:23:19 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:23:20 14 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 20:23:20 14 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 20:23:23 15 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 20:23:23 15 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 20:23:24 16 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 20:23:24 16 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 20:23:28 22 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 20:23:28 22 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 20:23:42 27 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:23:42 27 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:23:50 30 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 20:23:50 30 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 20:24:24 42 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:24:24 42 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:24:25 44 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:24:25 44 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 20:24:27 47 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 20:24:27 47 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 20:24:32 53 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 20:24:32 53 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 20:25:34 73 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 20:25:34 73 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 20:26:57 111 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:26:57 111 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:26:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fa028000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fa044747cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563021400f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x563020e973af] sigaction.c:0(__restore_rt)[0x7fa0a6aa8630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x563020dc5151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x563020f03545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x563020cb9ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x563020cafd4b] /usr/sbin/mysqld(+0x6ca792)[0x563020cba792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x563020ce26da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563020ce465d] /usr/sbin/mysqld(+0x65e3f3)[0x563020c4e3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x563020c4dcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x563020ce1b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563020ce465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x563020ce4761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x563020ce5377] /usr/sbin/mysqld(+0x58fd84)[0x563020b7fd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x563020c8cb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x563020ca4b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x563020ca4cd2] /usr/sbin/mysqld(+0x6b55e7)[0x563020ca55e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x563020ca5677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x563020c9456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x563020c9607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x563020d71afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x563020d71bdd] pthread_create.c:0(start_thread)[0x7fa0a6aa0ea5] /lib64/libc.so.6(clone+0x6d)[0x7fa0a4e418cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fa02800fe80): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 112 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:27:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:27:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:27:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:27:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:27:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:27:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:27:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:27:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:27:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:27:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136138544 2020-01-31 20:27:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000131', position 308075 2020-01-31 20:27:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:27:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:27:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:27:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:27:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:27:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:27:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136138553; transaction id 5900188 2020-01-31 20:27:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:27:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:27:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:27:04 0 [Note] Starting crash recovery... 2020-01-31 20:27:04 0 [Note] Crash recovery finished. 2020-01-31 20:27:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:27:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:27:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:27:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000063' at position 757619 2020-01-31 20:27:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:27:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:27:04 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:27:04 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:27:04 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:27:04 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:27:04 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:27:04 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:27:04 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:27:04 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:27:04 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:27:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000063' at position 757619 2020-01-31 20:27:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:27:05 2020-01-31 20:27:30 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:27:46 22 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:27:46 22 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:27:47 24 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:27:47 24 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:27:48 25 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:27:48 25 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 20:27:58 28 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000133, 516790) 2020-01-31 20:28:52 45 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 20:28:52 45 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 20:29:12 53 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 20:29:12 53 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 20:29:54 69 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 20:29:54 69 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 20:31:29 103 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 20:31:29 103 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 20:31:40 106 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 20:31:40 106 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 20:31:57 118 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:31:57 118 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:31:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9208000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f920c0a9cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560bfe158f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560bfdbef3af] sigaction.c:0(__restore_rt)[0x7f9227255630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x560bfdb1d151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x560bfdc5b545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x560bfda11ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x560bfda07d4b] /usr/sbin/mysqld(+0x6ca792)[0x560bfda12792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x560bfda3a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560bfda3c65d] /usr/sbin/mysqld(+0x65e3f3)[0x560bfd9a63f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x560bfd9a5cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x560bfda39b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560bfda3c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x560bfda3c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x560bfda3d377] /usr/sbin/mysqld(+0x58fd84)[0x560bfd8d7d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x560bfd9e4b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x560bfd9fcb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x560bfd9fccd2] /usr/sbin/mysqld(+0x6b55e7)[0x560bfd9fd5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x560bfd9fd677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x560bfd9ec56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x560bfd9ee07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x560bfdac9afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560bfdac9bdd] pthread_create.c:0(start_thread)[0x7f922724dea5] /lib64/libc.so.6(clone+0x6d)[0x7f92255ee8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f920800ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 119 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:32:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:32:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:32:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:32:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:32:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:32:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:32:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:32:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:32:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:32:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136141963 2020-01-31 20:32:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000131', position 308075 2020-01-31 20:32:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:32:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:32:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:32:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:32:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:32:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:32:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136141972; transaction id 5900199 2020-01-31 20:32:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:32:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:32:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:32:04 0 [Note] Starting crash recovery... 2020-01-31 20:32:04 0 [Note] Crash recovery finished. 2020-01-31 20:32:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:32:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:32:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:32:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000063' at position 885785 2020-01-31 20:32:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:32:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:32:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000063' at position 885785 2020-01-31 20:32:04 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:32:04 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:32:04 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:32:04 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:32:04 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:32:04 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:32:04 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:32:04 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:32:04 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:32:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:32:05 2020-01-31 20:32:31 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:32:33 18 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 20:32:33 18 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 20:32:47 24 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 20:32:47 24 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 20:32:59 37 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000134, 541533) 2020-01-31 20:33:00 38 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 20:33:00 38 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 20:33:04 41 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:33:04 41 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:33:15 46 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 20:33:15 46 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 20:33:24 52 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 20:33:24 52 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 20:33:28 58 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 20:33:28 58 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 20:33:28 59 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:33:28 59 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:33:29 59 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:33:29 59 [Warning] Checking table: './wpms/wp_usermeta' 2020-01-31 20:34:32 102 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 20:34:32 102 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 20:35:34 128 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 20:35:34 128 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 20:36:34 149 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 20:36:34 149 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 20:36:57 161 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:36:57 161 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:36:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fdd18000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fdd6c14dcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5628ad1e2f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5628acc793af] sigaction.c:0(__restore_rt)[0x7fdd87601630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5628acba7151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5628acce5545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5628aca9bccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5628aca91d4b] /usr/sbin/mysqld(+0x6ca792)[0x5628aca9c792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5628acac46da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5628acac665d] /usr/sbin/mysqld(+0x65e3f3)[0x5628aca303f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5628aca2fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5628acac3b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5628acac665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5628acac6761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5628acac7377] /usr/sbin/mysqld(+0x58fd84)[0x5628ac961d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5628aca6eb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5628aca86b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5628aca86cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5628aca875e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5628aca87677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5628aca7656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5628aca7807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5628acb53afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5628acb53bdd] pthread_create.c:0(start_thread)[0x7fdd875f9ea5] /lib64/libc.so.6(clone+0x6d)[0x7fdd8599a8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fdd1800fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 162 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:37:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:37:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:37:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:37:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:37:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:37:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:37:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:37:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:37:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:37:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136145069 2020-01-31 20:37:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000131', position 308075 2020-01-31 20:37:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:37:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:37:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:37:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:37:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:37:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:37:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136145078; transaction id 5900211 2020-01-31 20:37:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:37:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:37:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:37:04 0 [Note] Starting crash recovery... 2020-01-31 20:37:04 0 [Note] Crash recovery finished. 2020-01-31 20:37:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:37:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:37:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:37:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000063' at position 1025347 2020-01-31 20:37:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:37:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:37:04 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:37:04 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:37:04 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:37:04 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:37:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000063' at position 1025347 2020-01-31 20:37:04 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:37:04 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:37:04 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:37:04 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:37:04 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:37:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:37:05 2020-01-31 20:37:29 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 20:37:29 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000063' at position 1139082 2020-01-31 20:37:29 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 20:37:32 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:37:32 18 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 20:37:32 18 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 20:37:32 19 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:37:32 19 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:37:35 21 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000135, 807043) 2020-01-31 20:38:09 32 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:38:09 32 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:38:10 33 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:38:10 33 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 20:38:16 37 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 20:38:16 37 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 20:38:29 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000063' at position 1139082 2020-01-31 20:38:57 63 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:38:57 63 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 20:38:57 63 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 20:38:57 63 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 20:38:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fdf28000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fdf8458bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5616d3828f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5616d32bf3af] sigaction.c:0(__restore_rt)[0x7fdf9f73b630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5616d31ed151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5616d332b545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5616d30e1ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5616d30d7d4b] /usr/sbin/mysqld(+0x6ca792)[0x5616d30e2792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5616d310a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5616d310c65d] /usr/sbin/mysqld(+0x65e3f3)[0x5616d30763f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5616d3075cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5616d3109b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5616d310c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5616d310c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5616d310d377] /usr/sbin/mysqld(+0x58fd84)[0x5616d2fa7d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5616d30b4b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5616d30ccb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5616d30cccd2] /usr/sbin/mysqld(+0x6b55e7)[0x5616d30cd5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5616d30cd677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5616d30bc56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5616d30be07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5616d3199afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5616d3199bdd] pthread_create.c:0(start_thread)[0x7fdf9f733ea5] /lib64/libc.so.6(clone+0x6d)[0x7fdf9dad48cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fdf2800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 64 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:39:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:39:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:39:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:39:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:39:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:39:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:39:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:39:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:39:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:39:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136149226 2020-01-31 20:39:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000131', position 308075 2020-01-31 20:39:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:39:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:39:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:39:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:39:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:39:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:39:05 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136149235; transaction id 5900226 2020-01-31 20:39:05 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:39:05 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:39:05 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:39:05 0 [Note] Starting crash recovery... 2020-01-31 20:39:05 0 [Note] Crash recovery finished. 2020-01-31 20:39:05 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:39:05 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:39:05 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:39:05 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000064' at position 346 2020-01-31 20:39:05 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:39:05 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:39:05 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:39:05 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:39:05 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:39:05 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:39:05 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:39:05 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:39:05 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:39:05 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:39:05 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:39:06 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000064' at position 346 2020-01-31 20:39:07 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:39:07 2020-01-31 20:39:32 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:39:32 14 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 20:39:32 14 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 20:39:32 15 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:39:32 15 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:39:36 18 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:39:36 18 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:39:36 18 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:39:36 18 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 20:39:58 28 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000136, 141143) 2020-01-31 20:40:06 31 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 20:40:06 31 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 20:40:08 33 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 20:40:08 33 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 20:41:37 82 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 20:41:37 82 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 20:41:41 85 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 20:41:41 85 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 20:41:57 94 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:41:57 94 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:41:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fdb1c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fdb74ff4cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a844714f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a8441ab3af] sigaction.c:0(__restore_rt)[0x7fdb9015c630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a8440d9151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a844217545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a843fcdccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a843fc3d4b] /usr/sbin/mysqld(+0x6ca792)[0x55a843fce792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a843ff66da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a843ff865d] /usr/sbin/mysqld(+0x65e3f3)[0x55a843f623f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a843f61cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a843ff5b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a843ff865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a843ff8761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a843ff9377] /usr/sbin/mysqld(+0x58fd84)[0x55a843e93d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a843fa0b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a843fb8b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a843fb8cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a843fb95e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a843fb9677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a843fa856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a843faa07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a844085afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a844085bdd] pthread_create.c:0(start_thread)[0x7fdb90154ea5] /lib64/libc.so.6(clone+0x6d)[0x7fdb8e4f58cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fdb1c00ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 95 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:42:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:42:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:42:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:42:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:42:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:42:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:42:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:42:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:42:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:42:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136152729 2020-01-31 20:42:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000137', position 72439 2020-01-31 20:42:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:42:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:42:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:42:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:42:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:42:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:42:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136152738; transaction id 5900244 2020-01-31 20:42:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:42:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:42:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:42:04 0 [Note] Starting crash recovery... 2020-01-31 20:42:04 0 [Note] Crash recovery finished. 2020-01-31 20:42:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:42:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:42:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:42:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000064' at position 943 2020-01-31 20:42:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:42:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:42:04 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:42:04 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:42:04 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:42:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000064' at position 943 2020-01-31 20:42:04 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:42:04 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:42:04 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:42:04 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:42:04 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:42:04 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:42:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:42:05 2020-01-31 20:42:32 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:42:51 22 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:42:51 22 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:42:58 27 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000137, 713541) 2020-01-31 20:43:00 29 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 20:43:00 29 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 20:43:12 36 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:43:12 36 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:43:12 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc5340009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc594403cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a981657f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a9810ee3af] sigaction.c:0(__restore_rt)[0x7fc5b1578630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a98101c151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a98115a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a980f10ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a980f06d4b] /usr/sbin/mysqld(+0x6ca792)[0x55a980f11792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a980f396da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a980f3b65d] /usr/sbin/mysqld(+0x65e3f3)[0x55a980ea53f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a980ea4cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a980f38b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a980f3b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a980f3b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a980f3c377] /usr/sbin/mysqld(+0x58fd84)[0x55a980dd6d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a980ee3b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a980efbb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a980efbcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a980efc5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a980efc677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a980eeb56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a980eed07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a980fc8afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a980fc8bdd] pthread_create.c:0(start_thread)[0x7fc5b1570ea5] /lib64/libc.so.6(clone+0x6d)[0x7fc5af9118cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc53400fe00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 37 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:43:18 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:43:18 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:43:18 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:43:18 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:43:18 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:43:18 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:43:18 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:43:18 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:43:18 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:43:18 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136155716 2020-01-31 20:43:18 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000137', position 72439 2020-01-31 20:43:18 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:43:18 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:43:18 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:43:18 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:43:18 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:43:18 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:43:19 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136155725; transaction id 5900255 2020-01-31 20:43:19 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:43:19 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:43:19 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:43:19 0 [Note] Starting crash recovery... 2020-01-31 20:43:19 0 [Note] Crash recovery finished. 2020-01-31 20:43:19 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:43:19 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:43:19 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:43:19 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000064' at position 67420 2020-01-31 20:43:19 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:43:19 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:43:19 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:43:19 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:43:19 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:43:19 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:43:19 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:43:19 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:43:19 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:43:19 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:43:19 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:43:19 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:43:19 2020-01-31 20:43:20 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000064' at position 67420 2020-01-31 20:43:48 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:43:50 16 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 20:43:50 16 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 20:43:52 17 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:43:52 17 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:43:54 20 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:43:54 20 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:43:56 21 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 20:43:56 21 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 20:44:13 33 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000138, 34430) 2020-01-31 20:44:32 41 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 20:44:32 41 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 20:45:14 62 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 20:45:14 62 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 20:45:34 78 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 20:45:34 78 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 20:45:44 84 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:45:44 84 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:45:45 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f1b90000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f1baff68cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563a2a772f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x563a2a2093af] sigaction.c:0(__restore_rt)[0x7f1c1114d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x563a2a137151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x563a2a275545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x563a2a02bccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x563a2a021d4b] /usr/sbin/mysqld(+0x6ca792)[0x563a2a02c792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x563a2a0546da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563a2a05665d] /usr/sbin/mysqld(+0x65e3f3)[0x563a29fc03f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x563a29fbfcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x563a2a053b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563a2a05665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x563a2a056761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x563a2a057377] /usr/sbin/mysqld(+0x58fd84)[0x563a29ef1d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x563a29ffeb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x563a2a016b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x563a2a016cd2] /usr/sbin/mysqld(+0x6b55e7)[0x563a2a0175e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x563a2a017677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x563a2a00656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x563a2a00807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x563a2a0e3afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x563a2a0e3bdd] pthread_create.c:0(start_thread)[0x7f1c11145ea5] /lib64/libc.so.6(clone+0x6d)[0x7f1c0f4e68cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f1b9000fe70): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 85 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:45:51 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:45:51 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:45:51 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:45:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:45:51 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:45:51 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:45:51 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:45:51 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:45:51 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:45:51 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136160157 2020-01-31 20:45:51 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000139', position 336348 2020-01-31 20:45:51 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:45:51 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:45:51 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:45:51 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:45:51 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:45:51 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:45:51 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136160166; transaction id 5900279 2020-01-31 20:45:51 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:45:51 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:45:51 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:45:51 0 [Note] Starting crash recovery... 2020-01-31 20:45:51 0 [Note] Crash recovery finished. 2020-01-31 20:45:51 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:45:51 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:45:51 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:45:51 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000064' at position 158743 2020-01-31 20:45:51 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:45:51 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:45:51 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000064' at position 158743 2020-01-31 20:45:51 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:45:51 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:45:51 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:45:51 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:45:51 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:45:51 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:45:51 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:45:51 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:45:51 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:45:52 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:45:52 2020-01-31 20:46:18 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:46:20 15 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:46:20 15 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:46:33 20 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 20:46:33 20 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 20:46:46 27 [ERROR] mysqld: Table './pizzahut/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:46:46 27 [Warning] Checking table: './pizzahut/wp_usermeta' 2020-01-31 20:46:51 29 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000139, 379041) 2020-01-31 20:46:53 32 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 20:46:53 32 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 20:46:57 35 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:46:57 35 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:46:58 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9764000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f9781522cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c26f414f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c26eeab3af] sigaction.c:0(__restore_rt)[0x7f97e26e7630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c26edd9151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c26ef17545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c26eccdccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c26ecc3d4b] /usr/sbin/mysqld(+0x6ca792)[0x55c26ecce792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c26ecf66da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c26ecf865d] /usr/sbin/mysqld(+0x65e3f3)[0x55c26ec623f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c26ec61cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c26ecf5b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c26ecf865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c26ecf8761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c26ecf9377] /usr/sbin/mysqld(+0x58fd84)[0x55c26eb93d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c26eca0b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c26ecb8b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c26ecb8cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c26ecb95e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c26ecb9677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c26eca856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c26ecaa07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c26ed85afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c26ed85bdd] pthread_create.c:0(start_thread)[0x7f97e26dfea5] /lib64/libc.so.6(clone+0x6d)[0x7f97e0a808cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f976400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 36 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:47:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:47:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:47:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:47:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:47:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:47:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:47:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:47:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:47:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:47:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136163472 2020-01-31 20:47:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000140', position 240883 2020-01-31 20:47:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:47:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:47:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:47:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:47:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:47:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:47:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136163481; transaction id 5900295 2020-01-31 20:47:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:47:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:47:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:47:04 0 [Note] Starting crash recovery... 2020-01-31 20:47:04 0 [Note] Crash recovery finished. 2020-01-31 20:47:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:47:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:47:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:47:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000064' at position 288244 2020-01-31 20:47:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:47:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:47:04 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:47:04 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:47:04 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:47:04 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:47:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000064' at position 288244 2020-01-31 20:47:04 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:47:04 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:47:04 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:47:04 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:47:04 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:47:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:47:05 2020-01-31 20:47:33 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:47:43 19 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:47:43 19 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:47:55 26 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 20:47:55 26 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 20:47:55 27 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:47:55 27 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:47:55 25 [ERROR] mysqld: Table './pizzahut/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:47:55 25 [Warning] Checking table: './pizzahut/wp_usermeta' 2020-01-31 20:47:58 31 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000140, 243055) 2020-01-31 20:51:13 119 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 20:51:13 119 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 20:51:16 123 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 20:51:16 123 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 20:51:40 147 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 20:51:40 147 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 20:51:57 163 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:51:57 163 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:51:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=8 max_threads=502 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0728000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f06c05e2cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x562180484f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56217ff1b3af] sigaction.c:0(__restore_rt)[0x7f0731141630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56217fe49151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56217ff87545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56217fd3dccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56217fd33d4b] /usr/sbin/mysqld(+0x6ca792)[0x56217fd3e792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56217fd666da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56217fd6865d] /usr/sbin/mysqld(+0x65e3f3)[0x56217fcd23f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56217fcd1cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56217fd65b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56217fd6865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56217fd68761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56217fd69377] /usr/sbin/mysqld(+0x58fd84)[0x56217fc03d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56217fd10b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56217fd28b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56217fd28cd2] /usr/sbin/mysqld(+0x6b55e7)[0x56217fd295e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56217fd29677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56217fd1856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56217fd1a07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56217fdf5afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56217fdf5bdd] pthread_create.c:0(start_thread)[0x7f0731139ea5] /lib64/libc.so.6(clone+0x6d)[0x7f072f4da8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f072800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 164 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:52:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:52:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:52:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:52:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:52:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:52:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:52:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:52:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:52:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:52:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136166813 2020-01-31 20:52:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000140', position 240883 2020-01-31 20:52:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:52:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:52:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:52:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:52:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:52:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:52:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136166822; transaction id 5900306 2020-01-31 20:52:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:52:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:52:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:52:04 0 [Note] Starting crash recovery... 2020-01-31 20:52:04 0 [Note] Crash recovery finished. 2020-01-31 20:52:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:52:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:52:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:52:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000064' at position 288244 2020-01-31 20:52:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:52:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:52:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000064' at position 288244 2020-01-31 20:52:04 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:52:04 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:52:04 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:52:04 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:52:04 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:52:04 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:52:04 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:52:04 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:52:04 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:52:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:52:05 2020-01-31 20:52:29 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:52:30 18 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 20:52:30 18 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 20:52:32 19 [ERROR] mysqld: Table './pizzahut/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:52:32 19 [Warning] Checking table: './pizzahut/wp_usermeta' 2020-01-31 20:53:00 32 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000141, 534539) 2020-01-31 20:53:00 33 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 20:53:00 33 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 20:53:02 36 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 20:53:02 36 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 20:53:04 38 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:53:04 38 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:53:05 40 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:53:05 40 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 20:53:18 46 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 20:53:18 46 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 20:53:23 50 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 20:53:23 50 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 20:53:58 76 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:53:58 76 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:53:59 76 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:53:59 76 [Warning] Checking table: './wpms/wp_usermeta' 2020-01-31 20:54:28 99 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 20:54:28 99 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 20:54:32 102 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 20:54:32 102 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 20:55:34 133 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 20:55:34 133 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 20:56:33 176 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 20:56:33 176 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 20:56:57 192 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 20:56:57 192 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 20:56:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f6e940009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f6f08067cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55da5d39ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55da5ce353af] sigaction.c:0(__restore_rt)[0x7f6f0dbfa630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55da5cd63151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55da5cea1545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55da5cc57ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55da5cc4dd4b] /usr/sbin/mysqld(+0x6ca792)[0x55da5cc58792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55da5cc806da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55da5cc8265d] /usr/sbin/mysqld(+0x65e3f3)[0x55da5cbec3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55da5cbebcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55da5cc7fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55da5cc8265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55da5cc82761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55da5cc83377] /usr/sbin/mysqld(+0x58fd84)[0x55da5cb1dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55da5cc2ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55da5cc42b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55da5cc42cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55da5cc435e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55da5cc43677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55da5cc3256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55da5cc3407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55da5cd0fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55da5cd0fbdd] pthread_create.c:0(start_thread)[0x7f6f0dbf2ea5] /lib64/libc.so.6(clone+0x6d)[0x7f6f0bf938cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f6e9400fde0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 193 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 20:57:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 20:57:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 20:57:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 20:57:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 20:57:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 20:57:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 20:57:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 20:57:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 20:57:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 20:57:05 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136171067 2020-01-31 20:57:05 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000142', position 1237971 2020-01-31 20:57:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 20:57:05 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 20:57:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 20:57:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 20:57:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 20:57:05 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 20:57:05 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136171076; transaction id 5900329 2020-01-31 20:57:05 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 20:57:05 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 20:57:05 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 20:57:05 0 [Note] Starting crash recovery... 2020-01-31 20:57:05 0 [Note] Crash recovery finished. 2020-01-31 20:57:05 0 [Note] Server socket created on IP: '::'. 2020-01-31 20:57:05 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 20:57:05 0 [Note] Added new Master_info '' to hash table 2020-01-31 20:57:05 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000064' at position 452534 2020-01-31 20:57:05 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 20:57:05 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 20:57:05 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 20:57:05 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000064' at position 452534 2020-01-31 20:57:05 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 20:57:05 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 20:57:05 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 20:57:05 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 20:57:05 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 20:57:05 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 20:57:05 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 20:57:05 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 20:57:06 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 20:57:06 2020-01-31 20:57:30 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 20:57:31 13 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 20:57:31 13 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 20:57:31 13 [ERROR] mysqld: Table './pizzahut/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:57:31 13 [Warning] Checking table: './pizzahut/wp_usermeta' 2020-01-31 20:57:35 15 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 20:57:35 15 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 20:57:36 15 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 20:57:36 15 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-01-31 20:57:48 20 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 20:57:48 20 [Warning] Checking table: './wpms/wp_options' 2020-01-31 20:57:54 25 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 20:57:54 25 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 20:57:55 27 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 20:57:55 27 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 20:57:58 29 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000142, 1239079) 2020-01-31 20:58:23 42 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 200131 21:01:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f46b0006bf8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f471c3d2cf0 thread_stack 0x49000 *** buffer overflow detected ***: /usr/sbin/mysqld terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7f4735943567] /lib64/libc.so.6(+0x1166e2)[0x7f47359416e2] /lib64/libc.so.6(+0x1184c7)[0x7f47359434c7] /usr/sbin/mysqld(my_addr_resolve+0xda)[0x5648c92ff94a] /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x5648c92e80c2] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5648c8d7e3af] /lib64/libpthread.so.0(+0xf630)[0x7f4737590630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5648c8cac151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5648c8dea545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5648c8ba0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5648c8b96d4b] /usr/sbin/mysqld(+0x6ca792)[0x5648c8ba1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5648c8bc96da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5648c8bcb65d] /usr/sbin/mysqld(+0x65e3f3)[0x5648c8b353f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5648c8b34cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5648c8bc8b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5648c8bcb65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5648c8bcb761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5648c8bcc377] /usr/sbin/mysqld(+0x58fd84)[0x5648c8a66d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5648c8b73b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5648c8b8bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5648c8b8bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x5648c8b8c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5648c8b8c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5648c8b7b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5648c8b7d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5648c8c58afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5648c8c58bdd] /lib64/libpthread.so.0(+0x7ea5)[0x7f4737588ea5] /lib64/libc.so.6(clone+0x6d)[0x7f47359298cd] ======= Memory map: ======== 5648c84d7000-5648c9925000 r-xp 00000000 fd:00 2101454 /usr/sbin/mysqld 5648c9b24000-5648c9c50000 r--p 0144d000 fd:00 2101454 /usr/sbin/mysqld 5648c9c50000-5648c9d0a000 rw-p 01579000 fd:00 2101454 /usr/sbin/mysqld 5648c9d0a000-5648ca5b9000 rw-p 00000000 00:00 0 5648caa9d000-5648cf041000 rw-p 00000000 00:00 0 [heap] 7f4690000000-7f469002a000 rw-p 00000000 00:00 0 7f469002a000-7f4694000000 ---p 00000000 00:00 0 7f4694000000-7f4694021000 rw-p 00000000 00:00 0 7f4694021000-7f4698000000 ---p 00000000 00:00 0 7f4698000000-7f4698029000 rw-p 00000000 00:00 0 7f4698029000-7f469c000000 ---p 00000000 00:00 0 7f469c000000-7f469c021000 rw-p 00000000 00:00 0 7f469c021000-7f46a0000000 ---p 00000000 00:00 0 7f46a0000000-7f46a012a000 rw-p 00000000 00:00 0 7f46a012a000-7f46a4000000 ---p 00000000 00:00 0 7f46a4000000-7f46a4021000 rw-p 00000000 00:00 0 7f46a4021000-7f46a8000000 ---p 00000000 00:00 0 7f46a8000000-7f46a85fd000 rw-p 00000000 00:00 0 7f46a85fd000-7f46ac000000 ---p 00000000 00:00 0 7f46ac000000-7f46ac021000 rw-p 00000000 00:00 0 7f46ac021000-7f46b0000000 ---p 00000000 00:00 0 7f46b0000000-7f46b0515000 rw-p 00000000 00:00 0 7f46b0515000-7f46b4000000 ---p 00000000 00:00 0 7f46b4000000-7f46b44fa000 rw-p 00000000 00:00 0 7f46b44fa000-7f46b8000000 ---p 00000000 00:00 0 7f46b8000000-7f46ba3a0000 rw-p 00000000 00:00 0 7f46ba3a0000-7f46bc000000 ---p 00000000 00:00 0 7f46be7fd000-7f46be7fe000 ---p 00000000 00:00 0 7f46be7fe000-7f46beffe000 rw-p 00000000 00:00 0 7f46beffe000-7f46befff000 ---p 00000000 00:00 0 7f46befff000-7f46bf7ff000 rw-p 00000000 00:00 0 7f46bf7ff000-7f46bf800000 ---p 00000000 00:00 0 7f46bf800000-7f46c0000000 rw-p 00000000 00:00 0 7f46c0000000-7f46c0438000 rw-p 00000000 00:00 0 7f46c0438000-7f46c4000000 ---p 00000000 00:00 0 7f46c47f9000-7f46c47fa000 ---p 00000000 00:00 0 7f46c47fa000-7f46c4ffa000 rw-p 00000000 00:00 0 7f46c4ffa000-7f46c4ffb000 ---p 00000000 00:00 0 7f46c4ffb000-7f46c57fb000 rw-p 00000000 00:00 0 7f46c57fb000-7f46c57fc000 ---p 00000000 00:00 0 7f46c57fc000-7f46c5ffc000 rw-p 00000000 00:00 0 7f46c5ffc000-7f46c5ffd000 ---p 00000000 00:00 0 7f46c5ffd000-7f46c67fd000 rw-p 00000000 00:00 0 7f46c67fd000-7f46c67fe000 ---p 00000000 00:00 0 7f46c67fe000-7f46c6ffe000 rw-p 00000000 00:00 0 7f46c6ffe000-7f46c6fff000 ---p 00000000 00:00 0 7f46c6fff000-7f46c77ff000 rw-p 00000000 00:00 0 7f46c77ff000-7f46c7800000 ---p 00000000 00:00 0 7f46c7800000-7f46c8000000 rw-p 00000000 00:00 0 7f46c8000000-7f46c8021000 rw-p 00000000 00:00 0 7f46c8021000-7f46cc000000 ---p 00000000 00:00 0 7f46cc1c8000-7f46cc1da000 r-xp 00000000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f46cc1da000-7f46cc3d9000 ---p 00012000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f46cc3d9000-7f46cc3dc000 r--p 00011000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f46cc3dc000-7f46cc3dd000 rw-p 00014000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f46cc3dd000-7f46cc3e3000 r-xp 00000000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f46cc3e3000-7f46cc5e2000 ---p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f46cc5e2000-7f46cc5e3000 r--p 00005000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f46cc5e3000-7f46cc5e4000 rw-p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f46cc5e4000-7f46cc5f0000 r-xp 00000000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f46cc5f0000-7f46cc7ef000 ---p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f46cc7ef000-7f46cc7f0000 r--p 0000b000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f46cc7f0000-7f46cc7f1000 rw-p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f46cc7f1000-7f46cc7f7000 rw-p 00000000 00:00 0 7f46cc7f7000-7f46cc7f8000 ---p 00000000 00:00 0 7f46cc7f8000-7f46ccff8000 rw-p 00000000 00:00 0 7f46ccff8000-7f46ccff9000 ---p 00000000 00:00 0 7f46ccff9000-7f46cd9fe000 rw-p 00000000 00:00 0 7f46cd9fe000-7f46cd9ff000 ---p 00000000 00:00 0 7f46cd9ff000-7f46ce1ff000 rw-p 00000000 00:00 0 7f46ce1ff000-7f46ce200000 ---p 00000000 00:00 0 7f46ce200000-7f46cea00000 rw-p 00000000 00:00 0 7f46cea00000-7f46cea01000 ---p 00000000 00:00 0 7f46cea01000-7f46cf201000 rw-p 00000000 00:00 0 7f46cf201000-7f46cf202000 ---p 00000000 00:00 0 7f46cf202000-7f46cfa02000 rw-p 00000000 00:00 0 7f46cfa02000-7f46cfa03000 ---p 00000000 00:00 0 7f46cfa03000-7f46d0203000 rw-p 00000000 00:00 0 7f46d0203000-7f46d0204000 ---p 00000000 00:00 0 7f46d0204000-7f46d0a04000 rw-p 00000000 00:00 0 7f46d0a04000-7f46d0a05000 ---p 00000000 00:00 0 7f46d0a05000-7f46d1205000 rw-p 00000000 00:00 0 7f46d1205000-7f46d1206000 ---p 00000000 00:00 0 7f46d1206000-7f46d1a06000 rw-p 00000000 00:00 0 7f46d1a06000-7f46d1a07000 ---p 00000000 00:00 0 7f46d1a07000-7f46d2207000 rw-p 00000000 00:00 0 7f46d2207000-7f46d2208000 ---p 00000000 00:00 0 7f46d2208000-7f46d2a08000 rw-p 00000000 00:00 0 7f46d2a08000-7f46d2a09000 ---p 00000000 00:00 0 7f46d2a09000-7f46d3209000 rw-p 00000000 00:00 0 7f46d3209000-7f46d320a000 ---p 00000000 00:00 0 7f46d320a000-7f46d3a0a000 rw-p 00000000 00:00 0 7f46d3a0a000-7f46d3a0b000 ---p 00000000 00:00 0 7f46d3a0b000-7f46d420b000 rw-p 00000000 00:00 0 7f46d420b000-7f46d420c000 ---p 00000000 00:00 0 7f46d420c000-7f46d4f1e000 rw-p 00000000 00:00 0 7f46d4ffe000-7f46d4fff000 ---p 00000000 00:00 0 7f46d4fff000-7f46d57ff000 rw-p 00000000 00:00 0 7f46d57ff000-7f46d5800000 ---p 00000000 00:00 0 7f46d5800000-7f46d6000000 rw-p 00000000 00:00 0 7f46d6000000-7f4718000000 rw-p 00000000 00:00 0 7f4718000000-7f4718021000 rw-p 00000000 00:00 0 7f4718021000-7f471c000000 ---p 00000000 00:00 0 7f471c0c2000-7f471c34b000 rw-p 00000000 00:00 0 7f471c389000-7f471c38a000 ---p 00000000 00:00 0 7f471c38a000-7f471c3d4000 rw-p 00000000 00:00 0 7f471c3d4000-7f471c3d5000 ---p 00000000 00:00 0 7f471c3d5000-7f471c41f000 rw-p 00000000 00:00 0 7f471c41f000-7f471c420000 ---p 00000000 00:00 0 7f471c420000-7f471c46a000 rw-p 00000000 00:00 0 7f471c46a000-7f471c46b000 ---p 00000000 00:00 0 7f471c46b000-7f471c4b5000 rw-p 00000000 00:00 0 7f471c4b5000-7f471c4b6000 ---p 00000000 00:00 0 7f471c4b6000-7f471c500000 rw-p 00000000 00:00 0 7f471c500000-7f471c501000 ---p 00000000 00:00 0 7f471c501000-7f4724000000 rw-p 00000000 00:00 0 7f4724000000-7f4724021000 rw-p 00000000 00:00 0 7f4724021000-7f4728000000 ---p 00000000 00:00 0 7f4728012000-7f4728013000 ---p 00000000 00:00 0 7f4728013000-7f472805d000 rw-p 00000000 00:00 0 7f472805d000-7f472805e000 ---p 00000000 00:00 0 7f472805e000-7f4729591000 rw-p 00000000 00:00 0 7f4729591000-7f4729592000 ---p 00000000 00:00 0 7f4729592000-7f47324c8000 rw-p 00000000 00:00 0 7f47324c8000-7f47324c9000 ---p 00000000 00:00 0 7f47324c9000-7f4732cc9000 rw-p 00000000 00:00 0 7f4732cc9000-7f4732ccc000 r-xp 00000000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f4732ccc000-7f4732ecb000 ---p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f4732ecb000-7f4732ecc000 r--p 00002000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f4732ecc000-7f4732ecd000 rw-p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f4732ecd000-7f4732edb000 r-xp 00000000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f4732edb000-7f47330db000 ---p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f47330db000-7f47330dc000 r--p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f47330dc000-7f47330dd000 rw-p 0000f000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f47330dd000-7f47330ec000 r-xp 00000000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f47330ec000-7f47332eb000 ---p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f47332eb000-7f47332ec000 r--p 0000e000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f47332ec000-7f47332ed000 rw-p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f47332ed000-7f4733304000 r-xp 00000000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f4733304000-7f4733503000 ---p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f4733503000-7f4733504000 r--p 00016000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f4733504000-7f4733505000 rw-p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f4733505000-7f4733565000 r-xp 00000000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f4733565000-7f4733765000 ---p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f4733765000-7f4733766000 r--p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f4733766000-7f4733767000 rw-p 00061000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f4733767000-7f473376b000 r-xp 00000000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f473376b000-7f473396a000 ---p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f473396a000-7f473396b000 r--p 00003000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f473396b000-7f473396c000 rw-p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f473396c000-7f473399d000 r-xp 00000000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f473399d000-7f4733b9c000 ---p 00031000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f4733b9c000-7f4733b9e000 r--p 00030000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f4733b9e000-7f4733b9f000 rw-p 00032000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f4733b9f000-7f4733ba2000 r-xp 00000000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f4733ba2000-7f4733da1000 ---p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f4733da1000-7f4733da2000 r--p 00002000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f4733da2000-7f4733da3000 rw-p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f4733da3000-7f4733e7c000 r-xp 00000000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f4733e7c000-7f473407b000 ---p 000d9000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f473407b000-7f4734089000 r--p 000d8000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f4734089000-7f473408c000 rw-p 000e6000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f473408c000-7f47340d6000 r-xp 00000000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f47340d6000-7f47342d6000 ---p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f47342d6000-7f47342d7000 r--p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f47342d7000-7f47342d9000 rw-p 0004b000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f47342d9000-7f47342ee000 r-xp 00000000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f47342ee000-7f47344ed000 ---p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f47344ed000-7f47344ee000 r--p 00014000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f47344ee000-7f47344ef000 rw-p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f47344ef000-7f473453d000 r-xp 00000000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f473453d000-7f473473d000 ---p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f473473d000-7f473473f000 r--p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f473473f000-7f4734740000 rw-p 00050000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f4734740000-7f4734756000 r-xp 00000000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f4734756000-7f4734955000 ---p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f4734955000-7f4734956000 r--p 00015000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f4734956000-7f4734957000 rw-p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f4734957000-7f4734959000 rw-p 00000000 00:00 0 7f4734959000-7f473495d000 r-xp 00000000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f473495d000-7f4734b5c000 ---p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f4734b5c000-7f4734b5d000 r--p 00003000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f4734b5d000-7f4734b5e000 rw-p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f4734b5e000-7f4734bdb000 r-xp 00000000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f4734bdb000-7f4734dda000 ---p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f4734dda000-7f4734ddb000 r--p 0007c000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f4734ddb000-7f4734dde000 rw-p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f4734dde000-7f4734ddf000 rw-p 00000000 00:00 0 7f4734ddf000-7f4734df3000 r-xp 00000000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f4734df3000-7f4734ff2000 ---p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f4734ff2000-7f4734ff3000 r--p 00013000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f4734ff3000-7f4734ff4000 rw-p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f4734ff4000-7f4735018000 r-xp 00000000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f4735018000-7f4735217000 ---p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f4735217000-7f4735218000 r--p 00023000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f4735218000-7f4735219000 rw-p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f4735219000-7f473521b000 rw-p 00000000 00:00 0 7f473521b000-7f4735222000 r-xp 00000000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f4735222000-7f4735421000 ---p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f4735421000-7f4735422000 r--p 00006000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f4735422000-7f4735423000 rw-p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f4735423000-7f4735427000 r-xp 00000000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f4735427000-7f4735626000 ---p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f4735626000-7f4735627000 r--p 00003000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f4735627000-7f4735628000 rw-p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f4735628000-7f473562a000 r-xp 00000000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f473562a000-7f4735829000 ---p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f4735829000-7f473582a000 r--p 00001000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f473582a000-7f473582b000 rw-p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f473582b000-7f47359ee000 r-xp 00000000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f47359ee000-7f4735bee000 ---p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f4735bee000-7f4735bf2000 r--p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f4735bf2000-7f4735bf4000 rw-p 001c7000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f4735bf4000-7f4735bf9000 rw-p 00000000 00:00 0 7f4735bf9000-7f4735cfa000 r-xp 00000000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f4735cfa000-7f4735ef9000 ---p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f4735ef9000-7f4735efa000 r--p 00100000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f4735efa000-7f4735efb000 rw-p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f4735efb000-7f4735fe4000 r-xp 00000000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f4735fe4000-7f47361e3000 ---p 000e9000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f47361e3000-7f47361eb000 r--p 000e8000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f47361eb000-7f47361ed000 rw-p 000f0000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f47361ed000-7f4736202000 rw-p 00000000 00:00 0 7f4736202000-7f4736204000 r-xp 00000000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f4736204000-7f4736404000 ---p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f4736404000-7f4736405000 r--p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f4736405000-7f4736406000 rw-p 00003000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f4736406000-7f473663c000 r-xp 00000000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f473663c000-7f473683c000 ---p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f473683c000-7f4736858000 r--p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f4736858000-7f4736865000 rw-p 00252000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f4736865000-7f4736869000 rw-p 00000000 00:00 0 7f4736869000-7f47368d0000 r-xp 00000000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f47368d0000-7f4736ad0000 ---p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f4736ad0000-7f4736ad4000 r--p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f4736ad4000-7f4736adb000 rw-p 0006b000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f4736adb000-7f4736b0a000 r-xp 00000000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f4736b0a000-7f4736d0a000 ---p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f4736d0a000-7f4736d0b000 r--p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f4736d0b000-7f4736d0c000 rw-p 00030000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f4736d0c000-7f4736d14000 r-xp 00000000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f4736d14000-7f4736f13000 ---p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f4736f13000-7f4736f14000 r--p 00007000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f4736f14000-7f4736f15000 rw-p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f4736f15000-7f4736f43000 rw-p 00000000 00:00 0 7f4736f43000-7f4736f58000 r-xp 00000000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f4736f58000-7f4737157000 ---p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f4737157000-7f4737158000 r--p 00014000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f4737158000-7f4737159000 rw-p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f4737159000-7f473715a000 r-xp 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f473715a000-7f4737359000 ---p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f4737359000-7f473735a000 r--p 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f473735a000-7f473735b000 rw-p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f473735b000-7f4737380000 r-xp 00000000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f4737380000-7f473757f000 ---p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f473757f000-7f4737580000 r--p 00024000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f4737580000-7f4737581000 rw-p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f4737581000-7f4737598000 r-xp 00000000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f4737598000-7f4737797000 ---p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f4737797000-7f4737798000 r--p 00016000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f4737798000-7f4737799000 rw-p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f4737799000-7f473779d000 rw-p 00000000 00:00 0 7f473779d000-7f47377bf000 r-xp 00000000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f47377c3000-7f47377c4000 rw-p 00000000 00:00 0 7f47377c4000-7f47377c5000 ---p 00000000 00:00 0 7f47377c5000-7f473780f000 rw-p 00000000 00:00 0 7f473780f000-7f4737812000 rw-s 00000000 00:0b 50473809 /[aio] (deleted) 7f4737812000-7f4737815000 rw-s 00000000 00:0b 50473808 /[aio] (deleted) 7f4737815000-7f4737818000 rw-s 00000000 00:0b 50473807 /[aio] (deleted) 7f4737818000-7f473781b000 rw-s 00000000 00:0b 50473806 /[aio] (deleted) 7f473781b000-7f473781e000 rw-s 00000000 00:0b 50473805 /[aio] (deleted) 7f473781e000-7f4737821000 rw-s 00000000 00:0b 50473804 /[aio] (deleted) 7f4737821000-7f4737824000 rw-s 00000000 00:0b 50473803 /[aio] (deleted) 7f4737824000-7f4737827000 rw-s 00000000 00:0b 50473802 /[aio] (deleted) 7f4737827000-7f47379b5000 rw-p 00000000 00:00 0 7f47379b5000-7f47379b6000 rw-s 00000000 00:0b 50473810 /[aio] (deleted) 7f47379b6000-7f47379b9000 rw-s 00000000 00:0b 50473801 /[aio] (deleted) 7f47379b9000-7f47379bc000 rw-s 00000000 00:0b 50473800 /[aio] (deleted) 7f47379bc000-7f47379bd000 rw-s 00000000 00:0b 50473799 /[aio] (deleted) 7f47379bd000-7f47379be000 rw-p 00000000 00:00 0 7f47379be000-7f47379bf000 r--p 00021000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f47379bf000-7f47379c0000 rw-p 00022000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f47379c0000-7f47379c1000 rw-p 00000000 00:00 0 7ffc31909000-7ffc3192a000 rw-p 00000000 00:00 0 [stack] 7ffc3196e000-7ffc31970000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Fatal signal 6 while backtracing 2020-01-31 21:02:05 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 21:02:05 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 21:02:05 0 [Note] InnoDB: Uses event mutexes 2020-01-31 21:02:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 21:02:05 0 [Note] InnoDB: Number of pools: 1 2020-01-31 21:02:05 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 21:02:05 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 21:02:05 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 21:02:05 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 21:02:05 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136174659 2020-01-31 21:02:05 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000143', position 192863 2020-01-31 21:02:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 21:02:05 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 21:02:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 21:02:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 21:02:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 21:02:05 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 21:02:05 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136174668; transaction id 5900650 2020-01-31 21:02:05 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 21:02:05 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 21:02:05 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 21:02:05 0 [Note] Starting crash recovery... 2020-01-31 21:02:05 0 [Note] Crash recovery finished. 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/servers' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/servers' 2020-01-31 21:02:05 0 [Note] Server socket created on IP: '::'. 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/db' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/db' 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/proxies_priv' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/proxies_priv' 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/roles_mapping' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/roles_mapping' 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/global_priv' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/global_priv' 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/time_zone_leap_second' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/time_zone_leap_second' 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/time_zone_name' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/time_zone_name' 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/time_zone' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/time_zone' 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/time_zone_transition_type' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/time_zone_transition_type' 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/time_zone_transition' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/time_zone_transition' 2020-01-31 21:02:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 21:02:05 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/tables_priv' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/tables_priv' 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/columns_priv' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/columns_priv' 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/procs_priv' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/procs_priv' 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/func' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/func' 2020-01-31 21:02:05 0 [ERROR] mysqld: Table './mysql/event' is marked as crashed and should be repaired 2020-01-31 21:02:05 0 [Warning] Recovering table: './mysql/event' 2020-01-31 21:02:05 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 21:02:05 0 [Note] Added new Master_info '' to hash table 2020-01-31 21:02:05 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000064' at position 700496 2020-01-31 21:02:05 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 21:02:05 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 21:02:05 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 21:02:05 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 21:02:05 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 21:02:05 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 21:02:05 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 21:02:05 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 21:02:05 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 21:02:05 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 21:02:05 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 21:02:05 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000064' at position 700496 2020-01-31 21:02:31 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 21:02:31 18 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:02:31 18 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 21:02:32 19 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 21:02:32 19 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 21:02:36 23 [ERROR] mysqld: Table './pizzahut/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 21:02:36 23 [Warning] Checking table: './pizzahut/wp_usermeta' 2020-01-31 21:02:39 24 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 21:02:39 24 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 21:02:59 36 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000143, 528113) 2020-01-31 21:03:09 48 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 21:03:09 48 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 21:03:10 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f28e0006c48 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f293c58bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a308f9af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a308a313af] sigaction.c:0(__restore_rt)[0x7f2957733630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a30895f151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a308a9d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a308853ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a308849d4b] /usr/sbin/mysqld(+0x6ca792)[0x55a308854792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a30887c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a30887e65d] /usr/sbin/mysqld(+0x65e3f3)[0x55a3087e83f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a3087e7cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a30887bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a30887e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a30887e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a30887f377] /usr/sbin/mysqld(+0x58fd84)[0x55a308719d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a308826b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a30883eb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a30883ecd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a30883f5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a30883f677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a30882e56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a30883007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a30890bafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a30890bbdd] pthread_create.c:0(start_thread)[0x7f295772bea5] /lib64/libc.so.6(clone+0x6d)[0x7f2955acc8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f28e0010ef0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 49 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 21:03:16 0 [Note] mysqld: Aria engine: starting recovery recovered pages: 0% 12% 26% 41% 56% 71% 85% 97% 100% (0.0 seconds); tables to flush: 11 10 9 8 7 6 5 4 3 2 1 0 (0.5 seconds); 2020-01-31 21:03:17 0 [Note] mysqld: Aria engine: recovery done 2020-01-31 21:03:17 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 21:03:17 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 21:03:17 0 [Note] InnoDB: Uses event mutexes 2020-01-31 21:03:17 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 21:03:17 0 [Note] InnoDB: Number of pools: 1 2020-01-31 21:03:17 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 21:03:17 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 21:03:17 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 21:03:17 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 21:03:17 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136177478 2020-01-31 21:03:17 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000143', position 192863 2020-01-31 21:03:17 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 21:03:17 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 21:03:17 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 21:03:17 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 21:03:17 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 21:03:17 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 21:03:17 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136177487; transaction id 5900661 2020-01-31 21:03:17 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 21:03:17 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 21:03:17 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 21:03:17 0 [Note] Starting crash recovery... 2020-01-31 21:03:17 0 [Note] Crash recovery finished. 2020-01-31 21:03:17 0 [Note] Server socket created on IP: '::'. 2020-01-31 21:03:17 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 21:03:17 0 [Note] Added new Master_info '' to hash table 2020-01-31 21:03:17 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000064' at position 854845 2020-01-31 21:03:17 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 21:03:17 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 21:03:17 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 21:03:17 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000064' at position 854845 2020-01-31 21:03:17 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 21:03:17 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 21:03:17 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 21:03:17 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 21:03:17 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 21:03:17 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 21:03:17 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 21:03:17 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 21:03:17 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 21:03:17 2020-01-31 21:03:20 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 21:03:20 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000064' at position 855997 2020-01-31 21:03:20 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 21:03:26 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000144, 69320) 2020-01-31 21:03:41 13 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:03:41 14 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:03:41 15 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:03:46 16 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 21:03:46 17 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:03:46 17 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 21:03:49 19 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 21:03:49 19 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 21:03:49 22 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 21:03:49 22 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 21:03:56 26 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:03:56 26 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 21:03:57 31 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 21:03:57 31 [Warning] Checking table: './wpms/wp_options' 2020-01-31 21:03:59 34 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 21:03:59 34 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 21:03:59 35 [ERROR] mysqld: Table './pizzahut/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 21:03:59 35 [Warning] Checking table: './pizzahut/wp_usermeta' 2020-01-31 21:04:15 50 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:04:15 50 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 21:04:19 54 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 21:04:19 54 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-01-31 21:04:20 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000064' at position 855997 2020-01-31 21:04:32 70 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:04:32 71 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:04:32 72 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:04:42 77 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:04:42 78 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:04:42 79 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:04:43 80 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:05:22 101 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 200131 21:06:58 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=8 max_threads=502 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0afc000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0b699efcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a18d37af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a18ce113af] sigaction.c:0(__restore_rt)[0x7f0b84bd1630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a18cd3f151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a18ce7d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a18cc33ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a18cc29d4b] /usr/sbin/mysqld(+0x6ca792)[0x55a18cc34792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a18cc5c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a18cc5e65d] /usr/sbin/mysqld(+0x65e3f3)[0x55a18cbc83f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a18cbc7cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a18cc5bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a18cc5e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a18cc5e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a18cc5f377] /usr/sbin/mysqld(+0x58fd84)[0x55a18caf9d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a18cc06b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a18cc1eb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a18cc1ecd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a18cc1f5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a18cc1f677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a18cc0e56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a18cc1007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a18ccebafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a18ccebbdd] pthread_create.c:0(start_thread)[0x7f0b84bc9ea5] /lib64/libc.so.6(clone+0x6d)[0x7f0b82f6a8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f0afc00fe80): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 160 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 21:07:04 0 [Note] mysqld: Aria engine: starting recovery recovered pages: 0% 10% 21% 32% 44% 56% 68% 82% 97% 100% (0.0 seconds); tables to flush: 10 9 8 7 6 5 4 3 2 1 0 (0.0 seconds); 2020-01-31 21:07:04 0 [Note] mysqld: Aria engine: recovery done 2020-01-31 21:07:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 21:07:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 21:07:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 21:07:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 21:07:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 21:07:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 21:07:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 21:07:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 21:07:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 21:07:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136181634 2020-01-31 21:07:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000145', position 431341 2020-01-31 21:07:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 21:07:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 21:07:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 21:07:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 21:07:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 21:07:04 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 21:07:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136181643; transaction id 5900698 2020-01-31 21:07:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 21:07:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/plugin' 2020-01-31 21:07:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 21:07:04 0 [Note] Starting crash recovery... 2020-01-31 21:07:04 0 [Note] Crash recovery finished. 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/servers' 2020-01-31 21:07:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/db' 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/proxies_priv' 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/roles_mapping' 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/global_priv' 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/time_zone_leap_second' 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/time_zone_name' 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/time_zone' 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/time_zone_transition_type' 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/time_zone_transition' 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/tables_priv' 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/columns_priv' 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/procs_priv' 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/func' 2020-01-31 21:07:04 0 [Note] Zerofilling moved table: './mysql/event' 2020-01-31 21:07:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 21:07:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 21:07:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000065' at position 346 2020-01-31 21:07:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 21:07:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 21:07:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000065' at position 346 2020-01-31 21:07:04 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 21:07:04 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 21:07:04 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 21:07:04 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 21:07:04 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 21:07:04 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 21:07:04 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 21:07:04 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 21:07:04 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 21:07:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 21:07:05 2020-01-31 21:07:31 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 21:07:58 31 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000145, 1028120) 2020-01-31 21:09:05 63 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 200131 21:09:15 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fb46c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fb4c8e7dcf0 thread_stack 0x49000 *** buffer overflow detected ***: /usr/sbin/mysqld terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7fb4e243e567] /lib64/libc.so.6(+0x1166e2)[0x7fb4e243c6e2] /lib64/libc.so.6(+0x1184c7)[0x7fb4e243e4c7] /usr/sbin/mysqld(my_addr_resolve+0xda)[0x55dac571b94a] /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x55dac57040c2] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55dac519a3af] /lib64/libpthread.so.0(+0xf630)[0x7fb4e408b630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55dac50c8151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55dac5206545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55dac4fbcccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55dac4fb2d4b] /usr/sbin/mysqld(+0x6ca792)[0x55dac4fbd792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55dac4fe56da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55dac4fe765d] /usr/sbin/mysqld(+0x65e3f3)[0x55dac4f513f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55dac4f50cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55dac4fe4b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55dac4fe765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55dac4fe7761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55dac4fe8377] /usr/sbin/mysqld(+0x58fd84)[0x55dac4e82d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55dac4f8fb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55dac4fa7b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55dac4fa7cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55dac4fa85e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55dac4fa8677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55dac4f9756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55dac4f9907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55dac5074afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55dac5074bdd] /lib64/libpthread.so.0(+0x7ea5)[0x7fb4e4083ea5] /lib64/libc.so.6(clone+0x6d)[0x7fb4e24248cd] ======= Memory map: ======== 55dac48f3000-55dac5d41000 r-xp 00000000 fd:00 2101454 /usr/sbin/mysqld 55dac5f40000-55dac606c000 r--p 0144d000 fd:00 2101454 /usr/sbin/mysqld 55dac606c000-55dac6126000 rw-p 01579000 fd:00 2101454 /usr/sbin/mysqld 55dac6126000-55dac69d5000 rw-p 00000000 00:00 0 55dac6b58000-55dacb0fc000 rw-p 00000000 00:00 0 [heap] 7fb43b7ff000-7fb43b800000 ---p 00000000 00:00 0 7fb43b800000-7fb43c000000 rw-p 00000000 00:00 0 7fb43c000000-7fb43c030000 rw-p 00000000 00:00 0 7fb43c030000-7fb440000000 ---p 00000000 00:00 0 7fb442ffe000-7fb442fff000 ---p 00000000 00:00 0 7fb442fff000-7fb4437ff000 rw-p 00000000 00:00 0 7fb4437ff000-7fb443800000 ---p 00000000 00:00 0 7fb443800000-7fb444000000 rw-p 00000000 00:00 0 7fb444000000-7fb444021000 rw-p 00000000 00:00 0 7fb444021000-7fb448000000 ---p 00000000 00:00 0 7fb448000000-7fb448027000 rw-p 00000000 00:00 0 7fb448027000-7fb44c000000 ---p 00000000 00:00 0 7fb44c000000-7fb44c021000 rw-p 00000000 00:00 0 7fb44c021000-7fb450000000 ---p 00000000 00:00 0 7fb450000000-7fb450027000 rw-p 00000000 00:00 0 7fb450027000-7fb454000000 ---p 00000000 00:00 0 7fb454000000-7fb454021000 rw-p 00000000 00:00 0 7fb454021000-7fb458000000 ---p 00000000 00:00 0 7fb458000000-7fb458123000 rw-p 00000000 00:00 0 7fb458123000-7fb45c000000 ---p 00000000 00:00 0 7fb45c000000-7fb45c529000 rw-p 00000000 00:00 0 7fb45c529000-7fb460000000 ---p 00000000 00:00 0 7fb460000000-7fb462816000 rw-p 00000000 00:00 0 7fb462816000-7fb464000000 ---p 00000000 00:00 0 7fb464000000-7fb46406b000 rw-p 00000000 00:00 0 7fb46406b000-7fb468000000 ---p 00000000 00:00 0 7fb468000000-7fb468045000 rw-p 00000000 00:00 0 7fb468045000-7fb46c000000 ---p 00000000 00:00 0 7fb46c000000-7fb46c08b000 rw-p 00000000 00:00 0 7fb46c08b000-7fb470000000 ---p 00000000 00:00 0 7fb47070d000-7fb47070e000 ---p 00000000 00:00 0 7fb47070e000-7fb470f0e000 rw-p 00000000 00:00 0 7fb470f0e000-7fb470f0f000 ---p 00000000 00:00 0 7fb470f0f000-7fb47170f000 rw-p 00000000 00:00 0 7fb47170f000-7fb471710000 ---p 00000000 00:00 0 7fb471710000-7fb471f10000 rw-p 00000000 00:00 0 7fb471f10000-7fb471f11000 ---p 00000000 00:00 0 7fb471f11000-7fb472711000 rw-p 00000000 00:00 0 7fb472711000-7fb472712000 ---p 00000000 00:00 0 7fb472712000-7fb472f12000 rw-p 00000000 00:00 0 7fb472ffe000-7fb472fff000 ---p 00000000 00:00 0 7fb472fff000-7fb4737ff000 rw-p 00000000 00:00 0 7fb4737ff000-7fb473800000 ---p 00000000 00:00 0 7fb473800000-7fb474000000 rw-p 00000000 00:00 0 7fb474000000-7fb47405b000 rw-p 00000000 00:00 0 7fb47405b000-7fb478000000 ---p 00000000 00:00 0 7fb4784f8000-7fb47850a000 r-xp 00000000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7fb47850a000-7fb478709000 ---p 00012000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7fb478709000-7fb47870c000 r--p 00011000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7fb47870c000-7fb47870d000 rw-p 00014000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7fb47870d000-7fb47870e000 ---p 00000000 00:00 0 7fb47870e000-7fb478f0e000 rw-p 00000000 00:00 0 7fb478f0e000-7fb478f0f000 ---p 00000000 00:00 0 7fb478f0f000-7fb47970f000 rw-p 00000000 00:00 0 7fb47970f000-7fb479710000 ---p 00000000 00:00 0 7fb479710000-7fb479f10000 rw-p 00000000 00:00 0 7fb479f10000-7fb479f11000 ---p 00000000 00:00 0 7fb479f11000-7fb47a711000 rw-p 00000000 00:00 0 7fb47a711000-7fb47a712000 ---p 00000000 00:00 0 7fb47a712000-7fb47af12000 rw-p 00000000 00:00 0 7fb47af12000-7fb47af13000 ---p 00000000 00:00 0 7fb47af13000-7fb47b713000 rw-p 00000000 00:00 0 7fb47b713000-7fb47b714000 ---p 00000000 00:00 0 7fb47b714000-7fb47bf14000 rw-p 00000000 00:00 0 7fb47bf14000-7fb47bf15000 ---p 00000000 00:00 0 7fb47bf15000-7fb47c715000 rw-p 00000000 00:00 0 7fb47c715000-7fb47c716000 ---p 00000000 00:00 0 7fb47c716000-7fb47cf16000 rw-p 00000000 00:00 0 7fb47cf16000-7fb47cf17000 ---p 00000000 00:00 0 7fb47cf17000-7fb47d717000 rw-p 00000000 00:00 0 7fb47d717000-7fb47d718000 ---p 00000000 00:00 0 7fb47d718000-7fb47df18000 rw-p 00000000 00:00 0 7fb47df18000-7fb47df19000 ---p 00000000 00:00 0 7fb47df19000-7fb47e719000 rw-p 00000000 00:00 0 7fb47e719000-7fb47e71a000 ---p 00000000 00:00 0 7fb47e71a000-7fb47ef1a000 rw-p 00000000 00:00 0 7fb47ef1a000-7fb47ef1b000 ---p 00000000 00:00 0 7fb47ef1b000-7fb47f71b000 rw-p 00000000 00:00 0 7fb47f71b000-7fb47f71c000 ---p 00000000 00:00 0 7fb47f71c000-7fb47ff1c000 rw-p 00000000 00:00 0 7fb47ff1c000-7fb47ff1d000 ---p 00000000 00:00 0 7fb47ff1d000-7fb48071d000 rw-p 00000000 00:00 0 7fb48071d000-7fb48071e000 ---p 00000000 00:00 0 7fb48071e000-7fb480f1e000 rw-p 00000000 00:00 0 7fb480ffe000-7fb480fff000 ---p 00000000 00:00 0 7fb480fff000-7fb4817ff000 rw-p 00000000 00:00 0 7fb4817ff000-7fb481800000 ---p 00000000 00:00 0 7fb481800000-7fb482000000 rw-p 00000000 00:00 0 7fb482000000-7fb4c4000000 rw-p 00000000 00:00 0 7fb4c4000000-7fb4c4021000 rw-p 00000000 00:00 0 7fb4c4021000-7fb4c8000000 ---p 00000000 00:00 0 7fb4c803c000-7fb4c8042000 r-xp 00000000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7fb4c8042000-7fb4c8241000 ---p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7fb4c8241000-7fb4c8242000 r--p 00005000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7fb4c8242000-7fb4c8243000 rw-p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7fb4c8243000-7fb4c824f000 r-xp 00000000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7fb4c824f000-7fb4c844e000 ---p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7fb4c844e000-7fb4c844f000 r--p 0000b000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7fb4c844f000-7fb4c8450000 rw-p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7fb4c8450000-7fb4c8df6000 rw-p 00000000 00:00 0 7fb4c8e34000-7fb4c8e35000 ---p 00000000 00:00 0 7fb4c8e35000-7fb4c8e7f000 rw-p 00000000 00:00 0 7fb4c8e7f000-7fb4c8e80000 ---p 00000000 00:00 0 7fb4c8e80000-7fb4c8eca000 rw-p 00000000 00:00 0 7fb4c8eca000-7fb4c8ecb000 ---p 00000000 00:00 0 7fb4c8ecb000-7fb4c8f15000 rw-p 00000000 00:00 0 7fb4c8f15000-7fb4c8f16000 ---p 00000000 00:00 0 7fb4c8f16000-7fb4c8f60000 rw-p 00000000 00:00 0 7fb4c8f60000-7fb4c8f61000 ---p 00000000 00:00 0 7fb4c8f61000-7fb4c8fab000 rw-p 00000000 00:00 0 7fb4c8fab000-7fb4c8fac000 ---p 00000000 00:00 0 7fb4c8fac000-7fb4d8000000 rw-p 00000000 00:00 0 7fb4d8000000-7fb4d8021000 rw-p 00000000 00:00 0 7fb4d8021000-7fb4dc000000 ---p 00000000 00:00 0 7fb4dc017000-7fb4dc018000 ---p 00000000 00:00 0 7fb4dc018000-7fb4dc062000 rw-p 00000000 00:00 0 7fb4dc062000-7fb4dc063000 ---p 00000000 00:00 0 7fb4dc063000-7fb4dc0ad000 rw-p 00000000 00:00 0 7fb4dc0ad000-7fb4dc0ae000 ---p 00000000 00:00 0 7fb4dc0ae000-7fb4dc933000 rw-p 00000000 00:00 0 7fb4dc933000-7fb4dc934000 ---p 00000000 00:00 0 7fb4dc934000-7fb4defc3000 rw-p 00000000 00:00 0 7fb4defc3000-7fb4defc4000 ---p 00000000 00:00 0 7fb4defc4000-7fb4df7c4000 rw-p 00000000 00:00 0 7fb4df7c4000-7fb4df7c7000 r-xp 00000000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7fb4df7c7000-7fb4df9c6000 ---p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7fb4df9c6000-7fb4df9c7000 r--p 00002000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7fb4df9c7000-7fb4df9c8000 rw-p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7fb4df9c8000-7fb4df9d6000 r-xp 00000000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7fb4df9d6000-7fb4dfbd6000 ---p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7fb4dfbd6000-7fb4dfbd7000 r--p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7fb4dfbd7000-7fb4dfbd8000 rw-p 0000f000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7fb4dfbd8000-7fb4dfbe7000 r-xp 00000000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7fb4dfbe7000-7fb4dfde6000 ---p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7fb4dfde6000-7fb4dfde7000 r--p 0000e000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7fb4dfde7000-7fb4dfde8000 rw-p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7fb4dfde8000-7fb4dfdff000 r-xp 00000000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7fb4dfdff000-7fb4dfffe000 ---p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7fb4dfffe000-7fb4dffff000 r--p 00016000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7fb4dffff000-7fb4e0000000 rw-p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7fb4e0000000-7fb4e0060000 r-xp 00000000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7fb4e0060000-7fb4e0260000 ---p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7fb4e0260000-7fb4e0261000 r--p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7fb4e0261000-7fb4e0262000 rw-p 00061000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7fb4e0262000-7fb4e0266000 r-xp 00000000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7fb4e0266000-7fb4e0465000 ---p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7fb4e0465000-7fb4e0466000 r--p 00003000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7fb4e0466000-7fb4e0467000 rw-p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7fb4e0467000-7fb4e0498000 r-xp 00000000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7fb4e0498000-7fb4e0697000 ---p 00031000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7fb4e0697000-7fb4e0699000 r--p 00030000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7fb4e0699000-7fb4e069a000 rw-p 00032000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7fb4e069a000-7fb4e069d000 r-xp 00000000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7fb4e069d000-7fb4e089c000 ---p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7fb4e089c000-7fb4e089d000 r--p 00002000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7fb4e089d000-7fb4e089e000 rw-p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7fb4e089e000-7fb4e0977000 r-xp 00000000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7fb4e0977000-7fb4e0b76000 ---p 000d9000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7fb4e0b76000-7fb4e0b84000 r--p 000d8000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7fb4e0b84000-7fb4e0b87000 rw-p 000e6000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7fb4e0b87000-7fb4e0bd1000 r-xp 00000000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7fb4e0bd1000-7fb4e0dd1000 ---p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7fb4e0dd1000-7fb4e0dd2000 r--p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7fb4e0dd2000-7fb4e0dd4000 rw-p 0004b000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7fb4e0dd4000-7fb4e0de9000 r-xp 00000000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fb4e0de9000-7fb4e0fe8000 ---p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fb4e0fe8000-7fb4e0fe9000 r--p 00014000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fb4e0fe9000-7fb4e0fea000 rw-p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fb4e0fea000-7fb4e1038000 r-xp 00000000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7fb4e1038000-7fb4e1238000 ---p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7fb4e1238000-7fb4e123a000 r--p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7fb4e123a000-7fb4e123b000 rw-p 00050000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7fb4e123b000-7fb4e1251000 r-xp 00000000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7fb4e1251000-7fb4e1450000 ---p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7fb4e1450000-7fb4e1451000 r--p 00015000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7fb4e1451000-7fb4e1452000 rw-p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7fb4e1452000-7fb4e1454000 rw-p 00000000 00:00 0 7fb4e1454000-7fb4e1458000 r-xp 00000000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7fb4e1458000-7fb4e1657000 ---p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7fb4e1657000-7fb4e1658000 r--p 00003000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7fb4e1658000-7fb4e1659000 rw-p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7fb4e1659000-7fb4e16d6000 r-xp 00000000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7fb4e16d6000-7fb4e18d5000 ---p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7fb4e18d5000-7fb4e18d6000 r--p 0007c000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7fb4e18d6000-7fb4e18d9000 rw-p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7fb4e18d9000-7fb4e18da000 rw-p 00000000 00:00 0 7fb4e18da000-7fb4e18ee000 r-xp 00000000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7fb4e18ee000-7fb4e1aed000 ---p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7fb4e1aed000-7fb4e1aee000 r--p 00013000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7fb4e1aee000-7fb4e1aef000 rw-p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7fb4e1aef000-7fb4e1b13000 r-xp 00000000 fd:00 2101234 /usr/lib64/libselinux.so.1 7fb4e1b13000-7fb4e1d12000 ---p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7fb4e1d12000-7fb4e1d13000 r--p 00023000 fd:00 2101234 /usr/lib64/libselinux.so.1 7fb4e1d13000-7fb4e1d14000 rw-p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7fb4e1d14000-7fb4e1d16000 rw-p 00000000 00:00 0 7fb4e1d16000-7fb4e1d1d000 r-xp 00000000 fd:00 2101451 /usr/lib64/librt-2.17.so 7fb4e1d1d000-7fb4e1f1c000 ---p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7fb4e1f1c000-7fb4e1f1d000 r--p 00006000 fd:00 2101451 /usr/lib64/librt-2.17.so 7fb4e1f1d000-7fb4e1f1e000 rw-p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7fb4e1f1e000-7fb4e1f22000 r-xp 00000000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7fb4e1f22000-7fb4e2121000 ---p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7fb4e2121000-7fb4e2122000 r--p 00003000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7fb4e2122000-7fb4e2123000 rw-p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7fb4e2123000-7fb4e2125000 r-xp 00000000 fd:00 2227367 /usr/lib64/libfreebl3.so 7fb4e2125000-7fb4e2324000 ---p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7fb4e2324000-7fb4e2325000 r--p 00001000 fd:00 2227367 /usr/lib64/libfreebl3.so 7fb4e2325000-7fb4e2326000 rw-p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7fb4e2326000-7fb4e24e9000 r-xp 00000000 fd:00 2097970 /usr/lib64/libc-2.17.so 7fb4e24e9000-7fb4e26e9000 ---p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7fb4e26e9000-7fb4e26ed000 r--p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7fb4e26ed000-7fb4e26ef000 rw-p 001c7000 fd:00 2097970 /usr/lib64/libc-2.17.so 7fb4e26ef000-7fb4e26f4000 rw-p 00000000 00:00 0 7fb4e26f4000-7fb4e27f5000 r-xp 00000000 fd:00 2097978 /usr/lib64/libm-2.17.so 7fb4e27f5000-7fb4e29f4000 ---p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7fb4e29f4000-7fb4e29f5000 r--p 00100000 fd:00 2097978 /usr/lib64/libm-2.17.so 7fb4e29f5000-7fb4e29f6000 rw-p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7fb4e29f6000-7fb4e2adf000 r-xp 00000000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7fb4e2adf000-7fb4e2cde000 ---p 000e9000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7fb4e2cde000-7fb4e2ce6000 r--p 000e8000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7fb4e2ce6000-7fb4e2ce8000 rw-p 000f0000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7fb4e2ce8000-7fb4e2cfd000 rw-p 00000000 00:00 0 7fb4e2cfd000-7fb4e2cff000 r-xp 00000000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7fb4e2cff000-7fb4e2eff000 ---p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7fb4e2eff000-7fb4e2f00000 r--p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7fb4e2f00000-7fb4e2f01000 rw-p 00003000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7fb4e2f01000-7fb4e3137000 r-xp 00000000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7fb4e3137000-7fb4e3337000 ---p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7fb4e3337000-7fb4e3353000 r--p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7fb4e3353000-7fb4e3360000 rw-p 00252000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7fb4e3360000-7fb4e3364000 rw-p 00000000 00:00 0 7fb4e3364000-7fb4e33cb000 r-xp 00000000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7fb4e33cb000-7fb4e35cb000 ---p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7fb4e35cb000-7fb4e35cf000 r--p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7fb4e35cf000-7fb4e35d6000 rw-p 0006b000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7fb4e35d6000-7fb4e3605000 r-xp 00000000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7fb4e3605000-7fb4e3805000 ---p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7fb4e3805000-7fb4e3806000 r--p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7fb4e3806000-7fb4e3807000 rw-p 00030000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7fb4e3807000-7fb4e380f000 r-xp 00000000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7fb4e380f000-7fb4e3a0e000 ---p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7fb4e3a0e000-7fb4e3a0f000 r--p 00007000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7fb4e3a0f000-7fb4e3a10000 rw-p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7fb4e3a10000-7fb4e3a3e000 rw-p 00000000 00:00 0 7fb4e3a3e000-7fb4e3a53000 r-xp 00000000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7fb4e3a53000-7fb4e3c52000 ---p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7fb4e3c52000-7fb4e3c53000 r--p 00014000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7fb4e3c53000-7fb4e3c54000 rw-p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7fb4e3c54000-7fb4e3c55000 r-xp 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7fb4e3c55000-7fb4e3e54000 ---p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7fb4e3e54000-7fb4e3e55000 r--p 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7fb4e3e55000-7fb4e3e56000 rw-p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7fb4e3e56000-7fb4e3e7b000 r-xp 00000000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7fb4e3e7b000-7fb4e407a000 ---p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7fb4e407a000-7fb4e407b000 r--p 00024000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7fb4e407b000-7fb4e407c000 rw-p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7fb4e407c000-7fb4e4093000 r-xp 00000000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7fb4e4093000-7fb4e4292000 ---p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7fb4e4292000-7fb4e4293000 r--p 00016000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7fb4e4293000-7fb4e4294000 rw-p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7fb4e4294000-7fb4e4298000 rw-p 00000000 00:00 0 7fb4e4298000-7fb4e42ba000 r-xp 00000000 fd:00 2227369 /usr/lib64/ld-2.17.so 7fb4e42be000-7fb4e42bf000 rw-p 00000000 00:00 0 7fb4e42bf000-7fb4e42c0000 ---p 00000000 00:00 0 7fb4e42c0000-7fb4e430a000 rw-p 00000000 00:00 0 7fb4e430a000-7fb4e430d000 rw-s 00000000 00:0b 50484921 /[aio] (deleted) 7fb4e430d000-7fb4e4310000 rw-s 00000000 00:0b 50484920 /[aio] (deleted) 7fb4e4310000-7fb4e4313000 rw-s 00000000 00:0b 50484919 /[aio] (deleted) 7fb4e4313000-7fb4e4316000 rw-s 00000000 00:0b 50484918 /[aio] (deleted) 7fb4e4316000-7fb4e4319000 rw-s 00000000 00:0b 50484917 /[aio] (deleted) 7fb4e4319000-7fb4e431c000 rw-s 00000000 00:0b 50484916 /[aio] (deleted) 7fb4e431c000-7fb4e431f000 rw-s 00000000 00:0b 50484915 /[aio] (deleted) 7fb4e431f000-7fb4e4322000 rw-s 00000000 00:0b 50484914 /[aio] (deleted) 7fb4e4322000-7fb4e44b0000 rw-p 00000000 00:00 0 7fb4e44b0000-7fb4e44b1000 rw-s 00000000 00:0b 50484922 /[aio] (deleted) 7fb4e44b1000-7fb4e44b4000 rw-s 00000000 00:0b 50484913 /[aio] (deleted) 7fb4e44b4000-7fb4e44b7000 rw-s 00000000 00:0b 50484912 /[aio] (deleted) 7fb4e44b7000-7fb4e44b8000 rw-s 00000000 00:0b 50484911 /[aio] (deleted) 7fb4e44b8000-7fb4e44b9000 rw-p 00000000 00:00 0 7fb4e44b9000-7fb4e44ba000 r--p 00021000 fd:00 2227369 /usr/lib64/ld-2.17.so 7fb4e44ba000-7fb4e44bb000 rw-p 00022000 fd:00 2227369 /usr/lib64/ld-2.17.so 7fb4e44bb000-7fb4e44bc000 rw-p 00000000 00:00 0 7ffc79e63000-7ffc79e8e000 rw-p 00000000 00:00 0 [stack] 7ffc79fb6000-7ffc79fb8000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Fatal signal 6 while backtracing 2020-01-31 21:09:22 0 [Note] mysqld: Aria engine: starting recovery recovered pages: 0% 10% 21% 32% 44% 56% 68% 82% 97% 100% (0.0 seconds); tables to flush: 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 (0.0 seconds); 2020-01-31 21:09:22 0 [Note] mysqld: Aria engine: recovery done 2020-01-31 21:09:22 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 21:09:22 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 21:09:22 0 [Note] InnoDB: Uses event mutexes 2020-01-31 21:09:22 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 21:09:22 0 [Note] InnoDB: Number of pools: 1 2020-01-31 21:09:22 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 21:09:22 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 21:09:22 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 21:09:22 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 21:09:22 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136184781 2020-01-31 21:09:22 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000145', position 431341 2020-01-31 21:09:23 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 21:09:23 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 21:09:23 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 21:09:23 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 21:09:23 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 21:09:23 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 21:09:23 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136184790; transaction id 5900709 2020-01-31 21:09:23 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 21:09:23 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/plugin' 2020-01-31 21:09:23 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 21:09:23 0 [Note] Starting crash recovery... 2020-01-31 21:09:23 0 [Note] Crash recovery finished. 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/servers' 2020-01-31 21:09:23 0 [Note] Server socket created on IP: '::'. 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/db' 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/proxies_priv' 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/roles_mapping' 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/global_priv' 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/time_zone_leap_second' 2020-01-31 21:09:23 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 21:09:23 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/time_zone_name' 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/time_zone' 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/time_zone_transition_type' 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/time_zone_transition' 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/tables_priv' 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/columns_priv' 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/procs_priv' 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/func' 2020-01-31 21:09:23 0 [Note] Zerofilling moved table: './mysql/event' 2020-01-31 21:09:23 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 21:09:23 0 [Note] Added new Master_info '' to hash table 2020-01-31 21:09:23 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000065' at position 22133 2020-01-31 21:09:23 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 21:09:23 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 21:09:23 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000065' at position 22133 2020-01-31 21:09:23 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 21:09:23 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 21:09:23 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 21:09:23 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 21:09:23 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 21:09:23 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 21:09:23 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 21:09:23 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 21:09:23 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 21:09:52 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 21:10:05 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 21:10:05 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000065', position 149153 2020-01-31 21:11:19 55 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 21:11:19 55 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 21:11:19 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f68ec0009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f693cec8cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a1458d5f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a14536c3af] sigaction.c:0(__restore_rt)[0x7f6967faf630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a14529a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a1453d8545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a14518eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a145184d4b] /usr/sbin/mysqld(+0x6ca792)[0x55a14518f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a1451b76da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a1451b965d] /usr/sbin/mysqld(+0x65e3f3)[0x55a1451233f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a145122cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a1451b6b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a1451b965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a1451b9761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a1451ba377] /usr/sbin/mysqld(+0x58fd84)[0x55a145054d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a145161b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a145179b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a145179cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a14517a5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a14517a677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a14516956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a14516b07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a145246afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a145246bdd] pthread_create.c:0(start_thread)[0x7f6967fa7ea5] /lib64/libc.so.6(clone+0x6d)[0x7f69663488cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f68ec0109f0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 56 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 21:11:25 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 21:11:25 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 21:11:25 0 [Note] InnoDB: Uses event mutexes 2020-01-31 21:11:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 21:11:25 0 [Note] InnoDB: Number of pools: 1 2020-01-31 21:11:25 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 21:11:25 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 21:11:25 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 21:11:25 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 21:11:25 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136187656 2020-01-31 21:11:25 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000145', position 431341 2020-01-31 21:11:25 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 21:11:25 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 21:11:25 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 21:11:25 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 21:11:25 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 21:11:25 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 21:11:25 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136187665; transaction id 5900720 2020-01-31 21:11:25 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 21:11:26 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 21:11:26 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 21:11:26 0 [Note] Starting crash recovery... 2020-01-31 21:11:26 0 [Note] Crash recovery finished. 2020-01-31 21:11:26 0 [Note] Server socket created on IP: '::'. 2020-01-31 21:11:26 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 21:11:26 0 [Note] Added new Master_info '' to hash table 2020-01-31 21:11:26 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000065' at position 149153 2020-01-31 21:11:26 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 21:11:26 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 21:11:26 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000065' at position 149153 2020-01-31 21:11:26 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 21:11:26 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 21:11:26 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 21:11:26 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 21:11:26 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 21:11:26 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 21:11:26 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 21:11:26 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 21:11:26 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 21:11:27 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 21:11:27 2020-01-31 21:11:52 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 21:11:57 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 21:11:57 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 21:11:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f06a4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f06befb1cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5589d7835f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5589d72cc3af] sigaction.c:0(__restore_rt)[0x7f072130f630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5589d71fa151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5589d7338545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5589d70eeccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5589d70e4d4b] /usr/sbin/mysqld(+0x6ca792)[0x5589d70ef792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5589d71176da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5589d711965d] /usr/sbin/mysqld(+0x65e3f3)[0x5589d70833f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5589d7082cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5589d7116b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5589d711965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5589d7119761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5589d711a377] /usr/sbin/mysqld(+0x58fd84)[0x5589d6fb4d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5589d70c1b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5589d70d9b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5589d70d9cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5589d70da5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5589d70da677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5589d70c956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5589d70cb07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5589d71a6afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5589d71a6bdd] pthread_create.c:0(start_thread)[0x7f0721307ea5] /lib64/libc.so.6(clone+0x6d)[0x7f071f6a88cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f06a400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 21:12:04 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 21:12:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 21:12:04 0 [Note] InnoDB: Uses event mutexes 2020-01-31 21:12:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 21:12:04 0 [Note] InnoDB: Number of pools: 1 2020-01-31 21:12:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 21:12:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 21:12:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 21:12:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 21:12:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136190545 2020-01-31 21:12:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000145', position 431341 2020-01-31 21:12:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 21:12:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 21:12:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 21:12:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 21:12:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 21:12:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136190554; transaction id 5900732 2020-01-31 21:12:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 21:12:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 21:12:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 21:12:04 0 [Note] Starting crash recovery... 2020-01-31 21:12:04 0 [Note] Crash recovery finished. 2020-01-31 21:12:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 21:12:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 21:12:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 21:12:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000065' at position 157596 2020-01-31 21:12:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 21:12:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 21:12:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000065' at position 157596 2020-01-31 21:12:04 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 21:12:04 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 21:12:04 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 21:12:04 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 21:12:04 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 21:12:04 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 21:12:04 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 21:12:04 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 21:12:04 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 21:12:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 21:12:05 2020-01-31 21:12:32 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 21:13:05 22 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:13:19 27 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:13:24 30 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:13:26 32 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:13:28 34 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 21:13:48 41 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:13:56 44 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:14:26 51 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:14:32 54 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:14:44 60 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:15:30 71 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:15:34 74 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:15:42 76 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:16:03 20 [Warning] Aborted connection 20 to db: 'itstraining' user: 'itstraining' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:16:05 86 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:16:09 88 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:16:20 91 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:16:34 98 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:16:52 103 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:16:59 107 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:17:03 108 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:17:05 110 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:17:22 120 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:17:30 125 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:17:33 126 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:17:52 133 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:17:55 135 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:18:02 137 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:18:02 138 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:02 140 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:02 139 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:02 141 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:02 142 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:02 144 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:02 147 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:02 149 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:18:02 148 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:02 150 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:18:02 41 [Warning] Aborted connection 41 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:18:03 22 [Warning] Aborted connection 22 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:18:03 52 [Warning] Aborted connection 52 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:18:03 28 [Warning] Aborted connection 28 to db: 'wpsem' user: 'wpsem' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:18:03 31 [Warning] Aborted connection 31 to db: 'openrecords' user: 'openrecords' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:18:03 60 [Warning] Aborted connection 60 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:18:03 32 [Warning] Aborted connection 32 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:18:03 51 [Warning] Aborted connection 51 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:18:03 54 [Warning] Aborted connection 54 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:18:03 34 [Warning] Aborted connection 34 to db: 'pizzahut' user: 'pizzahut' host: '10.0.59.19' (Table './pizzahut/wp_options' is marked as crashed and should be) 2020-01-31 21:18:03 44 [Warning] Aborted connection 44 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:18:03 30 [Warning] Aborted connection 30 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:18:03 27 [Warning] Aborted connection 27 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:18:09 154 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:17 156 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:21 67 [Warning] Aborted connection 67 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:18:23 160 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:25 161 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:26 38 [Warning] Aborted connection 38 to db: 'itstraining' user: 'root' host: 'localhost' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:18:33 164 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:34 166 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:40 171 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:18:56 177 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:19:13 181 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:19:15 183 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:19:25 187 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:19:33 190 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:19:37 191 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:19:38 192 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:19:39 194 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:00 202 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:01 203 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:03 204 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:03 205 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:04 207 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:20 212 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:20 213 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:20:20 214 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:20:20 216 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:20 218 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:20 221 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:20 225 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:20 226 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:20:20 227 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:20:20 224 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:20 229 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:20 88 [Warning] Aborted connection 88 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:20:20 76 [Warning] Aborted connection 76 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:20:20 105 [Warning] Aborted connection 105 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:20:20 84 [Warning] Aborted connection 84 to db: 'itstraining' user: 'itstraining' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:20:20 98 [Warning] Aborted connection 98 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:20:21 107 [Warning] Aborted connection 107 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:20:21 71 [Warning] Aborted connection 71 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Table './wpstart/wp_options' is marked as crashed and should be ) 2020-01-31 21:20:21 96 [Warning] Aborted connection 96 to db: 'shockeripe' user: 'shockeripe' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:20:21 73 [Warning] Aborted connection 73 to db: 'ulrich' user: 'ulrich' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:20:21 110 [Warning] Aborted connection 110 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:20:21 86 [Warning] Aborted connection 86 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:20:21 108 [Warning] Aborted connection 108 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:20:21 79 [Warning] Aborted connection 79 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:20:21 95 [Warning] Aborted connection 95 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:20:21 100 [Warning] Aborted connection 100 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:20:21 103 [Warning] Aborted connection 103 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:20:21 74 [Warning] Aborted connection 74 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:20:21 167 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:21 91 [Warning] Aborted connection 91 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:20:25 232 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:20:58 244 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:21:10 249 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:21:16 230 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:21:34 256 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:21:45 228 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:21:53 264 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:21:54 266 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:21:57 267 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:15 282 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 288 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 285 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 287 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 299 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 296 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 300 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 301 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 310 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 302 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 297 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 293 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 294 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 307 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 298 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 295 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 309 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 315 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 303 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 306 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 304 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 308 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 316 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 319 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 323 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 321 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 154 [Warning] Aborted connection 154 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:22:20 326 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:20 160 [Warning] Aborted connection 160 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:22:20 133 [Warning] Aborted connection 133 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:22:20 120 [Warning] Aborted connection 120 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:22:20 156 [Warning] Aborted connection 156 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:22:20 164 [Warning] Aborted connection 164 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:22:20 138 [Warning] Aborted connection 138 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:22:20 141 [Warning] Aborted connection 141 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:22:20 152 [Warning] Aborted connection 152 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:22:20 139 [Warning] Aborted connection 139 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:22:20 140 [Warning] Aborted connection 140 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:22:20 143 [Warning] Aborted connection 143 to db: 'wpsem' user: 'wpsem' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:22:20 142 [Warning] Aborted connection 142 to db: 'pizzahut' user: 'pizzahut' host: '10.0.59.19' (Table './pizzahut/wp_options' is marked as crashed and should be) 2020-01-31 21:22:20 144 [Warning] Aborted connection 144 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:22:20 146 [Warning] Aborted connection 146 to db: 'openrecords' user: 'openrecords' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:22:20 145 [Warning] Aborted connection 145 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:22:20 184 [Warning] Aborted connection 184 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:22:20 147 [Warning] Aborted connection 147 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:22:20 170 [Warning] Aborted connection 170 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:22:20 148 [Warning] Aborted connection 148 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:22:20 150 [Warning] Aborted connection 150 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:22:20 149 [Warning] Aborted connection 149 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:22:20 173 [Warning] Aborted connection 173 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:22:21 161 [Warning] Aborted connection 161 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:22:21 177 [Warning] Aborted connection 177 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:22:21 181 [Warning] Aborted connection 181 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:22:21 128 [Warning] Aborted connection 128 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:22:21 171 [Warning] Aborted connection 171 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:22:21 135 [Warning] Aborted connection 135 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:22:21 168 [Warning] Aborted connection 168 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:22:21 166 [Warning] Aborted connection 166 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:22:21 123 [Warning] Aborted connection 123 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:22:21 126 [Warning] Aborted connection 126 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:22:21 122 [Warning] Aborted connection 122 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:22:21 165 [Warning] Aborted connection 165 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:22:21 125 [Warning] Aborted connection 125 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:22:21 119 [Warning] Aborted connection 119 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:22:21 183 [Warning] Aborted connection 183 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:22:21 137 [Warning] Aborted connection 137 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:22:21 158 [Warning] Aborted connection 158 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:22:27 325 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:45 339 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:53 343 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:22:57 313 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:23:12 314 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:23:27 353 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:00 362 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:06 364 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:16 367 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 311 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 371 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 370 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 373 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 375 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 312 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 374 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 376 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 324 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 383 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 381 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 380 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 386 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 379 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 388 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 391 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 384 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 387 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 393 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 390 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 389 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 396 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 392 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 403 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 404 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:24:20 203 [Warning] Aborted connection 203 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Table './wpstart/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:20 191 [Warning] Aborted connection 191 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:20 234 [Warning] Aborted connection 234 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:24:20 194 [Warning] Aborted connection 194 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:20 187 [Warning] Aborted connection 187 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:24:20 207 [Warning] Aborted connection 207 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:20 202 [Warning] Aborted connection 202 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Table './wpstart/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:21 190 [Warning] Aborted connection 190 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:21 244 [Warning] Aborted connection 244 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:24:21 230 [Warning] Aborted connection 230 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:21 210 [Warning] Aborted connection 210 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:24:21 233 [Warning] Aborted connection 233 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:24:21 197 [Warning] Aborted connection 197 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:24:21 204 [Warning] Aborted connection 204 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Table './wpstart/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:21 196 [Warning] Aborted connection 196 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:24:21 243 [Warning] Aborted connection 243 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:24:21 200 [Warning] Aborted connection 200 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:24:21 205 [Warning] Aborted connection 205 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:21 249 [Warning] Aborted connection 249 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:24:21 232 [Warning] Aborted connection 232 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:21 192 [Warning] Aborted connection 192 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Table './wpstart/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:21 214 [Warning] Aborted connection 214 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:24:21 215 [Warning] Aborted connection 215 to db: 'ulrich' user: 'ulrich' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:24:21 213 [Warning] Aborted connection 213 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:24:21 212 [Warning] Aborted connection 212 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Table './wpstart/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:21 217 [Warning] Aborted connection 217 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:24:21 216 [Warning] Aborted connection 216 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:24:21 218 [Warning] Aborted connection 218 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:24:21 220 [Warning] Aborted connection 220 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:24:21 221 [Warning] Aborted connection 221 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:21 219 [Warning] Aborted connection 219 to db: 'itstraining' user: 'itstraining' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:24:21 222 [Warning] Aborted connection 222 to db: 'shockeripe' user: 'shockeripe' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:24:21 223 [Warning] Aborted connection 223 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:24:21 227 [Warning] Aborted connection 227 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:24:21 226 [Warning] Aborted connection 226 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:24:21 242 [Warning] Aborted connection 242 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:24:21 224 [Warning] Aborted connection 224 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:21 225 [Warning] Aborted connection 225 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:21 229 [Warning] Aborted connection 229 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:24:22 410 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:25 412 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:26 413 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:29 416 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 21:24:46 420 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:25:16 427 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:25:25 430 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:25:31 433 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:01 443 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 452 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 460 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 455 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 454 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 463 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 330 [Warning] Aborted connection 330 to db: 'wpsem' user: 'wpsem' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:20 465 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 468 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 464 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 462 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 466 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 478 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 489 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 490 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 480 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 474 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 469 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 467 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 471 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 473 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 479 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 491 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 492 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 477 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 493 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 470 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 475 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 284 [Warning] Aborted connection 284 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:20 494 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 482 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 487 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 481 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 496 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 505 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 504 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 506 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 501 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 502 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 497 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:26:20 278 [Warning] Aborted connection 278 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:20 291 [Warning] Aborted connection 291 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:20 305 [Warning] Aborted connection 305 to db: 'openrecords' user: 'openrecords' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:20 342 [Warning] Aborted connection 342 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:20 332 [Warning] Aborted connection 332 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:20 256 [Warning] Aborted connection 256 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 264 [Warning] Aborted connection 264 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 281 [Warning] Aborted connection 281 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 343 [Warning] Aborted connection 343 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 257 [Warning] Aborted connection 257 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 334 [Warning] Aborted connection 334 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 329 [Warning] Aborted connection 329 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 266 [Warning] Aborted connection 266 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 259 [Warning] Aborted connection 259 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 336 [Warning] Aborted connection 336 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 327 [Warning] Aborted connection 327 to db: 'wpsem' user: 'wpsem' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 277 [Warning] Aborted connection 277 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 267 [Warning] Aborted connection 267 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:26:21 313 [Warning] Aborted connection 313 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:26:21 325 [Warning] Aborted connection 325 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:26:21 263 [Warning] Aborted connection 263 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 268 [Warning] Aborted connection 268 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 339 [Warning] Aborted connection 339 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 282 [Warning] Aborted connection 282 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 228 [Warning] Aborted connection 228 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 286 [Warning] Aborted connection 286 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 289 [Warning] Aborted connection 289 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 287 [Warning] Aborted connection 287 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 288 [Warning] Aborted connection 288 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 290 [Warning] Aborted connection 290 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 285 [Warning] Aborted connection 285 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 292 [Warning] Aborted connection 292 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 300 [Warning] Aborted connection 300 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 260 [Warning] Aborted connection 260 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 299 [Warning] Aborted connection 299 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 296 [Warning] Aborted connection 296 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 301 [Warning] Aborted connection 301 to db: 'pizzahut' user: 'pizzahut' host: '10.0.59.19' (Table './pizzahut/wp_options' is marked as crashed and should be) 2020-01-31 21:26:21 297 [Warning] Aborted connection 297 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:26:21 310 [Warning] Aborted connection 310 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:26:21 309 [Warning] Aborted connection 309 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:26:21 308 [Warning] Aborted connection 308 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 294 [Warning] Aborted connection 294 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 307 [Warning] Aborted connection 307 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 298 [Warning] Aborted connection 298 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:26:21 295 [Warning] Aborted connection 295 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 303 [Warning] Aborted connection 303 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:26:21 293 [Warning] Aborted connection 293 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 302 [Warning] Aborted connection 302 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:26:21 317 [Warning] Aborted connection 317 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 315 [Warning] Aborted connection 315 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 320 [Warning] Aborted connection 320 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 304 [Warning] Aborted connection 304 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:26:21 306 [Warning] Aborted connection 306 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:26:21 322 [Warning] Aborted connection 322 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 318 [Warning] Aborted connection 318 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:26:21 316 [Warning] Aborted connection 316 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:26:21 319 [Warning] Aborted connection 319 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:26:21 323 [Warning] Aborted connection 323 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:26:21 326 [Warning] Aborted connection 326 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 321 [Warning] Aborted connection 321 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:26:21 314 [Warning] Aborted connection 314 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:26:28 503 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:26:30 522 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:31 495 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:33 523 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:26:42 529 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:27:13 544 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:27:13 548 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:27:13 547 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:27:13 361 [Warning] Aborted connection 361 to db: 'itstraining' user: 'itstraining' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:14 364 [Warning] Aborted connection 364 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:14 352 [Warning] Aborted connection 352 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:14 353 [Warning] Aborted connection 353 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:14 362 [Warning] Aborted connection 362 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:14 483 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:27:18 175 [Warning] Aborted connection 175 to db: 'gots' user: 'gotsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 21:27:18 373 [Warning] Aborted connection 373 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 312 [Warning] Aborted connection 312 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:27:18 553 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:27:18 376 [Warning] Aborted connection 376 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 552 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:27:18 381 [Warning] Aborted connection 381 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Table './wpstart/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 551 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:27:18 550 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:27:18 380 [Warning] Aborted connection 380 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Table './wpstart/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 556 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:27:18 557 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:27:18 397 [Warning] Aborted connection 397 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 42 [Warning] Aborted connection 42 to db: 'gots' user: 'gotsuser' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 21:27:18 384 [Warning] Aborted connection 384 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:27:18 518 [Warning] Aborted connection 518 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 21:27:18 516 [Warning] Aborted connection 516 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 21:27:18 391 [Warning] Aborted connection 391 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:27:18 402 [Warning] Aborted connection 402 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 559 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:27:18 396 [Warning] Aborted connection 396 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:27:18 389 [Warning] Aborted connection 389 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 558 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:27:18 390 [Warning] Aborted connection 390 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 453 [Warning] Aborted connection 453 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 458 [Warning] Aborted connection 458 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 392 [Warning] Aborted connection 392 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 401 [Warning] Aborted connection 401 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 452 [Warning] Aborted connection 452 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:18 461 [Warning] Aborted connection 461 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 403 [Warning] Aborted connection 403 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:18 460 [Warning] Aborted connection 460 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 455 [Warning] Aborted connection 455 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 454 [Warning] Aborted connection 454 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 536 [Warning] Aborted connection 536 to db: 'shockeripe' user: 'shockeripe' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 547 [Warning] Aborted connection 547 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 451 [Warning] Aborted connection 451 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 511 [Warning] Aborted connection 511 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 21:27:18 512 [Warning] Aborted connection 512 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 21:27:18 545 [Warning] Aborted connection 545 to db: 'unconnected' user: 'itstraining' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 21:27:18 404 [Warning] Aborted connection 404 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:27:18 560 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:27:18 456 [Warning] Aborted connection 456 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 476 [Warning] Aborted connection 476 to db: 'wpsem' user: 'wpsem' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 554 [Warning] Aborted connection 554 to db: 'unconnected' user: 'wpcomm305' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 21:27:18 555 [Warning] Aborted connection 555 to db: 'unconnected' user: 'wpcomm305' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 21:27:18 561 [Warning] Aborted connection 561 to db: 'unconnected' user: 'wpcomm303' host: '10.0.59.19' (Got an error reading communication packets) 2020-01-31 21:27:18 462 [Warning] Aborted connection 462 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:18 465 [Warning] Aborted connection 465 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:18 468 [Warning] Aborted connection 468 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 463 [Warning] Aborted connection 463 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 464 [Warning] Aborted connection 464 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 466 [Warning] Aborted connection 466 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 472 [Warning] Aborted connection 472 to db: 'wpsem' user: 'wpsem' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 485 [Warning] Aborted connection 485 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 529 [Warning] Aborted connection 529 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:18 457 [Warning] Aborted connection 457 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 488 [Warning] Aborted connection 488 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 450 [Warning] Aborted connection 450 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 500 [Warning] Aborted connection 500 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 398 [Warning] Aborted connection 398 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 478 [Warning] Aborted connection 478 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:27:18 491 [Warning] Aborted connection 491 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:27:18 400 [Warning] Aborted connection 400 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 459 [Warning] Aborted connection 459 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 487 [Warning] Aborted connection 487 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:18 474 [Warning] Aborted connection 474 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:18 480 [Warning] Aborted connection 480 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:18 477 [Warning] Aborted connection 477 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:18 475 [Warning] Aborted connection 475 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:18 489 [Warning] Aborted connection 489 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:18 490 [Warning] Aborted connection 490 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:18 498 [Warning] Aborted connection 498 to db: 'openrecords' user: 'openrecords' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 499 [Warning] Aborted connection 499 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 368 [Warning] Aborted connection 368 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 493 [Warning] Aborted connection 493 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 496 [Warning] Aborted connection 496 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 482 [Warning] Aborted connection 482 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 470 [Warning] Aborted connection 470 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 494 [Warning] Aborted connection 494 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 479 [Warning] Aborted connection 479 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 473 [Warning] Aborted connection 473 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 467 [Warning] Aborted connection 467 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 504 [Warning] Aborted connection 504 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 492 [Warning] Aborted connection 492 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 505 [Warning] Aborted connection 505 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 471 [Warning] Aborted connection 471 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 469 [Warning] Aborted connection 469 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 481 [Warning] Aborted connection 481 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 506 [Warning] Aborted connection 506 to db: 'pizzahut' user: 'pizzahut' host: '10.0.59.19' (Table './pizzahut/wp_options' is marked as crashed and should be) 2020-01-31 21:27:18 497 [Warning] Aborted connection 497 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:27:18 501 [Warning] Aborted connection 501 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:27:18 502 [Warning] Aborted connection 502 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:27:18 483 [Warning] Aborted connection 483 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:18 443 [Warning] Aborted connection 443 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:18 377 [Warning] Aborted connection 377 to db: 'ulrich' user: 'ulrich' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 394 [Warning] Aborted connection 394 to db: 'shockeripe' user: 'shockeripe' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 436 [Warning] Aborted connection 436 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 407 [Warning] Aborted connection 407 to db: 'wpsem' user: 'wpsem' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:18 503 [Warning] Aborted connection 503 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:27:19 528 [Warning] Aborted connection 528 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 533 [Warning] Aborted connection 533 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 413 [Warning] Aborted connection 413 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 399 [Warning] Aborted connection 399 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 495 [Warning] Aborted connection 495 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 414 [Warning] Aborted connection 414 to db: 'openrecords' user: 'openrecords' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 448 [Warning] Aborted connection 448 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 523 [Warning] Aborted connection 523 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:19 522 [Warning] Aborted connection 522 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 444 [Warning] Aborted connection 444 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 427 [Warning] Aborted connection 427 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 367 [Warning] Aborted connection 367 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 420 [Warning] Aborted connection 420 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 433 [Warning] Aborted connection 433 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 416 [Warning] Aborted connection 416 to db: 'pizzahut' user: 'pizzahut' host: '10.0.59.19' (Table './pizzahut/wp_options' is marked as crashed and should be) 2020-01-31 21:27:19 412 [Warning] Aborted connection 412 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:19 431 [Warning] Aborted connection 431 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 410 [Warning] Aborted connection 410 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Table './wpstart/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 531 [Warning] Aborted connection 531 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 430 [Warning] Aborted connection 430 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 435 [Warning] Aborted connection 435 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 372 [Warning] Aborted connection 372 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 311 [Warning] Aborted connection 311 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:19 371 [Warning] Aborted connection 371 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Table './wpstart/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 370 [Warning] Aborted connection 370 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 375 [Warning] Aborted connection 375 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 374 [Warning] Aborted connection 374 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 385 [Warning] Aborted connection 385 to db: 'itstraining' user: 'itstraining' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 382 [Warning] Aborted connection 382 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 378 [Warning] Aborted connection 378 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 525 [Warning] Aborted connection 525 to db: 'ulrich' user: 'ulrich' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 324 [Warning] Aborted connection 324 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:19 552 [Warning] Aborted connection 552 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 383 [Warning] Aborted connection 383 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Table './wpstart/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 386 [Warning] Aborted connection 386 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Table './wpstart/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 379 [Warning] Aborted connection 379 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 550 [Warning] Aborted connection 550 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:27:19 553 [Warning] Aborted connection 553 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Table './wpcomm305/comm305_options' is marked as crashed and sho) 2020-01-31 21:27:19 544 [Warning] Aborted connection 544 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 388 [Warning] Aborted connection 388 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 551 [Warning] Aborted connection 551 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Table './wpstart/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 556 [Warning] Aborted connection 556 to db: 'wpstart' user: 'wpstart' host: '10.0.59.19' (Table './wpstart/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 557 [Warning] Aborted connection 557 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 393 [Warning] Aborted connection 393 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:27:19 546 [Warning] Aborted connection 546 to db: 'itstraining' user: 'itstraining' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 395 [Warning] Aborted connection 395 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 387 [Warning] Aborted connection 387 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 527 [Warning] Aborted connection 527 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-01-31 21:27:19 558 [Warning] Aborted connection 558 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 559 [Warning] Aborted connection 559 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 548 [Warning] Aborted connection 548 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Table './wpwsutv/wp_options' is marked as crashed and should be ) 2020-01-31 21:27:19 560 [Warning] Aborted connection 560 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:31:32 167 [Warning] Aborted connection 167 to db: 'ksarn' user: 'root' host: 'localhost' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:31:40 566 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:31:45 567 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:31:52 568 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:31:58 574 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:32:09 575 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:32:58 575 [Warning] Aborted connection 575 to db: 'ksarn' user: 'root' host: 'localhost' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:34:25 578 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-01-31 21:34:35 579 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:38:46 579 [Warning] Aborted connection 579 to db: 'ksarn' user: 'root' host: 'localhost' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-01-31 21:39:22 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown 2020-01-31 21:39:22 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-01-31 21:39:22 0 [Note] InnoDB: FTS optimize thread exiting. 2020-01-31 21:39:22 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000065', position 245979 2020-01-31 21:39:24 0 [Note] InnoDB: Starting shutdown... 2020-01-31 21:39:24 0 [Note] InnoDB: Dumping buffer pool(s) to /data/mysql/ib_buffer_pool 2020-01-31 21:39:24 0 [Note] InnoDB: Instance 0, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-01-31 21:39:24 0 [Note] InnoDB: Instance 1, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-01-31 21:39:24 0 [Note] InnoDB: Instance 2, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-01-31 21:39:24 0 [Note] InnoDB: Instance 3, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-01-31 21:39:24 0 [Note] InnoDB: Instance 4, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-01-31 21:39:24 0 [Note] InnoDB: Instance 5, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-01-31 21:39:24 0 [Note] InnoDB: Instance 6, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-01-31 21:39:24 0 [Note] InnoDB: Instance 7, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-01-31 21:39:24 0 [Note] InnoDB: Buffer pool(s) dump completed at 200131 21:39:24 2020-01-31 21:39:25 0 [Note] InnoDB: Shutdown completed; log sequence number 5136194804; transaction id 5900851 2020-01-31 21:39:25 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 21:39:25 0 [Note] /usr/sbin/mysqld: Shutdown complete 2020-01-31 21:39:25 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 21:39:25 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 21:39:25 0 [Note] InnoDB: Uses event mutexes 2020-01-31 21:39:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 21:39:25 0 [Note] InnoDB: Number of pools: 1 2020-01-31 21:39:25 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 21:39:25 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 21:39:25 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 21:39:25 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 21:39:25 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 21:39:25 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 21:39:25 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 21:39:25 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 21:39:25 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 21:39:25 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136194804; transaction id 5900743 2020-01-31 21:39:25 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 21:39:25 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 21:39:25 0 [Note] Server socket created on IP: '::'. 2020-01-31 21:39:26 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 21:39:26 0 [Note] Added new Master_info '' to hash table 2020-01-31 21:39:26 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000065' at position 245979 2020-01-31 21:39:26 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 21:39:26 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 21:39:26 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000065' at position 245979 2020-01-31 21:39:26 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 21:39:26 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 21:39:26 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 21:39:26 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 21:39:26 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 21:39:26 2020-01-31 21:40:00 12 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:40:00 12 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 21:40:01 12 [Note] Zerofilling moved table: './mysql/proc' 2020-01-31 21:40:01 12 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 21:40:01 12 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 21:40:03 12 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 21:40:03 12 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 21:40:14 12 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:40:14 12 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 21:40:18 12 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:40:18 12 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 21:40:18 12 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:40:18 12 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 21:40:18 12 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 21:40:18 12 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 21:41:12 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 21:41:12 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000065', position 245979 2020-01-31 21:41:50 16 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 200131 21:42:09 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fed68000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fedc40c9cf0 thread_stack 0x49000 *** buffer overflow detected ***: /usr/sbin/mysqld terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7fede01ea567] /lib64/libc.so.6(+0x1166e2)[0x7fede01e86e2] /lib64/libc.so.6(+0x1184c7)[0x7fede01ea4c7] /usr/sbin/mysqld(my_addr_resolve+0xda)[0x5654af64794a] /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x5654af6300c2] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5654af0c63af] /lib64/libpthread.so.0(+0xf630)[0x7fede1e37630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5654aeff4151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5654af132545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5654aeee8ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5654aeeded4b] /usr/sbin/mysqld(+0x6ca792)[0x5654aeee9792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5654aef116da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5654aef1365d] /usr/sbin/mysqld(+0x65e3f3)[0x5654aee7d3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5654aee7ccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5654aef10b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5654aef1365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5654aef13761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5654aef14377] /usr/sbin/mysqld(+0x58fd84)[0x5654aedaed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5654aeebbb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5654aeed3b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5654aeed3cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5654aeed45e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5654aeed4677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5654aeec356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5654aeec507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5654aefa0afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5654aefa0bdd] /lib64/libpthread.so.0(+0x7ea5)[0x7fede1e2fea5] /lib64/libc.so.6(clone+0x6d)[0x7fede01d08cd] ======= Memory map: ======== 5654ae81f000-5654afc6d000 r-xp 00000000 fd:00 2101454 /usr/sbin/mysqld 5654afe6c000-5654aff98000 r--p 0144d000 fd:00 2101454 /usr/sbin/mysqld 5654aff98000-5654b0052000 rw-p 01579000 fd:00 2101454 /usr/sbin/mysqld 5654b0052000-5654b0901000 rw-p 00000000 00:00 0 5654b0f75000-5654b41fa000 rw-p 00000000 00:00 0 [heap] 7fed3c000000-7fed3c021000 rw-p 00000000 00:00 0 7fed3c021000-7fed40000000 ---p 00000000 00:00 0 7fed40000000-7fed40030000 rw-p 00000000 00:00 0 7fed40030000-7fed44000000 ---p 00000000 00:00 0 7fed44000000-7fed44021000 rw-p 00000000 00:00 0 7fed44021000-7fed48000000 ---p 00000000 00:00 0 7fed48000000-7fed48021000 rw-p 00000000 00:00 0 7fed48021000-7fed4c000000 ---p 00000000 00:00 0 7fed4c000000-7fed4c12a000 rw-p 00000000 00:00 0 7fed4c12a000-7fed50000000 ---p 00000000 00:00 0 7fed50000000-7fed50027000 rw-p 00000000 00:00 0 7fed50027000-7fed54000000 ---p 00000000 00:00 0 7fed54000000-7fed54021000 rw-p 00000000 00:00 0 7fed54021000-7fed58000000 ---p 00000000 00:00 0 7fed58000000-7fed5bda1000 rw-p 00000000 00:00 0 7fed5bda1000-7fed5c000000 ---p 00000000 00:00 0 7fed5cffa000-7fed5cffb000 ---p 00000000 00:00 0 7fed5cffb000-7fed5d7fb000 rw-p 00000000 00:00 0 7fed5d7fb000-7fed5d7fc000 ---p 00000000 00:00 0 7fed5d7fc000-7fed5dffc000 rw-p 00000000 00:00 0 7fed5dffc000-7fed5dffd000 ---p 00000000 00:00 0 7fed5dffd000-7fed5e7fd000 rw-p 00000000 00:00 0 7fed5e7fd000-7fed5e7fe000 ---p 00000000 00:00 0 7fed5e7fe000-7fed5effe000 rw-p 00000000 00:00 0 7fed5effe000-7fed5efff000 ---p 00000000 00:00 0 7fed5efff000-7fed5f7ff000 rw-p 00000000 00:00 0 7fed5f7ff000-7fed5f800000 ---p 00000000 00:00 0 7fed5f800000-7fed60000000 rw-p 00000000 00:00 0 7fed60000000-7fed603d9000 rw-p 00000000 00:00 0 7fed603d9000-7fed64000000 ---p 00000000 00:00 0 7fed64000000-7fed643d1000 rw-p 00000000 00:00 0 7fed643d1000-7fed68000000 ---p 00000000 00:00 0 7fed68000000-7fed683d2000 rw-p 00000000 00:00 0 7fed683d2000-7fed6c000000 ---p 00000000 00:00 0 7fed6c000000-7fed6c3ea000 rw-p 00000000 00:00 0 7fed6c3ea000-7fed70000000 ---p 00000000 00:00 0 7fed70000000-7fed70021000 rw-p 00000000 00:00 0 7fed70021000-7fed74000000 ---p 00000000 00:00 0 7fed742ed000-7fed742ff000 r-xp 00000000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7fed742ff000-7fed744fe000 ---p 00012000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7fed744fe000-7fed74501000 r--p 00011000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7fed74501000-7fed74502000 rw-p 00014000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7fed74502000-7fed74508000 r-xp 00000000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7fed74508000-7fed74707000 ---p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7fed74707000-7fed74708000 r--p 00005000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7fed74708000-7fed74709000 rw-p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7fed74709000-7fed7470a000 ---p 00000000 00:00 0 7fed7470a000-7fed74f0a000 rw-p 00000000 00:00 0 7fed74f0a000-7fed74f0b000 ---p 00000000 00:00 0 7fed74f0b000-7fed7570b000 rw-p 00000000 00:00 0 7fed7570b000-7fed7570c000 ---p 00000000 00:00 0 7fed7570c000-7fed75f0c000 rw-p 00000000 00:00 0 7fed75f0c000-7fed75f0d000 ---p 00000000 00:00 0 7fed75f0d000-7fed7670d000 rw-p 00000000 00:00 0 7fed7670d000-7fed7670e000 ---p 00000000 00:00 0 7fed7670e000-7fed76f0e000 rw-p 00000000 00:00 0 7fed76f0e000-7fed76f0f000 ---p 00000000 00:00 0 7fed76f0f000-7fed7770f000 rw-p 00000000 00:00 0 7fed7770f000-7fed77710000 ---p 00000000 00:00 0 7fed77710000-7fed77f10000 rw-p 00000000 00:00 0 7fed77f10000-7fed77f11000 ---p 00000000 00:00 0 7fed77f11000-7fed78711000 rw-p 00000000 00:00 0 7fed78711000-7fed78712000 ---p 00000000 00:00 0 7fed78712000-7fed78f12000 rw-p 00000000 00:00 0 7fed78f12000-7fed78f13000 ---p 00000000 00:00 0 7fed78f13000-7fed79713000 rw-p 00000000 00:00 0 7fed79713000-7fed79714000 ---p 00000000 00:00 0 7fed79714000-7fed79f14000 rw-p 00000000 00:00 0 7fed79f14000-7fed79f15000 ---p 00000000 00:00 0 7fed79f15000-7fed7a715000 rw-p 00000000 00:00 0 7fed7a715000-7fed7a716000 ---p 00000000 00:00 0 7fed7a716000-7fed7af16000 rw-p 00000000 00:00 0 7fed7af16000-7fed7af17000 ---p 00000000 00:00 0 7fed7af17000-7fed7b717000 rw-p 00000000 00:00 0 7fed7b717000-7fed7b718000 ---p 00000000 00:00 0 7fed7b718000-7fed7bf18000 rw-p 00000000 00:00 0 7fed7bf18000-7fed7bf19000 ---p 00000000 00:00 0 7fed7bf19000-7fed7c719000 rw-p 00000000 00:00 0 7fed7c719000-7fed7c71a000 ---p 00000000 00:00 0 7fed7c71a000-7fed7cf1a000 rw-p 00000000 00:00 0 7fed7cf1a000-7fed7cf1b000 ---p 00000000 00:00 0 7fed7cf1b000-7fed7d71b000 rw-p 00000000 00:00 0 7fed7d71b000-7fed7d71c000 ---p 00000000 00:00 0 7fed7d71c000-7fed7df1c000 rw-p 00000000 00:00 0 7fed7df1c000-7fed7df1d000 ---p 00000000 00:00 0 7fed7df1d000-7fed7e71d000 rw-p 00000000 00:00 0 7fed7e71d000-7fed7e71e000 ---p 00000000 00:00 0 7fed7e71e000-7fed7ef1e000 rw-p 00000000 00:00 0 7fed7effe000-7fed7efff000 ---p 00000000 00:00 0 7fed7efff000-7fed7f7ff000 rw-p 00000000 00:00 0 7fed7f7ff000-7fed7f800000 ---p 00000000 00:00 0 7fed7f800000-7fed80000000 rw-p 00000000 00:00 0 7fed80000000-7fedc0000000 rw-p 00000000 00:00 0 7fedc0000000-7fedc0070000 rw-p 00000000 00:00 0 7fedc0070000-7fedc4000000 ---p 00000000 00:00 0 7fedc4080000-7fedc4081000 ---p 00000000 00:00 0 7fedc4081000-7fedc40cb000 rw-p 00000000 00:00 0 7fedc40cb000-7fedc40d7000 r-xp 00000000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7fedc40d7000-7fedc42d6000 ---p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7fedc42d6000-7fedc42d7000 r--p 0000b000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7fedc42d7000-7fedc42d8000 rw-p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7fedc42d8000-7fedc4c7e000 rw-p 00000000 00:00 0 7fedc4cbc000-7fedc4cbd000 ---p 00000000 00:00 0 7fedc4cbd000-7fedc4d07000 rw-p 00000000 00:00 0 7fedc4d07000-7fedc4d08000 ---p 00000000 00:00 0 7fedc4d08000-7fedc4d52000 rw-p 00000000 00:00 0 7fedc4d52000-7fedc4d53000 ---p 00000000 00:00 0 7fedc4d53000-7fedc4d9d000 rw-p 00000000 00:00 0 7fedc4d9d000-7fedc4d9e000 ---p 00000000 00:00 0 7fedc4d9e000-7fedc4de8000 rw-p 00000000 00:00 0 7fedc4de8000-7fedc4de9000 ---p 00000000 00:00 0 7fedc4de9000-7fedc4e33000 rw-p 00000000 00:00 0 7fedc4e33000-7fedc4e34000 ---p 00000000 00:00 0 7fedc4e34000-7fedc4e7e000 rw-p 00000000 00:00 0 7fedc4e7e000-7fedc6e7e000 rw-p 00000000 00:00 0 7fedc6e7e000-7fedc7d49000 rw-p 00000000 00:00 0 7fedc7d49000-7fedc7d4a000 ---p 00000000 00:00 0 7fedc7d4a000-7fedd8000000 rw-p 00000000 00:00 0 7fedd8000000-7fedd8021000 rw-p 00000000 00:00 0 7fedd8021000-7feddc000000 ---p 00000000 00:00 0 7feddc01c000-7feddcd6f000 rw-p 00000000 00:00 0 7feddcd6f000-7feddcd70000 ---p 00000000 00:00 0 7feddcd70000-7feddd570000 rw-p 00000000 00:00 0 7feddd570000-7feddd573000 r-xp 00000000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7feddd573000-7feddd772000 ---p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7feddd772000-7feddd773000 r--p 00002000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7feddd773000-7feddd774000 rw-p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7feddd774000-7feddd782000 r-xp 00000000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7feddd782000-7feddd982000 ---p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7feddd982000-7feddd983000 r--p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7feddd983000-7feddd984000 rw-p 0000f000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7feddd984000-7feddd993000 r-xp 00000000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7feddd993000-7fedddb92000 ---p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7fedddb92000-7fedddb93000 r--p 0000e000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7fedddb93000-7fedddb94000 rw-p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7fedddb94000-7fedddbab000 r-xp 00000000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7fedddbab000-7feddddaa000 ---p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7feddddaa000-7feddddab000 r--p 00016000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7feddddab000-7feddddac000 rw-p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7feddddac000-7feddde0c000 r-xp 00000000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7feddde0c000-7fedde00c000 ---p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7fedde00c000-7fedde00d000 r--p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7fedde00d000-7fedde00e000 rw-p 00061000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7fedde00e000-7fedde012000 r-xp 00000000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7fedde012000-7fedde211000 ---p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7fedde211000-7fedde212000 r--p 00003000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7fedde212000-7fedde213000 rw-p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7fedde213000-7fedde244000 r-xp 00000000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7fedde244000-7fedde443000 ---p 00031000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7fedde443000-7fedde445000 r--p 00030000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7fedde445000-7fedde446000 rw-p 00032000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7fedde446000-7fedde449000 r-xp 00000000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7fedde449000-7fedde648000 ---p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7fedde648000-7fedde649000 r--p 00002000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7fedde649000-7fedde64a000 rw-p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7fedde64a000-7fedde723000 r-xp 00000000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7fedde723000-7fedde922000 ---p 000d9000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7fedde922000-7fedde930000 r--p 000d8000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7fedde930000-7fedde933000 rw-p 000e6000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7fedde933000-7fedde97d000 r-xp 00000000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7fedde97d000-7feddeb7d000 ---p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7feddeb7d000-7feddeb7e000 r--p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7feddeb7e000-7feddeb80000 rw-p 0004b000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7feddeb80000-7feddeb95000 r-xp 00000000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7feddeb95000-7fedded94000 ---p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fedded94000-7fedded95000 r--p 00014000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fedded95000-7fedded96000 rw-p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fedded96000-7feddede4000 r-xp 00000000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7feddede4000-7feddefe4000 ---p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7feddefe4000-7feddefe6000 r--p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7feddefe6000-7feddefe7000 rw-p 00050000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7feddefe7000-7feddeffd000 r-xp 00000000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7feddeffd000-7feddf1fc000 ---p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7feddf1fc000-7feddf1fd000 r--p 00015000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7feddf1fd000-7feddf1fe000 rw-p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7feddf1fe000-7feddf200000 rw-p 00000000 00:00 0 7feddf200000-7feddf204000 r-xp 00000000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7feddf204000-7feddf403000 ---p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7feddf403000-7feddf404000 r--p 00003000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7feddf404000-7feddf405000 rw-p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7feddf405000-7feddf482000 r-xp 00000000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7feddf482000-7feddf681000 ---p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7feddf681000-7feddf682000 r--p 0007c000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7feddf682000-7feddf685000 rw-p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7feddf685000-7feddf686000 rw-p 00000000 00:00 0 7feddf686000-7feddf69a000 r-xp 00000000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7feddf69a000-7feddf899000 ---p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7feddf899000-7feddf89a000 r--p 00013000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7feddf89a000-7feddf89b000 rw-p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7feddf89b000-7feddf8bf000 r-xp 00000000 fd:00 2101234 /usr/lib64/libselinux.so.1 7feddf8bf000-7feddfabe000 ---p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7feddfabe000-7feddfabf000 r--p 00023000 fd:00 2101234 /usr/lib64/libselinux.so.1 7feddfabf000-7feddfac0000 rw-p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7feddfac0000-7feddfac2000 rw-p 00000000 00:00 0 7feddfac2000-7feddfac9000 r-xp 00000000 fd:00 2101451 /usr/lib64/librt-2.17.so 7feddfac9000-7feddfcc8000 ---p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7feddfcc8000-7feddfcc9000 r--p 00006000 fd:00 2101451 /usr/lib64/librt-2.17.so 7feddfcc9000-7feddfcca000 rw-p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7feddfcca000-7feddfcce000 r-xp 00000000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7feddfcce000-7feddfecd000 ---p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7feddfecd000-7feddfece000 r--p 00003000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7feddfece000-7feddfecf000 rw-p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7feddfecf000-7feddfed1000 r-xp 00000000 fd:00 2227367 /usr/lib64/libfreebl3.so 7feddfed1000-7fede00d0000 ---p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7fede00d0000-7fede00d1000 r--p 00001000 fd:00 2227367 /usr/lib64/libfreebl3.so 7fede00d1000-7fede00d2000 rw-p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7fede00d2000-7fede0295000 r-xp 00000000 fd:00 2097970 /usr/lib64/libc-2.17.so 7fede0295000-7fede0495000 ---p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7fede0495000-7fede0499000 r--p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7fede0499000-7fede049b000 rw-p 001c7000 fd:00 2097970 /usr/lib64/libc-2.17.so 7fede049b000-7fede04a0000 rw-p 00000000 00:00 0 7fede04a0000-7fede05a1000 r-xp 00000000 fd:00 2097978 /usr/lib64/libm-2.17.so 7fede05a1000-7fede07a0000 ---p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7fede07a0000-7fede07a1000 r--p 00100000 fd:00 2097978 /usr/lib64/libm-2.17.so 7fede07a1000-7fede07a2000 rw-p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7fede07a2000-7fede088b000 r-xp 00000000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7fede088b000-7fede0a8a000 ---p 000e9000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7fede0a8a000-7fede0a92000 r--p 000e8000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7fede0a92000-7fede0a94000 rw-p 000f0000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7fede0a94000-7fede0aa9000 rw-p 00000000 00:00 0 7fede0aa9000-7fede0aab000 r-xp 00000000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7fede0aab000-7fede0cab000 ---p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7fede0cab000-7fede0cac000 r--p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7fede0cac000-7fede0cad000 rw-p 00003000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7fede0cad000-7fede0ee3000 r-xp 00000000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7fede0ee3000-7fede10e3000 ---p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7fede10e3000-7fede10ff000 r--p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7fede10ff000-7fede110c000 rw-p 00252000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7fede110c000-7fede1110000 rw-p 00000000 00:00 0 7fede1110000-7fede1177000 r-xp 00000000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7fede1177000-7fede1377000 ---p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7fede1377000-7fede137b000 r--p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7fede137b000-7fede1382000 rw-p 0006b000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7fede1382000-7fede13b1000 r-xp 00000000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7fede13b1000-7fede15b1000 ---p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7fede15b1000-7fede15b2000 r--p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7fede15b2000-7fede15b3000 rw-p 00030000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7fede15b3000-7fede15bb000 r-xp 00000000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7fede15bb000-7fede17ba000 ---p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7fede17ba000-7fede17bb000 r--p 00007000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7fede17bb000-7fede17bc000 rw-p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7fede17bc000-7fede17ea000 rw-p 00000000 00:00 0 7fede17ea000-7fede17ff000 r-xp 00000000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7fede17ff000-7fede19fe000 ---p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7fede19fe000-7fede19ff000 r--p 00014000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7fede19ff000-7fede1a00000 rw-p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7fede1a00000-7fede1a01000 r-xp 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7fede1a01000-7fede1c00000 ---p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7fede1c00000-7fede1c01000 r--p 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7fede1c01000-7fede1c02000 rw-p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7fede1c02000-7fede1c27000 r-xp 00000000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7fede1c27000-7fede1e26000 ---p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7fede1e26000-7fede1e27000 r--p 00024000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7fede1e27000-7fede1e28000 rw-p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7fede1e28000-7fede1e3f000 r-xp 00000000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7fede1e3f000-7fede203e000 ---p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7fede203e000-7fede203f000 r--p 00016000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7fede203f000-7fede2040000 rw-p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7fede2040000-7fede2044000 rw-p 00000000 00:00 0 7fede2044000-7fede2066000 r-xp 00000000 fd:00 2227369 /usr/lib64/ld-2.17.so 7fede206a000-7fede206b000 ---p 00000000 00:00 0 7fede206b000-7fede20b6000 rw-p 00000000 00:00 0 7fede20b6000-7fede20b9000 rw-s 00000000 00:0b 50518329 /[aio] (deleted) 7fede20b9000-7fede20bc000 rw-s 00000000 00:0b 50518328 /[aio] (deleted) 7fede20bc000-7fede20bf000 rw-s 00000000 00:0b 50518327 /[aio] (deleted) 7fede20bf000-7fede20c2000 rw-s 00000000 00:0b 50518326 /[aio] (deleted) 7fede20c2000-7fede20c5000 rw-s 00000000 00:0b 50518325 /[aio] (deleted) 7fede20c5000-7fede20c8000 rw-s 00000000 00:0b 50518324 /[aio] (deleted) 7fede20c8000-7fede20cb000 rw-s 00000000 00:0b 50518323 /[aio] (deleted) 7fede20cb000-7fede20ce000 rw-s 00000000 00:0b 50518322 /[aio] (deleted) 7fede20ce000-7fede225c000 rw-p 00000000 00:00 0 7fede225c000-7fede225d000 rw-s 00000000 00:0b 50518330 /[aio] (deleted) 7fede225d000-7fede2260000 rw-s 00000000 00:0b 50518321 /[aio] (deleted) 7fede2260000-7fede2263000 rw-s 00000000 00:0b 50518320 /[aio] (deleted) 7fede2263000-7fede2264000 rw-s 00000000 00:0b 50518319 /[aio] (deleted) 7fede2264000-7fede2265000 rw-p 00000000 00:00 0 7fede2265000-7fede2266000 r--p 00021000 fd:00 2227369 /usr/lib64/ld-2.17.so 7fede2266000-7fede2267000 rw-p 00022000 fd:00 2227369 /usr/lib64/ld-2.17.so 7fede2267000-7fede2268000 rw-p 00000000 00:00 0 7ffc7a4d9000-7ffc7a4fa000 rw-p 00000000 00:00 0 [stack] 7ffc7a5f9000-7ffc7a5fb000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Fatal signal 6 while backtracing 2020-01-31 21:42:17 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 21:42:17 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 21:42:17 0 [Note] InnoDB: Uses event mutexes 2020-01-31 21:42:17 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 21:42:17 0 [Note] InnoDB: Number of pools: 1 2020-01-31 21:42:17 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 21:42:17 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 21:42:17 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 21:42:17 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 21:42:17 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136197795 2020-01-31 21:42:17 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000145', position 431341 2020-01-31 21:42:17 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 21:42:17 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 21:42:17 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 21:42:17 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 21:42:17 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 21:42:17 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 21:42:17 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136197804; transaction id 5900754 2020-01-31 21:42:17 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 21:42:17 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 21:42:17 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 21:42:17 0 [Note] Starting crash recovery... 2020-01-31 21:42:17 0 [Note] Crash recovery finished. 2020-01-31 21:42:17 0 [Note] Server socket created on IP: '::'. 2020-01-31 21:42:17 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 21:42:17 0 [Note] Added new Master_info '' to hash table 2020-01-31 21:42:17 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000065' at position 245979 2020-01-31 21:42:17 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 21:42:17 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 21:42:17 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000065' at position 245979 2020-01-31 21:42:17 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 21:42:17 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 21:42:17 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 21:42:17 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 21:42:17 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 21:42:17 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 21:42:17 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 21:42:17 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 21:42:17 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 21:42:19 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 21:42:19 2020-01-31 21:42:45 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 21:42:50 16 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 21:42:50 16 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 21:43:39 10 [Note] Slave: received end packet from server, apparent master shutdown: 2020-01-31 21:43:39 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000065' at position 666488285 2020-01-31 21:43:39 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 21:43:46 46 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000146, 302328) 2020-01-31 21:43:50 49 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:43:50 49 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 21:43:52 50 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 21:43:52 50 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 21:44:05 54 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 21:44:05 54 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 21:44:39 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000065' at position 666488285 2020-01-31 21:45:37 90 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 21:45:37 90 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 21:45:38 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f5100000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f515c0a9cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5574f092df2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5574f03c43af] sigaction.c:0(__restore_rt)[0x7f5177242630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5574f02f2151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5574f0430545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5574f01e6ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5574f01dcd4b] /usr/sbin/mysqld(+0x6ca792)[0x5574f01e7792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5574f020f6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5574f021165d] /usr/sbin/mysqld(+0x65e3f3)[0x5574f017b3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5574f017acb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5574f020eb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5574f021165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5574f0211761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5574f0212377] /usr/sbin/mysqld(+0x58fd84)[0x5574f00acd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5574f01b9b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5574f01d1b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5574f01d1cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5574f01d25e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5574f01d2677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5574f01c156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5574f01c307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5574f029eafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5574f029ebdd] pthread_create.c:0(start_thread)[0x7f517723aea5] /lib64/libc.so.6(clone+0x6d)[0x7f51755db8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f510000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 91 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 21:45:44 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 21:45:44 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 21:45:44 0 [Note] InnoDB: Uses event mutexes 2020-01-31 21:45:44 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 21:45:44 0 [Note] InnoDB: Number of pools: 1 2020-01-31 21:45:44 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 21:45:44 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 21:45:44 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 21:45:44 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 21:45:44 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136275680 2020-01-31 21:45:44 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000142', position 1237971 2020-01-31 21:45:45 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 21:45:45 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 21:45:45 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 21:45:45 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 21:45:45 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 21:45:45 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 21:45:45 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136275689; transaction id 5900850 2020-01-31 21:45:45 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 21:45:45 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 21:45:45 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 21:45:45 0 [Note] Starting crash recovery... 2020-01-31 21:45:45 0 [Note] Crash recovery finished. 2020-01-31 21:45:45 0 [Note] Server socket created on IP: '::'. 2020-01-31 21:45:45 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 21:45:45 0 [Note] Added new Master_info '' to hash table 2020-01-31 21:45:45 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000067' at position 146167970 2020-01-31 21:45:45 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 21:45:45 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 21:45:45 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 21:45:45 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 21:45:45 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 21:45:45 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 21:45:45 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000067' at position 146167970 2020-01-31 21:45:45 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 21:45:45 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 21:45:45 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 21:45:45 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 21:45:45 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 21:45:46 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 21:45:46 2020-01-31 21:46:11 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 21:46:12 16 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 21:46:12 16 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 21:46:13 16 [ERROR] mysqld: Table './wpcomm303/wp_wfConfig' is marked as crashed and should be repaired 2020-01-31 21:46:13 16 [Warning] Checking table: './wpcomm303/wp_wfConfig' 2020-01-31 21:46:14 17 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:46:14 17 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 21:46:15 19 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 21:46:15 19 [Warning] Checking table: './wpms/wp_options' 2020-01-31 21:46:15 18 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 21:46:15 18 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 21:46:16 21 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 21:46:16 21 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 21:46:17 23 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:46:17 23 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 21:46:17 24 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 21:46:17 24 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 21:46:28 31 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 21:46:28 31 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 21:48:13 99 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000002, 332) 2020-01-31 21:48:13 100 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:48:13 100 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 21:48:49 115 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 21:48:49 115 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 21:48:49 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0394000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0351052cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55557c455f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55557beec3af] sigaction.c:0(__restore_rt)[0x7f03b2279630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55557be1a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55557bf58545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55557bd0eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55557bd04d4b] /usr/sbin/mysqld(+0x6ca792)[0x55557bd0f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55557bd376da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55557bd3965d] /usr/sbin/mysqld(+0x65e3f3)[0x55557bca33f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55557bca2cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55557bd36b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55557bd3965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55557bd39761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55557bd3a377] /usr/sbin/mysqld(+0x58fd84)[0x55557bbd4d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55557bce1b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55557bcf9b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55557bcf9cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55557bcfa5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55557bcfa677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55557bce956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55557bceb07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55557bdc6afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55557bdc6bdd] pthread_create.c:0(start_thread)[0x7f03b2271ea5] /lib64/libc.so.6(clone+0x6d)[0x7f03b06128cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f039400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 117 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 21:48:56 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 21:48:56 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 21:48:56 0 [Note] InnoDB: Uses event mutexes 2020-01-31 21:48:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 21:48:56 0 [Note] InnoDB: Number of pools: 1 2020-01-31 21:48:56 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 21:48:56 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 21:48:56 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 21:48:56 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 21:48:56 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136279417 2020-01-31 21:48:56 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000142', position 1237971 2020-01-31 21:48:56 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 21:48:56 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 21:48:56 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 21:48:56 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 21:48:56 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 21:48:56 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 21:48:56 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136279426; transaction id 5900865 2020-01-31 21:48:56 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 21:48:56 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 21:48:56 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 21:48:56 0 [Note] Starting crash recovery... 2020-01-31 21:48:56 0 [Note] Crash recovery finished. 2020-01-31 21:48:56 0 [Note] Server socket created on IP: '::'. 2020-01-31 21:48:56 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 21:48:56 0 [Note] Added new Master_info '' to hash table 2020-01-31 21:48:56 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000067' at position 402434931 2020-01-31 21:48:56 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 21:48:56 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 21:48:56 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000067' at position 402434931 2020-01-31 21:48:56 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 21:48:56 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 21:48:56 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 21:48:56 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 21:48:56 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 21:48:56 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 21:48:56 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 21:48:56 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 21:48:56 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 21:48:57 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 21:48:57 2020-01-31 21:49:21 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 21:49:22 14 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:49:22 14 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 21:49:24 17 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 21:49:24 17 [Warning] Checking table: './wpms/wp_options' 2020-01-31 21:49:25 17 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 21:49:25 17 [Warning] Checking table: './wpms/wp_usermeta' 2020-01-31 21:49:42 39 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 21:49:42 39 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 21:49:50 44 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000004, 944380) 2020-01-31 21:50:40 56 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 21:50:40 56 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 21:50:41 58 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 21:50:41 58 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 21:50:41 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f416c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f41ccff4cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55fd26e2bf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55fd268c23af] sigaction.c:0(__restore_rt)[0x7f41e813d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55fd267f0151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55fd2692e545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55fd266e4ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55fd266dad4b] /usr/sbin/mysqld(+0x6ca792)[0x55fd266e5792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55fd2670d6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55fd2670f65d] /usr/sbin/mysqld(+0x65e3f3)[0x55fd266793f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55fd26678cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55fd2670cb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55fd2670f65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55fd2670f761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55fd26710377] /usr/sbin/mysqld(+0x58fd84)[0x55fd265aad84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55fd266b7b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55fd266cfb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55fd266cfcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55fd266d05e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55fd266d0677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55fd266bf56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55fd266c107c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55fd2679cafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55fd2679cbdd] pthread_create.c:0(start_thread)[0x7f41e8135ea5] /lib64/libc.so.6(clone+0x6d)[0x7f41e64d68cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f416c00feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 59 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 21:50:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 21:50:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 21:50:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 21:50:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 21:50:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 21:50:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 21:50:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 21:50:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 21:50:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 21:50:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136282457 2020-01-31 21:50:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000142', position 1237971 2020-01-31 21:50:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 21:50:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 21:50:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 21:50:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 21:50:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 21:50:47 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 21:50:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136282466; transaction id 5900876 2020-01-31 21:50:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 21:50:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 21:50:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 21:50:47 0 [Note] Starting crash recovery... 2020-01-31 21:50:47 0 [Note] Crash recovery finished. 2020-01-31 21:50:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 21:50:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 21:50:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 21:50:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000067' at position 402434931 2020-01-31 21:50:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000058' at position 535047, relay log './dbsvc-528-relay-bin.000266' position: 559 2020-01-31 21:50:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 21:50:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000067' at position 402434931 2020-01-31 21:50:47 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 21:50:47 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 21:50:47 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 21:50:47 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 21:50:47 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table ojsdata.sessions; Table './ojsdata/sessions' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000058, end_log_pos 535583, Gtid 0-1-2480602, Internal MariaDB error code: 1062 2020-01-31 21:50:47 11 [Warning] Slave: Table './ojsdata/sessions' is marked as crashed and should be repaired Error_code: 145 2020-01-31 21:50:47 11 [Warning] Slave: Duplicate entry 'ab54a752d648471603733ed193c5e303' for key 'sessions_pkey' Error_code: 1062 2020-01-31 21:50:47 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000058' position 535047 2020-01-31 21:50:47 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000058' at position 535047 2020-01-31 21:50:47 10 [ERROR] Error reading packet from server: Could not find first log file name in binary log index file (server_errno=1236) 2020-01-31 21:50:47 10 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file', Internal MariaDB error code: 1236 2020-01-31 21:50:47 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000067', position 402434931 2020-01-31 21:50:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 21:50:48 2020-01-31 21:51:16 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 21:51:18 14 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:51:18 14 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 21:51:19 16 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 21:51:19 16 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 21:51:19 16 [ERROR] mysqld: Table './wpcomm306/wp_wfConfig' is marked as crashed and should be repaired 2020-01-31 21:51:19 16 [Warning] Checking table: './wpcomm306/wp_wfConfig' 2020-01-31 21:51:19 15 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 21:51:19 15 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 21:51:20 18 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 21:51:20 18 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 21:51:24 21 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:51:24 21 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 21:51:40 31 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2020-01-31 21:51:40 31 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='dbsvc-527.wichita.edu', master_port='3306', master_log_file='dbsvc-527-bin.000067', master_log_pos='402434931'. New state master_host='dbsvc-527.wichita.edu', master_port='3306', master_log_file='dbsvc-527-bin.000001', master_log_pos='332'. 2020-01-31 21:51:42 31 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2020-01-31 21:51:42 33 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000005, 103333) 2020-01-31 21:51:42 31 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='dbsvc-527.wichita.edu', master_port='3306', master_log_file='dbsvc-527-bin.000001', master_log_pos='332'. New state master_host='dbsvc-527.wichita.edu', master_port='3306', master_log_file='dbsvc-527-bin.000001', master_log_pos='332'. 2020-01-31 21:51:46 36 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 21:51:46 36 [Warning] Checking table: './wpms/wp_options' 2020-01-31 21:52:19 57 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000001' at position 332 2020-01-31 21:52:19 58 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000001' at position 332, relay log './dbsvc-528-relay-bin.000001' position: 4 2020-01-31 21:52:19 57 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000001' at position 332 2020-01-31 21:53:34 93 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 21:53:34 93 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 21:55:24 134 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 21:55:24 134 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 21:55:40 144 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 21:55:40 144 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 21:55:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe5d4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe574154cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ab0449df2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ab03f343af] sigaction.c:0(__restore_rt)[0x7fe5dd2cc630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55ab03e62151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55ab03fa0545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55ab03d56ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55ab03d4cd4b] /usr/sbin/mysqld(+0x6ca792)[0x55ab03d57792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55ab03d7f6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ab03d8165d] /usr/sbin/mysqld(+0x65e3f3)[0x55ab03ceb3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55ab03ceacb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55ab03d7eb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ab03d8165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55ab03d81761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55ab03d82377] /usr/sbin/mysqld(+0x58fd84)[0x55ab03c1cd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55ab03d29b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55ab03d41b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55ab03d41cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55ab03d425e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55ab03d42677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55ab03d3156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55ab03d3307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55ab03e0eafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55ab03e0ebdd] pthread_create.c:0(start_thread)[0x7fe5dd2c4ea5] /lib64/libc.so.6(clone+0x6d)[0x7fe5db6658cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe5d400ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 145 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 21:55:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 21:55:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 21:55:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 21:55:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 21:55:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 21:55:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 21:55:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 21:55:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 21:55:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 21:55:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136292661 2020-01-31 21:55:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 21:55:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 21:55:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 21:55:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 21:55:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 21:55:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 21:55:47 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 21:55:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136292670; transaction id 5900913 2020-01-31 21:55:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 21:55:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 21:55:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 21:55:47 0 [Note] Starting crash recovery... 2020-01-31 21:55:47 0 [Note] Crash recovery finished. 2020-01-31 21:55:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 21:55:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 21:55:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 21:55:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000001' at position 332 2020-01-31 21:55:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000001' at position 332 2020-01-31 21:55:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000001' at position 332, relay log './dbsvc-528-relay-bin.000002' position: 559 2020-01-31 21:55:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 21:55:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 21:55:48 2020-01-31 21:55:51 11 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 21:55:51 11 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 21:55:51 11 [Warning] wpstart.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 21:55:51 11 [Note] wpstart.wp_options: Table is fixed 2020-01-31 21:55:52 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 21:55:52 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 21:55:52 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 21:55:52 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 21:56:12 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 21:56:13 15 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 21:56:13 15 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 21:56:16 17 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:56:16 17 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 21:56:17 18 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 21:56:17 18 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 21:56:18 21 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 21:56:18 21 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 21:56:32 28 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 21:56:32 28 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 21:56:42 33 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000006, 452489) 2020-01-31 21:57:11 50 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 21:57:11 50 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 21:58:07 70 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:58:07 70 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 21:58:13 74 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 21:58:13 74 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 21:58:27 85 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 21:58:27 85 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 21:58:27 87 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 21:58:27 87 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 21:58:28 87 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 21:58:28 87 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 21:58:28 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fa96c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fa95d398cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x557351d02f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5573517993af] sigaction.c:0(__restore_rt)[0x7fa97854c630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5573516c7151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x557351805545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5573515bbccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5573515b1d4b] /usr/sbin/mysqld(+0x6ca792)[0x5573515bc792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5573515e46da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5573515e665d] /usr/sbin/mysqld(+0x65e3f3)[0x5573515503f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55735154fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5573515e3b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5573515e665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5573515e6761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5573515e7377] /usr/sbin/mysqld(+0x58fd84)[0x557351481d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55735158eb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5573515a6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5573515a6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5573515a75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5573515a7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55735159656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55735159807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x557351673afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x557351673bdd] pthread_create.c:0(start_thread)[0x7fa978544ea5] /lib64/libc.so.6(clone+0x6d)[0x7fa9768e58cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fa96c00ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 88 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 21:58:34 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 21:58:34 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 21:58:34 0 [Note] InnoDB: Uses event mutexes 2020-01-31 21:58:34 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 21:58:34 0 [Note] InnoDB: Number of pools: 1 2020-01-31 21:58:34 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 21:58:34 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 21:58:34 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 21:58:34 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 21:58:34 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136333093 2020-01-31 21:58:34 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 21:58:34 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 21:58:34 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 21:58:34 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 21:58:34 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 21:58:34 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 21:58:34 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136333102; transaction id 5901075 2020-01-31 21:58:34 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 21:58:34 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 21:58:34 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 21:58:34 0 [Note] Starting crash recovery... 2020-01-31 21:58:34 0 [Note] Crash recovery finished. 2020-01-31 21:58:35 0 [Note] Server socket created on IP: '::'. 2020-01-31 21:58:35 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 21:58:35 0 [Note] Added new Master_info '' to hash table 2020-01-31 21:58:35 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000001' at position 336396 2020-01-31 21:58:35 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000001' at position 336396, relay log './dbsvc-528-relay-bin.000004' position: 220504 2020-01-31 21:58:35 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 21:58:35 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000001' at position 336396 2020-01-31 21:58:36 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 21:58:36 2020-01-31 21:58:38 11 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 21:58:38 11 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 21:58:38 11 [Warning] ulrich.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 21:58:38 11 [Note] ulrich.wp_options: Table is fixed 2020-01-31 21:58:42 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 21:58:42 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 21:58:42 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 21:58:42 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 21:58:58 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 21:58:58 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 21:58:58 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 21:58:58 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 21:58:58 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 21:58:58 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 21:58:58 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 21:58:58 11 [Note] ojsdata.scheduled_tasks: Table is fixed 2020-01-31 21:59:01 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 21:59:01 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000001' at position 432390 2020-01-31 21:59:02 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 21:59:03 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 21:59:07 14 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000007, 236022) 2020-01-31 21:59:27 20 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 21:59:27 20 [Warning] Checking table: './wpms/wp_options' 2020-01-31 21:59:36 31 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 21:59:36 31 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 21:59:37 32 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 21:59:37 32 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 21:59:49 45 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 21:59:49 45 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:00:02 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000001' at position 432390 200131 22:00:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f7e40000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f7ea485fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c30721af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c306cb13af] sigaction.c:0(__restore_rt)[0x7f7ebf99e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c306bdf151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c306d1d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c306ad3ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c306ac9d4b] /usr/sbin/mysqld(+0x6ca792)[0x55c306ad4792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c306afc6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c306afe65d] /usr/sbin/mysqld(+0x65e3f3)[0x55c306a683f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c306a67cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c306afbb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c306afe65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c306afe761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c306aff377] /usr/sbin/mysqld(+0x58fd84)[0x55c306999d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c306aa6b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c306abeb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c306abecd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c306abf5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c306abf677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c306aae56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c306ab007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c306b8bafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c306b8bbdd] pthread_create.c:0(start_thread)[0x7f7ebf996ea5] /lib64/libc.so.6(clone+0x6d)[0x7f7ebdd378cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f7e4000ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 63 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:00:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:00:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:00:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:00:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:00:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:00:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:00:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:00:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:00:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:00:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136352930 2020-01-31 22:00:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:00:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:00:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:00:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:00:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:00:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:00:47 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:00:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136352939; transaction id 5901129 2020-01-31 22:00:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:00:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:00:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:00:47 0 [Note] Starting crash recovery... 2020-01-31 22:00:47 0 [Note] Crash recovery finished. 2020-01-31 22:00:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:00:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:00:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:00:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000002' at position 346 2020-01-31 22:00:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000002' at position 346, relay log './dbsvc-528-relay-bin.000008' position: 649 2020-01-31 22:00:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:00:48 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000002' at position 346 2020-01-31 22:00:49 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:00:49 2020-01-31 22:01:13 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:01:13 13 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 22:01:13 13 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 22:01:16 15 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:01:16 15 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 22:01:34 27 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:01:34 27 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:01:35 29 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 22:01:35 29 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 22:01:41 32 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000008, 401138) 2020-01-31 22:02:03 43 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:02:03 43 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:02:44 63 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 22:02:44 63 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 22:03:01 70 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 22:03:01 70 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 22:04:45 131 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 22:04:45 131 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 22:05:24 145 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 22:05:24 145 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 22:05:40 159 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:05:40 159 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:05:40 160 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:05:40 160 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 22:05:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=502 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f11a8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f11b4ffccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55951f491f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55951ef283af] sigaction.c:0(__restore_rt)[0x7f12177ae630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55951ee56151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55951ef94545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55951ed4accf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55951ed40d4b] /usr/sbin/mysqld(+0x6ca792)[0x55951ed4b792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55951ed736da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55951ed7565d] /usr/sbin/mysqld(+0x65e3f3)[0x55951ecdf3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55951ecdecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55951ed72b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55951ed7565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55951ed75761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55951ed76377] /usr/sbin/mysqld(+0x58fd84)[0x55951ec10d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55951ed1db6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55951ed35b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55951ed35cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55951ed365e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55951ed36677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55951ed2556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55951ed2707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55951ee02afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55951ee02bdd] pthread_create.c:0(start_thread)[0x7f12177a6ea5] /lib64/libc.so.6(clone+0x6d)[0x7f1215b478cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f11a800feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 161 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:05:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:05:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:05:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:05:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:05:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:05:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:05:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:05:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:05:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:05:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136358730 2020-01-31 22:05:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:05:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:05:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:05:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:05:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:05:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:05:47 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:05:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136358739; transaction id 5901163 2020-01-31 22:05:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:05:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:05:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:05:47 0 [Note] Starting crash recovery... 2020-01-31 22:05:47 0 [Note] Crash recovery finished. 2020-01-31 22:05:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:05:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:05:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:05:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000002' at position 346 2020-01-31 22:05:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000002' at position 346, relay log './dbsvc-528-relay-bin.000010' position: 559 2020-01-31 22:05:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:05:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000002' at position 346 2020-01-31 22:05:47 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:05:47 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:05:47 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:05:47 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 22:05:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:05:48 2020-01-31 22:05:52 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:05:52 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:05:52 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:05:52 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 22:06:14 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:06:14 11 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 22:06:14 11 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 22:06:14 11 [Warning] wpcomm306.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:06:14 11 [Note] wpcomm306.wp_options: Table is fixed 2020-01-31 22:06:17 14 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:06:17 14 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 22:06:18 16 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 22:06:18 16 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 22:06:26 20 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:06:26 20 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:06:27 20 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:06:27 20 [Warning] Checking table: './wpms/wp_usermeta' 2020-01-31 22:06:30 23 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 22:06:30 23 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 22:06:33 25 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 22:06:33 25 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 22:06:42 29 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000009, 1041578) 2020-01-31 22:08:13 70 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 22:08:13 70 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 22:08:38 82 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 22:08:38 82 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 22:09:36 106 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 22:09:36 106 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 22:10:40 126 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:10:40 126 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:10:40 127 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:10:40 127 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 22:10:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f73f0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f74483b8cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560b7c810f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560b7c2a73af] sigaction.c:0(__restore_rt)[0x7f74655c2630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x560b7c1d5151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x560b7c313545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x560b7c0c9ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x560b7c0bfd4b] /usr/sbin/mysqld(+0x6ca792)[0x560b7c0ca792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x560b7c0f26da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560b7c0f465d] /usr/sbin/mysqld(+0x65e3f3)[0x560b7c05e3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x560b7c05dcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x560b7c0f1b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560b7c0f465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x560b7c0f4761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x560b7c0f5377] /usr/sbin/mysqld(+0x58fd84)[0x560b7bf8fd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x560b7c09cb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x560b7c0b4b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x560b7c0b4cd2] /usr/sbin/mysqld(+0x6b55e7)[0x560b7c0b55e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x560b7c0b5677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x560b7c0a456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x560b7c0a607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x560b7c181afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560b7c181bdd] pthread_create.c:0(start_thread)[0x7f74655baea5] /lib64/libc.so.6(clone+0x6d)[0x7f746395b8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f73f000ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 128 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:10:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:10:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:10:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:10:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:10:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:10:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:10:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:10:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:10:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:10:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136369169 2020-01-31 22:10:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:10:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:10:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:10:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:10:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:10:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:10:47 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:10:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136369178; transaction id 5901208 2020-01-31 22:10:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:10:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:10:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:10:47 0 [Note] Starting crash recovery... 2020-01-31 22:10:47 0 [Note] Crash recovery finished. 2020-01-31 22:10:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:10:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:10:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:10:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000002' at position 70133 2020-01-31 22:10:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000002' at position 70133, relay log './dbsvc-528-relay-bin.000012' position: 45719 2020-01-31 22:10:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:10:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000002' at position 70133 2020-01-31 22:10:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:10:48 2020-01-31 22:11:12 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:11:12 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:11:12 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:11:12 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 22:11:13 11 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 22:11:13 11 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 22:11:13 11 [Warning] wpcomm306.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:11:13 11 [Note] wpcomm306.wp_options: Table is fixed 2020-01-31 22:11:15 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:11:30 17 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 22:11:30 17 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 22:11:33 19 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 22:11:33 19 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 22:11:41 25 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000010, 468700) 2020-01-31 22:11:52 29 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:11:52 29 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:12:37 51 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:12:37 51 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 22:13:19 67 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:13:19 67 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:13:43 78 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 22:13:43 78 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 22:15:09 125 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:15:09 125 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:15:09 126 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:15:09 126 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 22:15:09 126 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 22:15:09 126 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 22:15:10 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff16c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff120101cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x564d579c2f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x564d574593af] sigaction.c:0(__restore_rt)[0x7ff18d3c5630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564d57387151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564d574c5545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564d5727bccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x564d57271d4b] /usr/sbin/mysqld(+0x6ca792)[0x564d5727c792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x564d572a46da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564d572a665d] /usr/sbin/mysqld(+0x65e3f3)[0x564d572103f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x564d5720fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x564d572a3b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564d572a665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x564d572a6761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x564d572a7377] /usr/sbin/mysqld(+0x58fd84)[0x564d57141d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x564d5724eb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564d57266b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564d57266cd2] /usr/sbin/mysqld(+0x6b55e7)[0x564d572675e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564d57267677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x564d5725656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x564d5725807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x564d57333afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x564d57333bdd] pthread_create.c:0(start_thread)[0x7ff18d3bdea5] /lib64/libc.so.6(clone+0x6d)[0x7ff18b75e8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff16c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 127 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:15:16 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:15:16 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:15:16 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:15:16 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:15:16 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:15:16 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:15:16 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:15:16 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:15:16 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:15:16 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136383306 2020-01-31 22:15:16 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:15:16 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:15:16 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:15:16 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:15:16 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:15:16 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:15:16 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:15:16 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136383315; transaction id 5901262 2020-01-31 22:15:16 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:15:16 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:15:16 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:15:16 0 [Note] Starting crash recovery... 2020-01-31 22:15:16 0 [Note] Crash recovery finished. 2020-01-31 22:15:16 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:15:16 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:15:16 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:15:16 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000002' at position 135257 2020-01-31 22:15:16 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000002' at position 135257, relay log './dbsvc-528-relay-bin.000014' position: 42943 2020-01-31 22:15:16 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:15:16 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000002' at position 135257 2020-01-31 22:15:18 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:15:18 2020-01-31 22:15:21 11 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 22:15:21 11 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 22:15:21 11 [Warning] wpstart.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:15:21 11 [Note] wpstart.wp_options: Table is fixed 2020-01-31 22:15:27 11 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 22:15:27 11 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 22:15:27 11 [Warning] wpsem.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:15:27 11 [Note] wpsem.wp_options: Table is fixed 2020-01-31 22:15:30 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:15:30 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:15:30 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:15:30 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 22:15:41 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:15:41 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:15:41 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 22:15:41 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 22:15:41 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:15:41 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 22:15:41 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 22:15:41 11 [Note] ojsdata.scheduled_tasks: Table is fixed 2020-01-31 22:15:42 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 22:15:42 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000002' at position 387995 2020-01-31 22:15:42 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 22:15:45 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:15:48 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000011, 855021) 2020-01-31 22:16:13 23 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 22:16:13 23 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 22:16:13 24 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 22:16:13 24 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 22:16:16 27 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:16:16 27 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:16:18 30 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 22:16:18 30 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 22:16:30 36 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 22:16:30 36 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 22:16:32 38 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 22:16:32 38 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 22:16:33 41 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:16:33 41 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:16:34 43 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 22:16:34 43 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 22:16:42 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000002' at position 387995 2020-01-31 22:18:13 99 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 22:18:13 99 [Warning] Checking table: './wpcomm305/comm305_options' 200131 22:18:37 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ffa00000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ffa5c078cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5632a330cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5632a2da33af] sigaction.c:0(__restore_rt)[0x7ffa76b82630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5632a2cd1151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5632a2e0f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5632a2bc5ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5632a2bbbd4b] /usr/sbin/mysqld(+0x6ca792)[0x5632a2bc6792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5632a2bee6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5632a2bf065d] /usr/sbin/mysqld(+0x65e3f3)[0x5632a2b5a3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5632a2b59cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5632a2bedb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5632a2bf065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5632a2bf0761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5632a2bf1377] /usr/sbin/mysqld(+0x58fd84)[0x5632a2a8bd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5632a2b98b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5632a2bb0b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5632a2bb0cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5632a2bb15e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5632a2bb1677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5632a2ba056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5632a2ba207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5632a2c7dafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5632a2c7dbdd] pthread_create.c:0(start_thread)[0x7ffa76b7aea5] /lib64/libc.so.6(clone+0x6d)[0x7ffa74f1b8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ffa0000feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 116 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:18:43 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:18:43 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:18:43 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:18:43 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:18:43 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:18:43 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:18:43 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:18:43 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:18:43 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:18:43 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136415406 2020-01-31 22:18:43 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:18:43 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:18:43 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:18:43 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:18:43 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:18:43 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:18:43 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:18:43 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136415415; transaction id 5901394 2020-01-31 22:18:43 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:18:43 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:18:43 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:18:43 0 [Note] Starting crash recovery... 2020-01-31 22:18:43 0 [Note] Crash recovery finished. 2020-01-31 22:18:43 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:18:43 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:18:43 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:18:43 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000003' at position 346 2020-01-31 22:18:43 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000003' at position 346, relay log './dbsvc-528-relay-bin.000018' position: 649 2020-01-31 22:18:43 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:18:43 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 22:18:45 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:18:45 2020-01-31 22:18:48 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000012, 1038156) 2020-01-31 22:19:11 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:19:12 15 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:19:12 15 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:19:12 16 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:19:12 16 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:19:16 18 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:19:16 18 [Warning] Checking table: './wpms/wp_usermeta' 2020-01-31 22:19:25 22 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 22:19:25 22 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 22:19:27 25 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 22:19:27 25 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 22:19:36 32 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 22:19:36 32 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 22:19:43 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000003' at position 346 2020-01-31 22:19:43 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:19:43 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:19:43 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 22:19:43 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 22:19:43 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:19:43 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 22:19:43 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 22:19:43 11 [Note] ojsdata.scheduled_tasks: Table is fixed 200131 22:20:20 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fcd20000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fcd7c05ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a94179cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a9412333af] sigaction.c:0(__restore_rt)[0x7fcd97264630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a941161151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a94129f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a941055ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a94104bd4b] /usr/sbin/mysqld(+0x6ca792)[0x55a941056792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a94107e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a94108065d] /usr/sbin/mysqld(+0x65e3f3)[0x55a940fea3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a940fe9cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a94107db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a94108065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a941080761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a941081377] /usr/sbin/mysqld(+0x58fd84)[0x55a940f1bd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a941028b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a941040b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a941040cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a9410415e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a941041677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a94103056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a94103207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a94110dafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a94110dbdd] pthread_create.c:0(start_thread)[0x7fcd9725cea5] /lib64/libc.so.6(clone+0x6d)[0x7fcd955fd8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fcd2000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 47 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:20:26 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:20:26 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:20:26 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:20:26 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:20:26 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:20:26 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:20:26 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:20:26 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:20:26 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:20:26 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136422148 2020-01-31 22:20:26 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:20:26 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:20:26 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:20:26 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:20:26 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:20:26 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:20:26 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:20:27 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136422157; transaction id 5901414 2020-01-31 22:20:27 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:20:27 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:20:27 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:20:27 0 [Note] Starting crash recovery... 2020-01-31 22:20:27 0 [Note] Crash recovery finished. 2020-01-31 22:20:27 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:20:27 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:20:27 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:20:27 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000004' at position 346 2020-01-31 22:20:27 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000004' at position 346, relay log './dbsvc-528-relay-bin.000021' position: 649 2020-01-31 22:20:27 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:20:27 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000004' at position 346 2020-01-31 22:20:28 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:20:28 2020-01-31 22:20:35 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:20:35 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:20:35 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:20:35 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 22:20:40 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:20:40 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:20:40 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 22:20:40 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 22:20:40 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:20:40 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 22:20:40 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 22:20:40 11 [Note] ojsdata.scheduled_tasks: Table is fixed 2020-01-31 22:20:40 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 22:20:40 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000004' at position 25313 2020-01-31 22:20:40 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 22:20:46 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000013, 296902) 2020-01-31 22:20:56 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:20:58 16 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:20:58 16 [Warning] Checking table: './wpwsutv/wp_usermeta' 200131 22:21:10 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff950000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff9bc0fccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ed44b0ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ed445a53af] sigaction.c:0(__restore_rt)[0x7ff9d63e0630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55ed444d3151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55ed44611545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55ed443c7ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55ed443bdd4b] /usr/sbin/mysqld(+0x6ca792)[0x55ed443c8792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55ed443f06da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ed443f265d] /usr/sbin/mysqld(+0x65e3f3)[0x55ed4435c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55ed4435bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55ed443efb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ed443f265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55ed443f2761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55ed443f3377] /usr/sbin/mysqld(+0x58fd84)[0x55ed4428dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55ed4439ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55ed443b2b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55ed443b2cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55ed443b35e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55ed443b3677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55ed443a256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55ed443a407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55ed4447fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55ed4447fbdd] pthread_create.c:0(start_thread)[0x7ff9d63d8ea5] /lib64/libc.so.6(clone+0x6d)[0x7ff9d47798cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff95000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 24 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:21:16 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:21:16 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:21:16 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:21:16 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:21:16 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:21:16 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:21:16 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:21:16 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:21:16 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:21:16 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136429103 2020-01-31 22:21:16 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:21:16 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:21:16 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:21:16 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:21:16 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:21:16 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:21:16 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:21:16 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136429112; transaction id 5901441 2020-01-31 22:21:16 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:21:16 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:21:16 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:21:16 0 [Note] Starting crash recovery... 2020-01-31 22:21:16 0 [Note] Crash recovery finished. 2020-01-31 22:21:16 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:21:16 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:21:16 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:21:16 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000004' at position 25313 2020-01-31 22:21:16 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000004' at position 25313, relay log './dbsvc-528-relay-bin.000023' position: 16765 2020-01-31 22:21:16 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:21:16 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000004' at position 25313 2020-01-31 22:21:17 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:21:17 2020-01-31 22:21:41 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:21:55 19 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:21:55 19 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:21:59 22 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 22:21:59 22 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 22:22:06 25 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 22:22:06 25 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 22:22:10 27 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000014, 26768) 2020-01-31 22:22:40 38 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:22:40 38 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:23:01 52 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:23:01 52 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:23:43 71 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 22:23:43 71 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 22:25:24 110 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 22:25:24 110 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 22:25:40 119 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:25:40 119 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:25:40 120 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:25:40 120 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 22:25:41 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2cb40009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2cd9305cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5564cc2a5f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5564cbd3c3af] sigaction.c:0(__restore_rt)[0x7f2d3a450630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5564cbc6a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5564cbda8545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5564cbb5eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5564cbb54d4b] /usr/sbin/mysqld(+0x6ca792)[0x5564cbb5f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5564cbb876da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5564cbb8965d] /usr/sbin/mysqld(+0x65e3f3)[0x5564cbaf33f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5564cbaf2cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5564cbb86b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5564cbb8965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5564cbb89761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5564cbb8a377] /usr/sbin/mysqld(+0x58fd84)[0x5564cba24d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5564cbb31b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5564cbb49b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5564cbb49cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5564cbb4a5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5564cbb4a677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5564cbb3956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5564cbb3b07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5564cbc16afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5564cbc16bdd] pthread_create.c:0(start_thread)[0x7f2d3a448ea5] /lib64/libc.so.6(clone+0x6d)[0x7f2d387e98cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f2cb400fe20): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 121 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:25:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:25:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:25:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:25:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:25:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:25:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:25:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:25:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:25:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:25:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136434723 2020-01-31 22:25:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:25:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:25:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:25:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:25:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:25:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:25:47 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:25:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136434732; transaction id 5901452 2020-01-31 22:25:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:25:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:25:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:25:47 0 [Note] Starting crash recovery... 2020-01-31 22:25:47 0 [Note] Crash recovery finished. 2020-01-31 22:25:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:25:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:25:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:25:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000005' at position 346 2020-01-31 22:25:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000005' at position 346, relay log './dbsvc-528-relay-bin.000026' position: 649 2020-01-31 22:25:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:25:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000005' at position 346 2020-01-31 22:25:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:25:48 2020-01-31 22:26:17 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:26:17 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:26:17 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:26:17 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 22:26:17 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:26:18 14 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 22:26:18 14 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 22:26:30 19 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 22:26:30 19 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 22:26:32 22 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 22:26:32 22 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 22:26:36 24 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:26:36 24 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:26:37 25 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 22:26:37 25 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 22:26:42 29 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000015, 442893) 2020-01-31 22:26:58 34 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 22:26:58 34 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 22:27:06 38 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:27:06 38 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 22:27:11 41 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 22:27:11 41 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 22:27:31 51 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 22:27:31 51 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 22:27:31 53 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 22:27:31 53 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 22:27:52 63 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:27:52 63 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:28:38 116 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 22:28:38 116 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 22:29:24 138 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 22:29:24 138 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 22:29:36 145 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 22:29:36 145 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 22:30:40 167 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:30:40 167 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:30:40 168 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:30:40 168 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 22:30:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2ebc000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2e70148cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5626da27ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5626d9d153af] sigaction.c:0(__restore_rt)[0x7f2edca13630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5626d9c43151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5626d9d81545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5626d9b37ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5626d9b2dd4b] /usr/sbin/mysqld(+0x6ca792)[0x5626d9b38792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5626d9b606da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5626d9b6265d] /usr/sbin/mysqld(+0x65e3f3)[0x5626d9acc3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5626d9acbcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5626d9b5fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5626d9b6265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5626d9b62761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5626d9b63377] /usr/sbin/mysqld(+0x58fd84)[0x5626d99fdd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5626d9b0ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5626d9b22b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5626d9b22cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5626d9b235e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5626d9b23677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5626d9b1256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5626d9b1407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5626d9befafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5626d9befbdd] pthread_create.c:0(start_thread)[0x7f2edca0bea5] /lib64/libc.so.6(clone+0x6d)[0x7f2edadac8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f2ebc00ff10): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 169 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:30:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:30:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:30:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:30:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:30:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:30:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:30:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:30:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:30:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:30:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136446411 2020-01-31 22:30:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:30:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:30:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:30:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:30:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:30:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:30:47 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:30:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136446420; transaction id 5901496 2020-01-31 22:30:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:30:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:30:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:30:47 0 [Note] Starting crash recovery... 2020-01-31 22:30:47 0 [Note] Crash recovery finished. 2020-01-31 22:30:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:30:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:30:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:30:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000005' at position 27493 2020-01-31 22:30:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000005' at position 27493, relay log './dbsvc-528-relay-bin.000028' position: 18176 2020-01-31 22:30:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:30:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000005' at position 27493 2020-01-31 22:30:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:30:48 2020-01-31 22:31:02 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:31:02 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:31:02 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:31:02 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 22:31:13 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:31:13 14 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 22:31:13 14 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 22:31:33 21 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 22:31:33 21 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 22:31:35 22 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 22:31:35 22 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 22:31:41 25 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000016, 912770) 2020-01-31 22:32:17 34 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:32:17 34 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:32:29 41 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:32:29 41 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:32:29 42 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:32:29 42 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 22:32:29 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7efee0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7eff3c0c3cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x559c24dd9f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x559c248703af] sigaction.c:0(__restore_rt)[0x7eff56803630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x559c2479e151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x559c248dc545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x559c24692ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x559c24688d4b] /usr/sbin/mysqld(+0x6ca792)[0x559c24693792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x559c246bb6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559c246bd65d] /usr/sbin/mysqld(+0x65e3f3)[0x559c246273f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x559c24626cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x559c246bab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559c246bd65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x559c246bd761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x559c246be377] /usr/sbin/mysqld(+0x58fd84)[0x559c24558d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x559c24665b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x559c2467db16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x559c2467dcd2] /usr/sbin/mysqld(+0x6b55e7)[0x559c2467e5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x559c2467e677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x559c2466d56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x559c2466f07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x559c2474aafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x559c2474abdd] pthread_create.c:0(start_thread)[0x7eff567fbea5] /lib64/libc.so.6(clone+0x6d)[0x7eff54b9c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7efee000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 43 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:32:35 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:32:35 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:32:35 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:32:35 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:32:35 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:32:35 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:32:35 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:32:35 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:32:35 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:32:35 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136458259 2020-01-31 22:32:35 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:32:35 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:32:35 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:32:35 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:32:35 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:32:35 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:32:35 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:32:35 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136458268; transaction id 5901536 2020-01-31 22:32:35 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:32:35 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:32:36 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:32:36 0 [Note] Starting crash recovery... 2020-01-31 22:32:36 0 [Note] Crash recovery finished. 2020-01-31 22:32:36 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:32:36 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:32:36 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:32:36 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000005' at position 51695 2020-01-31 22:32:36 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000005' at position 51695, relay log './dbsvc-528-relay-bin.000030' position: 16247 2020-01-31 22:32:36 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:32:36 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000005' at position 51695 2020-01-31 22:32:37 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:32:37 2020-01-31 22:32:39 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:32:39 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:32:39 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:32:39 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 22:32:59 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:32:59 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:32:59 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:32:59 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 22:33:02 11 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 22:33:02 11 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 22:33:02 11 [Warning] wpcomm305.comm305_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:33:02 11 [Note] wpcomm305.comm305_options: Table is fixed 2020-01-31 22:33:03 20 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:33:08 22 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 22:33:08 22 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 22:33:15 26 [ERROR] mysqld: Table './wpcomm305/comm305_posts' is marked as crashed and should be repaired 2020-01-31 22:33:15 26 [Warning] Checking table: './wpcomm305/comm305_posts' 2020-01-31 22:33:30 31 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000017, 132576) 2020-01-31 22:33:43 35 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 22:33:43 35 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 22:33:50 38 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 22:33:50 38 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 22:34:28 52 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:34:28 52 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 22:35:16 73 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:35:16 73 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:35:24 79 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 22:35:24 79 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 22:35:41 91 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:35:41 91 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:35:41 92 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:35:41 92 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 22:35:41 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0504000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f055c1eccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56041f478f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56041ef0f3af] sigaction.c:0(__restore_rt)[0x7f05796d4630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56041ee3d151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56041ef7b545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56041ed31ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56041ed27d4b] /usr/sbin/mysqld(+0x6ca792)[0x56041ed32792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56041ed5a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56041ed5c65d] /usr/sbin/mysqld(+0x65e3f3)[0x56041ecc63f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56041ecc5cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56041ed59b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56041ed5c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56041ed5c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56041ed5d377] /usr/sbin/mysqld(+0x58fd84)[0x56041ebf7d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56041ed04b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56041ed1cb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56041ed1ccd2] /usr/sbin/mysqld(+0x6b55e7)[0x56041ed1d5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56041ed1d677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56041ed0c56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56041ed0e07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56041ede9afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56041ede9bdd] pthread_create.c:0(start_thread)[0x7f05796ccea5] /lib64/libc.so.6(clone+0x6d)[0x7f0577a6d8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f050400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 93 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:35:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:35:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:35:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:35:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:35:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:35:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:35:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:35:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:35:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:35:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136476070 2020-01-31 22:35:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:35:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:35:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:35:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:35:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:35:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:35:47 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:35:48 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136476079; transaction id 5901605 2020-01-31 22:35:48 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:35:48 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:35:48 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:35:48 0 [Note] Starting crash recovery... 2020-01-31 22:35:48 0 [Note] Crash recovery finished. 2020-01-31 22:35:48 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:35:48 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:35:48 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:35:48 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000005' at position 133116 2020-01-31 22:35:48 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000005' at position 133116, relay log './dbsvc-528-relay-bin.000032' position: 53225 2020-01-31 22:35:48 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:35:48 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000005' at position 133116 2020-01-31 22:35:49 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:35:49 2020-01-31 22:35:49 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:35:49 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:35:49 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:35:49 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 22:36:12 11 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:36:12 11 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 22:36:12 11 [Warning] wpwsutv.wp_usermeta: 1 client is using or hasn't closed the table properly 2020-01-31 22:36:12 11 [Note] wpwsutv.wp_usermeta: Table is fixed 2020-01-31 22:36:14 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:36:18 16 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 22:36:18 16 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 22:36:26 20 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:36:26 20 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:36:31 23 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 22:36:31 23 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 22:36:32 25 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 22:36:32 25 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 22:36:40 32 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:36:40 32 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:36:40 32 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:36:40 32 [Warning] Checking table: './wpms/wp_usermeta' 2020-01-31 22:36:42 34 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000018, 399998) 2020-01-31 22:38:14 93 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 22:38:14 93 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 22:38:40 105 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 22:38:40 105 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 22:39:36 134 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 22:39:36 134 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 22:39:56 143 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:39:56 143 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:39:56 144 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:39:56 144 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 22:39:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff3d00009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff3e50e8cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5620101a0f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56200fc373af] sigaction.c:0(__restore_rt)[0x7ff4462b9630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56200fb65151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56200fca3545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56200fa59ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56200fa4fd4b] /usr/sbin/mysqld(+0x6ca792)[0x56200fa5a792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56200fa826da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56200fa8465d] /usr/sbin/mysqld(+0x65e3f3)[0x56200f9ee3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56200f9edcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56200fa81b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56200fa8465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56200fa84761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56200fa85377] /usr/sbin/mysqld(+0x58fd84)[0x56200f91fd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56200fa2cb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56200fa44b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56200fa44cd2] /usr/sbin/mysqld(+0x6b55e7)[0x56200fa455e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56200fa45677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56200fa3456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56200fa3607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56200fb11afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56200fb11bdd] pthread_create.c:0(start_thread)[0x7ff4462b1ea5] /lib64/libc.so.6(clone+0x6d)[0x7ff4446528cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff3d000fdd0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 145 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:40:03 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:40:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:40:03 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:40:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:40:03 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:40:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:40:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:40:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:40:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:40:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136492274 2020-01-31 22:40:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:40:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:40:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:40:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:40:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:40:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:40:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136492283; transaction id 5901665 2020-01-31 22:40:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:40:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:40:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:40:03 0 [Note] Starting crash recovery... 2020-01-31 22:40:03 0 [Note] Crash recovery finished. 2020-01-31 22:40:04 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:40:04 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:40:04 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:40:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000005' at position 158904 2020-01-31 22:40:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000005' at position 158904, relay log './dbsvc-528-relay-bin.000034' position: 17354 2020-01-31 22:40:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:40:04 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 22:40:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:40:05 2020-01-31 22:40:06 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000019, 809711) 2020-01-31 22:40:29 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:40:32 14 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:40:32 14 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:40:32 14 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:40:32 14 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 22:40:32 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:40:32 15 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:40:32 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:40:32 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 22:40:32 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2e18000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2e200bacf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560a35b28f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560a355bf3af] sigaction.c:0(__restore_rt)[0x7f2e8b2f6630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x560a354ed151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x560a3562b545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x560a353e1ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x560a353d7d4b] /usr/sbin/mysqld(+0x6ca792)[0x560a353e2792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x560a3540a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560a3540c65d] /usr/sbin/mysqld(+0x65e3f3)[0x560a353763f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x560a35375cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x560a35409b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560a3540c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x560a3540c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x560a3540d377] /usr/sbin/mysqld(+0x58fd84)[0x560a352a7d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x560a353b4b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x560a353ccb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x560a353cccd2] /usr/sbin/mysqld(+0x6b55e7)[0x560a353cd5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x560a353cd677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x560a353bc56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x560a353be07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x560a35499afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560a35499bdd] pthread_create.c:0(start_thread)[0x7f2e8b2eeea5] /lib64/libc.so.6(clone+0x6d)[0x7f2e8968f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f2e1800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:40:38 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:40:38 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:40:38 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:40:38 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:40:38 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:40:38 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:40:38 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:40:38 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:40:38 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:40:38 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136498666 2020-01-31 22:40:38 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:40:38 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:40:38 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:40:38 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:40:38 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:40:38 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:40:38 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:40:38 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136498675; transaction id 5901679 2020-01-31 22:40:38 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:40:38 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:40:38 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:40:38 0 [Note] Starting crash recovery... 2020-01-31 22:40:38 0 [Note] Crash recovery finished. 2020-01-31 22:40:39 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:40:39 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:40:39 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:40:39 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000005' at position 158904 2020-01-31 22:40:39 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000005' at position 158904, relay log './dbsvc-528-relay-bin.000035' position: 4 2020-01-31 22:40:39 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:40:39 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000005' at position 158904 2020-01-31 22:40:39 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:40:39 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:40:39 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 22:40:39 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 22:40:39 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:40:39 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 22:40:39 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 22:40:39 11 [Note] ojsdata.scheduled_tasks: Table is fixed 2020-01-31 22:40:40 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:40:40 2020-01-31 22:40:40 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:40:40 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:40:40 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:40:40 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 22:40:41 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 22:40:41 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000006' at position 13586 2020-01-31 22:40:41 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 22:40:47 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000020, 2529) 2020-01-31 22:41:04 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:41:05 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:41:05 14 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:41:05 14 [ERROR] mysqld: Table './ksarn/wp_postmeta' is marked as crashed and should be repaired 2020-01-31 22:41:05 14 [Warning] Checking table: './ksarn/wp_postmeta' 2020-01-31 22:41:06 15 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:41:06 15 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 22:41:09 18 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 22:41:09 18 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 22:41:09 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff850000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff864154cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b99685df2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55b9962f43af] sigaction.c:0(__restore_rt)[0x7ff8cd3dc630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55b996222151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55b996360545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55b996116ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55b99610cd4b] /usr/sbin/mysqld(+0x6ca792)[0x55b996117792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55b99613f6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b99614165d] /usr/sbin/mysqld(+0x65e3f3)[0x55b9960ab3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55b9960aacb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55b99613eb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b99614165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55b996141761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55b996142377] /usr/sbin/mysqld(+0x58fd84)[0x55b995fdcd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55b9960e9b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55b996101b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55b996101cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55b9961025e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55b996102677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55b9960f156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55b9960f307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55b9961ceafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55b9961cebdd] pthread_create.c:0(start_thread)[0x7ff8cd3d4ea5] /lib64/libc.so.6(clone+0x6d)[0x7ff8cb7758cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff85000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:41:15 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:41:15 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:41:15 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:41:15 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:41:15 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:41:15 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:41:15 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:41:15 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:41:15 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:41:15 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136505786 2020-01-31 22:41:15 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:41:15 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:41:15 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:41:15 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:41:15 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:41:15 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:41:15 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:41:15 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136505795; transaction id 5901704 2020-01-31 22:41:15 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:41:15 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:41:15 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:41:15 0 [Note] Starting crash recovery... 2020-01-31 22:41:15 0 [Note] Crash recovery finished. 2020-01-31 22:41:15 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:41:15 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:41:15 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:41:15 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000006' at position 13586 2020-01-31 22:41:15 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000006' at position 13586, relay log './dbsvc-528-relay-bin.000038' position: 9087 2020-01-31 22:41:15 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:41:15 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000006' at position 13586 2020-01-31 22:41:15 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:41:15 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:41:15 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:41:15 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 22:41:16 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:41:16 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:41:16 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:41:16 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 22:41:16 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:41:16 2020-01-31 22:41:20 11 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity' is marked as crashed and should be repaired 2020-01-31 22:41:20 11 [Warning] Checking table: './ksarn/wp_learndash_user_activity' 2020-01-31 22:41:20 11 [Warning] ksarn.wp_learndash_user_activity: 1 client is using or hasn't closed the table properly 2020-01-31 22:41:20 11 [Note] ksarn.wp_learndash_user_activity: Table is fixed 2020-01-31 22:41:20 11 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity_meta' is marked as crashed and should be repaired 2020-01-31 22:41:20 11 [Warning] Checking table: './ksarn/wp_learndash_user_activity_meta' 2020-01-31 22:41:20 11 [Warning] ksarn.wp_learndash_user_activity_meta: 1 client is using or hasn't closed the table properly 2020-01-31 22:41:20 11 [Note] ksarn.wp_learndash_user_activity_meta: Table is fixed 2020-01-31 22:41:45 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:41:46 14 [ERROR] mysqld: Table './ksarn/wp_bp_activity' is marked as crashed and should be repaired 2020-01-31 22:41:46 14 [Warning] Checking table: './ksarn/wp_bp_activity' 2020-01-31 22:42:09 23 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000021, 119663) 2020-01-31 22:42:15 30 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 22:42:15 30 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 22:42:15 31 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 22:42:15 31 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 22:42:25 38 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 22:42:25 38 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 22:42:30 40 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 22:42:30 40 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 22:42:40 44 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:42:40 44 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:42:54 55 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 22:42:54 55 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 22:43:01 60 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 22:43:01 60 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 22:45:19 136 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 22:45:19 136 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 22:45:40 152 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:45:40 152 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:45:40 153 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:45:40 153 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 22:45:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=502 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fbca4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fbc580a7cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5579b12ebf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5579b0d823af] sigaction.c:0(__restore_rt)[0x7fbcc28c4630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5579b0cb0151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5579b0dee545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5579b0ba4ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5579b0b9ad4b] /usr/sbin/mysqld(+0x6ca792)[0x5579b0ba5792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5579b0bcd6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5579b0bcf65d] /usr/sbin/mysqld(+0x65e3f3)[0x5579b0b393f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5579b0b38cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5579b0bccb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5579b0bcf65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5579b0bcf761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5579b0bd0377] /usr/sbin/mysqld(+0x58fd84)[0x5579b0a6ad84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5579b0b77b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5579b0b8fb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5579b0b8fcd2] /usr/sbin/mysqld(+0x6b55e7)[0x5579b0b905e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5579b0b90677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5579b0b7f56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5579b0b8107c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5579b0c5cafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5579b0c5cbdd] pthread_create.c:0(start_thread)[0x7fbcc28bcea5] /lib64/libc.so.6(clone+0x6d)[0x7fbcc0c5d8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fbca400feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 154 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:45:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:45:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:45:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:45:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:45:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:45:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:45:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:45:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:45:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:45:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136533267 2020-01-31 22:45:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:45:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:45:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:45:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:45:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:45:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:45:47 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:45:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136533276; transaction id 5901836 2020-01-31 22:45:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:45:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:45:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:45:47 0 [Note] Starting crash recovery... 2020-01-31 22:45:47 0 [Note] Crash recovery finished. 2020-01-31 22:45:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:45:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:45:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:45:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000007' at position 376865 2020-01-31 22:45:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000007' at position 376865, relay log './dbsvc-528-relay-bin.000041' position: 245050 2020-01-31 22:45:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:45:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000007' at position 376865 2020-01-31 22:45:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:45:48 2020-01-31 22:45:51 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:45:51 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:45:51 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:45:51 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 22:46:15 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:46:16 13 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:46:16 13 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:46:17 14 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:46:17 14 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 22:46:18 16 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 22:46:18 16 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 22:46:31 23 [ERROR] mysqld: Table './openrecords/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 22:46:31 23 [Warning] Checking table: './openrecords/wp_gf_form_view' 2020-01-31 22:46:32 25 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:46:32 25 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 22:46:32 26 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 22:46:32 26 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 22:46:34 25 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity' is marked as crashed and should be repaired 2020-01-31 22:46:34 25 [Warning] Checking table: './ksarn/wp_learndash_user_activity' 2020-01-31 22:46:34 25 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity_meta' is marked as crashed and should be repaired 2020-01-31 22:46:34 25 [Warning] Checking table: './ksarn/wp_learndash_user_activity_meta' 2020-01-31 22:46:35 31 [ERROR] mysqld: Table './ksarn/wp_bp_activity' is marked as crashed and should be repaired 2020-01-31 22:46:35 31 [Warning] Checking table: './ksarn/wp_bp_activity' 2020-01-31 22:46:35 32 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:46:35 32 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:46:35 33 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:46:35 33 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 22:46:35 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f61bc000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f6210078cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c73beacf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c73b9433af] sigaction.c:0(__restore_rt)[0x7f622a77b630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c73b871151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c73b9af545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c73b765ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c73b75bd4b] /usr/sbin/mysqld(+0x6ca792)[0x55c73b766792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c73b78e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c73b79065d] /usr/sbin/mysqld(+0x65e3f3)[0x55c73b6fa3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c73b6f9cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c73b78db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c73b79065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c73b790761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c73b791377] /usr/sbin/mysqld(+0x58fd84)[0x55c73b62bd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c73b738b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c73b750b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c73b750cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c73b7515e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c73b751677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c73b74056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c73b74207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c73b81dafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c73b81dbdd] pthread_create.c:0(start_thread)[0x7f622a773ea5] /lib64/libc.so.6(clone+0x6d)[0x7f6228b148cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f61bc00fe80): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 34 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:46:42 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:46:42 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:46:42 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:46:42 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:46:42 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:46:42 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:46:42 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:46:42 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:46:42 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:46:42 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136548074 2020-01-31 22:46:42 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:46:42 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:46:42 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:46:42 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:46:42 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:46:42 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:46:42 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:46:42 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136548083; transaction id 5901866 2020-01-31 22:46:42 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:46:42 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:46:42 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:46:42 0 [Note] Starting crash recovery... 2020-01-31 22:46:42 0 [Note] Crash recovery finished. 2020-01-31 22:46:42 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:46:42 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:46:42 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:46:42 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000007' at position 471602 2020-01-31 22:46:42 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000007' at position 471602, relay log './dbsvc-528-relay-bin.000043' position: 62069 2020-01-31 22:46:42 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:46:42 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000007' at position 471602 2020-01-31 22:46:42 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:46:42 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:46:42 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:46:42 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 22:46:43 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:46:43 2020-01-31 22:46:43 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:46:43 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:46:43 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:46:43 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 22:47:10 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:47:21 19 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:47:21 19 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 22:47:28 25 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:47:28 25 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:47:41 36 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000022, 2295716) 2020-01-31 22:48:03 56 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity' is marked as crashed and should be repaired 2020-01-31 22:48:03 56 [Warning] Checking table: './ksarn/wp_learndash_user_activity' 2020-01-31 22:48:03 56 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity_meta' is marked as crashed and should be repaired 2020-01-31 22:48:03 56 [Warning] Checking table: './ksarn/wp_learndash_user_activity_meta' 2020-01-31 22:48:07 58 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 22:48:07 58 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 22:48:15 62 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 22:48:15 62 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 22:48:39 73 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 22:48:39 73 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 22:48:59 83 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:48:59 83 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:48:59 84 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:48:59 84 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 22:48:59 84 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 22:48:59 84 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 22:49:00 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f7cf0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f7cdc904cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c6cf054f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c6ceaeb3af] sigaction.c:0(__restore_rt)[0x7f7cf9b1e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c6cea19151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c6ceb57545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c6ce90dccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c6ce903d4b] /usr/sbin/mysqld(+0x6ca792)[0x55c6ce90e792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c6ce9366da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c6ce93865d] /usr/sbin/mysqld(+0x65e3f3)[0x55c6ce8a23f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c6ce8a1cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c6ce935b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c6ce93865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c6ce938761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c6ce939377] /usr/sbin/mysqld(+0x58fd84)[0x55c6ce7d3d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c6ce8e0b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c6ce8f8b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c6ce8f8cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c6ce8f95e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c6ce8f9677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c6ce8e856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c6ce8ea07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c6ce9c5afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c6ce9c5bdd] pthread_create.c:0(start_thread)[0x7f7cf9b16ea5] /lib64/libc.so.6(clone+0x6d)[0x7f7cf7eb78cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f7cf000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 85 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:49:06 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:49:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:49:06 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:49:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:49:06 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:49:06 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:49:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:49:06 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:49:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:49:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136565277 2020-01-31 22:49:07 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:49:07 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:49:07 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:49:07 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:49:07 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:49:07 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:49:07 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:49:07 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136565286; transaction id 5901936 2020-01-31 22:49:07 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:49:07 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:49:07 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:49:07 0 [Note] Starting crash recovery... 2020-01-31 22:49:07 0 [Note] Crash recovery finished. 2020-01-31 22:49:07 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:49:07 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:49:07 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:49:07 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000007' at position 668166 2020-01-31 22:49:07 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000007' at position 668166, relay log './dbsvc-528-relay-bin.000045' position: 128005 2020-01-31 22:49:07 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:49:07 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000007' at position 668166 2020-01-31 22:49:07 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:49:07 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:49:07 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:49:07 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 22:49:09 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:49:09 2020-01-31 22:49:28 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:49:28 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:49:28 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:49:28 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 22:49:33 11 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 22:49:33 11 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 22:49:33 11 [Warning] wpstart.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:49:33 11 [Note] wpstart.wp_options: Table is fixed 2020-01-31 22:49:36 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:49:37 13 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 22:49:37 13 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 22:50:01 20 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000024, 1285175) 2020-01-31 22:50:03 22 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:50:03 22 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 22:50:12 27 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:50:12 27 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:50:13 28 [ERROR] mysqld: Table './ksarn/wp_bp_activity' is marked as crashed and should be repaired 2020-01-31 22:50:13 28 [Warning] Checking table: './ksarn/wp_bp_activity' 2020-01-31 22:50:13 29 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:50:13 29 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 22:50:40 43 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:50:40 43 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:50:40 44 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:50:40 44 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 22:50:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fbd48000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fbd4c110cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a7bc175f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a7bbc0c3af] sigaction.c:0(__restore_rt)[0x7fbd6753a630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a7bbb3a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a7bbc78545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a7bba2eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a7bba24d4b] /usr/sbin/mysqld(+0x6ca792)[0x55a7bba2f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a7bba576da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a7bba5965d] /usr/sbin/mysqld(+0x65e3f3)[0x55a7bb9c33f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a7bb9c2cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a7bba56b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a7bba5965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a7bba59761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a7bba5a377] /usr/sbin/mysqld(+0x58fd84)[0x55a7bb8f4d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a7bba01b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a7bba19b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a7bba19cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a7bba1a5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a7bba1a677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a7bba0956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a7bba0b07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a7bbae6afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a7bbae6bdd] pthread_create.c:0(start_thread)[0x7fbd67532ea5] /lib64/libc.so.6(clone+0x6d)[0x7fbd658d38cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fbd4800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 45 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:50:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:50:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:50:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:50:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:50:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:50:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:50:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:50:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:50:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:50:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136599687 2020-01-31 22:50:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:50:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:50:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:50:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:50:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:50:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:50:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136599696; transaction id 5902075 2020-01-31 22:50:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:50:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:50:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:50:47 0 [Note] Starting crash recovery... 2020-01-31 22:50:47 0 [Note] Crash recovery finished. 2020-01-31 22:50:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:50:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:50:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:50:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000007' at position 1078841 2020-01-31 22:50:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000007' at position 1078841, relay log './dbsvc-528-relay-bin.000047' position: 268128 2020-01-31 22:50:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:50:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000007' at position 1078841 2020-01-31 22:50:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:50:48 2020-01-31 22:50:56 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:50:56 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:50:56 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:50:56 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 22:50:57 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:50:57 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:50:57 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 22:50:57 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 22:50:57 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:50:57 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 22:50:57 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 22:50:57 11 [Note] ojsdata.scheduled_tasks: Table is fixed 2020-01-31 22:50:58 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 22:50:58 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000007' at position 1126975 2020-01-31 22:50:58 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 22:51:04 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000025, 388944) 2020-01-31 22:51:16 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:51:18 16 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:51:18 16 [Warning] Checking table: './wpwsutv/wp_options' 200131 22:51:19 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=502 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2078000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f20dca43cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x555c73322f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x555c72db93af] sigaction.c:0(__restore_rt)[0x7f20f7c5f630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x555c72ce7151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x555c72e25545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x555c72bdbccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x555c72bd1d4b] /usr/sbin/mysqld(+0x6ca792)[0x555c72bdc792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x555c72c046da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x555c72c0665d] /usr/sbin/mysqld(+0x65e3f3)[0x555c72b703f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x555c72b6fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x555c72c03b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x555c72c0665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x555c72c06761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x555c72c07377] /usr/sbin/mysqld(+0x58fd84)[0x555c72aa1d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x555c72baeb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x555c72bc6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x555c72bc6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x555c72bc75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x555c72bc7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x555c72bb656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x555c72bb807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x555c72c93afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x555c72c93bdd] pthread_create.c:0(start_thread)[0x7f20f7c57ea5] /lib64/libc.so.6(clone+0x6d)[0x7f20f5ff88cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f207800feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 21 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:51:25 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:51:25 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:51:25 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:51:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:51:25 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:51:25 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:51:25 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:51:25 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:51:25 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:51:25 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136607170 2020-01-31 22:51:25 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:51:25 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:51:25 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:51:25 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:51:25 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:51:25 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:51:25 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:51:25 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136607179; transaction id 5902098 2020-01-31 22:51:25 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:51:25 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:51:25 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:51:25 0 [Note] Starting crash recovery... 2020-01-31 22:51:25 0 [Note] Crash recovery finished. 2020-01-31 22:51:25 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:51:25 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:51:25 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:51:25 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000007' at position 1126975 2020-01-31 22:51:25 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000007' at position 1126975, relay log './dbsvc-528-relay-bin.000049' position: 31774 2020-01-31 22:51:25 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:51:25 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000007' at position 1126975 2020-01-31 22:51:26 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:51:26 2020-01-31 22:51:30 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:51:30 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:51:30 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:51:30 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 22:51:31 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:51:31 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:51:31 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 22:51:31 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 22:51:31 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:51:31 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 22:51:31 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 22:51:31 11 [Note] ojsdata.scheduled_tasks: Table is fixed 2020-01-31 22:51:31 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 22:51:31 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000008' at position 2542 2020-01-31 22:51:31 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 22:51:37 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000026, 90163) 2020-01-31 22:51:51 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:51:55 14 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 22:51:55 14 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 22:51:56 17 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:51:56 17 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:51:59 21 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 22:51:59 21 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 22:52:06 33 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity' is marked as crashed and should be repaired 2020-01-31 22:52:06 33 [Warning] Checking table: './ksarn/wp_learndash_user_activity' 2020-01-31 22:52:06 33 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity_meta' is marked as crashed and should be repaired 2020-01-31 22:52:06 33 [Warning] Checking table: './ksarn/wp_learndash_user_activity_meta' 2020-01-31 22:52:15 45 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 22:52:15 45 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 22:52:31 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000008' at position 2542 2020-01-31 22:53:06 74 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 22:53:06 74 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 22:53:16 78 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 22:53:16 78 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 22:54:45 140 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 22:54:45 140 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 22:55:24 157 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 22:55:24 157 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 22:55:25 159 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:55:25 159 [Warning] Checking table: './wpms/wp_options' 200131 22:55:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=8 max_threads=502 thread_count=17 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x56382caf0788 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7feef0095cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563827f2ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5638279c63af] sigaction.c:0(__restore_rt)[0x7fef0db80630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5638278f4151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x563827a32545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5638277e8ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5638277ded4b] /usr/sbin/mysqld(+0x6ca792)[0x5638277e9792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5638278116da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56382781365d] /usr/sbin/mysqld(+0x65e3f3)[0x56382777d3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56382777ccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x563827810b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56382781365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x563827813761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x563827814377] /usr/sbin/mysqld(+0x58fd84)[0x5638276aed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5638277bbb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5638277d3b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5638277d3cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5638277d45e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5638277d4677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5638277c356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5638277c507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5638278a0afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5638278a0bdd] pthread_create.c:0(start_thread)[0x7fef0db78ea5] /lib64/libc.so.6(clone+0x6d)[0x7fef0bf198cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x56382cafee60): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 182 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:55:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:55:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:55:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:55:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:55:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:55:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:55:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:55:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:55:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:55:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136616690 2020-01-31 22:55:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:55:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:55:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:55:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:55:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:55:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:55:47 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:55:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136616699; transaction id 5902141 2020-01-31 22:55:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:55:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:55:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:55:47 0 [Note] Starting crash recovery... 2020-01-31 22:55:47 0 [Note] Crash recovery finished. 2020-01-31 22:55:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:55:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:55:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:55:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000009' at position 346 2020-01-31 22:55:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000009' at position 346, relay log './dbsvc-528-relay-bin.000054' position: 649 2020-01-31 22:55:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:55:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000009' at position 346 2020-01-31 22:55:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:55:48 2020-01-31 22:55:55 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:55:55 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:55:55 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:55:55 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 22:56:09 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:56:09 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:56:09 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:56:09 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 22:56:12 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:56:16 14 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:56:16 14 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:56:17 15 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:56:17 15 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 22:56:18 18 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 22:56:18 18 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 22:56:41 37 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000027, 3160484) 2020-01-31 22:56:56 42 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:56:56 42 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 22:56:57 42 [ERROR] mysqld: Table './ksarn/wp_bp_activity' is marked as crashed and should be repaired 2020-01-31 22:56:57 42 [Warning] Checking table: './ksarn/wp_bp_activity' 2020-01-31 22:57:07 48 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity' is marked as crashed and should be repaired 2020-01-31 22:57:07 48 [Warning] Checking table: './ksarn/wp_learndash_user_activity' 2020-01-31 22:58:07 85 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 22:58:07 85 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 22:58:07 86 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 22:58:07 86 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 22:58:07 86 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 22:58:07 86 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 22:58:07 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f005c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f00711d9cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ecf7637f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ecf70ce3af] sigaction.c:0(__restore_rt)[0x7f00d2449630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55ecf6ffc151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55ecf713a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55ecf6ef0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55ecf6ee6d4b] /usr/sbin/mysqld(+0x6ca792)[0x55ecf6ef1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55ecf6f196da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ecf6f1b65d] /usr/sbin/mysqld(+0x65e3f3)[0x55ecf6e853f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55ecf6e84cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55ecf6f18b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ecf6f1b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55ecf6f1b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55ecf6f1c377] /usr/sbin/mysqld(+0x58fd84)[0x55ecf6db6d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55ecf6ec3b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55ecf6edbb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55ecf6edbcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55ecf6edc5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55ecf6edc677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55ecf6ecb56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55ecf6ecd07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55ecf6fa8afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55ecf6fa8bdd] pthread_create.c:0(start_thread)[0x7f00d2441ea5] /lib64/libc.so.6(clone+0x6d)[0x7f00d07e28cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f005c00fe80): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 87 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 22:58:14 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 22:58:14 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 22:58:14 0 [Note] InnoDB: Uses event mutexes 2020-01-31 22:58:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 22:58:14 0 [Note] InnoDB: Number of pools: 1 2020-01-31 22:58:14 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 22:58:14 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 22:58:14 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 22:58:14 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 22:58:14 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136628887 2020-01-31 22:58:14 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 22:58:14 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 22:58:14 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 22:58:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 22:58:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 22:58:14 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 22:58:14 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 22:58:14 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136628896; transaction id 5902176 2020-01-31 22:58:14 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 22:58:14 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 22:58:14 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 22:58:14 0 [Note] Starting crash recovery... 2020-01-31 22:58:14 0 [Note] Crash recovery finished. 2020-01-31 22:58:14 0 [Note] Server socket created on IP: '::'. 2020-01-31 22:58:14 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 22:58:14 0 [Note] Added new Master_info '' to hash table 2020-01-31 22:58:14 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000009' at position 112941 2020-01-31 22:58:14 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000009' at position 112941, relay log './dbsvc-528-relay-bin.000056' position: 73711 2020-01-31 22:58:14 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 22:58:14 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000009' at position 112941 2020-01-31 22:58:14 11 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 22:58:14 11 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 22:58:14 11 [Warning] wpcomm305.comm305_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:58:14 11 [Note] wpcomm305.comm305_options: Table is fixed 2020-01-31 22:58:15 11 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 22:58:15 11 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 22:58:15 11 [Warning] wpcomm303.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:58:15 11 [Note] wpcomm303.wp_options: Table is fixed 2020-01-31 22:58:17 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 22:58:17 2020-01-31 22:58:24 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 22:58:24 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 22:58:24 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:58:24 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 22:58:24 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 22:58:24 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 22:58:24 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:58:24 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 22:58:30 11 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 22:58:30 11 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 22:58:30 11 [Warning] wpstart.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:58:30 11 [Note] wpstart.wp_options: Table is fixed 2020-01-31 22:58:40 11 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 22:58:40 11 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 22:58:40 11 [Warning] ulrich.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 22:58:40 11 [Note] ulrich.wp_options: Table is fixed 2020-01-31 22:58:42 11 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 22:58:42 11 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 22:58:42 11 [Warning] ksarn.wp_usermeta: 1 client is using or hasn't closed the table properly 2020-01-31 22:58:42 11 [Note] ksarn.wp_usermeta: Table is fixed 2020-01-31 22:58:42 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 22:58:43 11 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity_meta' is marked as crashed and should be repaired 2020-01-31 22:58:43 11 [Warning] Checking table: './ksarn/wp_learndash_user_activity_meta' 2020-01-31 22:58:43 11 [Warning] ksarn.wp_learndash_user_activity_meta: 1 client is using or hasn't closed the table properly 2020-01-31 22:58:43 11 [Note] ksarn.wp_learndash_user_activity_meta: Table is fixed 2020-01-31 22:59:08 23 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000028, 659540) 2020-01-31 22:59:27 28 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 22:59:27 28 [Warning] Checking table: './wpms/wp_options' 2020-01-31 22:59:34 37 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 22:59:34 37 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 22:59:36 40 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 22:59:36 40 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 23:00:40 71 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:00:40 71 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:00:40 72 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:00:40 72 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:00:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe544000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe548c55cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5596421a2f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x559641c393af] sigaction.c:0(__restore_rt)[0x7fe56507e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x559641b67151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x559641ca5545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x559641a5bccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x559641a51d4b] /usr/sbin/mysqld(+0x6ca792)[0x559641a5c792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x559641a846da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559641a8665d] /usr/sbin/mysqld(+0x65e3f3)[0x5596419f03f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5596419efcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x559641a83b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559641a8665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x559641a86761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x559641a87377] /usr/sbin/mysqld(+0x58fd84)[0x559641921d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x559641a2eb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x559641a46b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x559641a46cd2] /usr/sbin/mysqld(+0x6b55e7)[0x559641a475e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x559641a47677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x559641a3656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x559641a3807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x559641b13afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x559641b13bdd] pthread_create.c:0(start_thread)[0x7fe565076ea5] /lib64/libc.so.6(clone+0x6d)[0x7fe5634178cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe54400ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 73 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:00:46 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:00:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:00:46 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:00:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:00:46 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:00:46 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:00:46 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:00:46 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:00:46 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:00:46 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136680213 2020-01-31 23:00:46 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:00:46 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:00:46 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:00:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:00:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:00:46 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:00:46 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:00:46 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136680222; transaction id 5902415 2020-01-31 23:00:46 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:00:46 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:00:46 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:00:46 0 [Note] Starting crash recovery... 2020-01-31 23:00:46 0 [Note] Crash recovery finished. 2020-01-31 23:00:46 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:00:46 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:00:46 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:00:46 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000009' at position 610726 2020-01-31 23:00:46 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:00:46 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000009' at position 610726, relay log './dbsvc-528-relay-bin.000058' position: 322808 2020-01-31 23:00:46 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000009' at position 610726 2020-01-31 23:00:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:00:48 2020-01-31 23:00:54 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:00:54 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:00:54 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:00:54 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 23:01:13 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:01:15 13 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:01:15 13 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:01:16 14 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:01:16 14 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 23:01:17 14 [ERROR] mysqld: Table './wpwsutv/wp_postmeta' is marked as crashed and should be repaired 2020-01-31 23:01:17 14 [Warning] Checking table: './wpwsutv/wp_postmeta' 2020-01-31 23:01:41 25 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000029, 557677) 2020-01-31 23:02:44 50 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:02:44 50 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 23:02:45 50 [ERROR] mysqld: Table './ksarn/wp_bp_activity' is marked as crashed and should be repaired 2020-01-31 23:02:45 50 [Warning] Checking table: './ksarn/wp_bp_activity' 2020-01-31 23:02:46 50 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity' is marked as crashed and should be repaired 2020-01-31 23:02:46 50 [Warning] Checking table: './ksarn/wp_learndash_user_activity' 2020-01-31 23:02:46 50 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity_meta' is marked as crashed and should be repaired 2020-01-31 23:02:46 50 [Warning] Checking table: './ksarn/wp_learndash_user_activity_meta' 2020-01-31 23:02:46 53 [ERROR] mysqld: Table './ksarn/wp_bp_activity_meta' is marked as crashed and should be repaired 2020-01-31 23:02:46 53 [Warning] Checking table: './ksarn/wp_bp_activity_meta' 2020-01-31 23:02:46 53 [ERROR] mysqld: Table './ksarn/wp_wp_pro_quiz_statistic' is marked as crashed and should be repaired 2020-01-31 23:02:46 53 [Warning] Checking table: './ksarn/wp_wp_pro_quiz_statistic' 2020-01-31 23:02:46 53 [ERROR] mysqld: Table './ksarn/wp_wp_pro_quiz_statistic_ref' is marked as crashed and should be repaired 2020-01-31 23:02:46 53 [Warning] Checking table: './ksarn/wp_wp_pro_quiz_statistic_ref' 2020-01-31 23:03:02 64 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 23:03:02 64 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 23:03:13 71 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 23:03:13 71 [Warning] Checking table: './wpms/wp_options' 2020-01-31 23:03:15 72 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 23:03:15 72 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 23:03:15 74 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 23:03:15 74 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 23:03:16 75 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 23:03:16 75 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 23:04:48 135 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 23:04:48 135 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 23:05:15 150 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 23:05:15 150 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 23:05:24 158 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 23:05:24 158 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 23:05:40 179 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:05:40 179 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:05:40 180 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:05:40 180 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:05:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=10 max_threads=502 thread_count=19 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f59a8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f59b414fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56507afb9f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56507aa503af] sigaction.c:0(__restore_rt)[0x7f5a203ee630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56507a97e151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56507aabc545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56507a872ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56507a868d4b] /usr/sbin/mysqld(+0x6ca792)[0x56507a873792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56507a89b6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56507a89d65d] /usr/sbin/mysqld(+0x65e3f3)[0x56507a8073f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56507a806cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56507a89ab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56507a89d65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56507a89d761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56507a89e377] /usr/sbin/mysqld(+0x58fd84)[0x56507a738d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56507a845b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56507a85db16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56507a85dcd2] /usr/sbin/mysqld(+0x6b55e7)[0x56507a85e5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56507a85e677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56507a84d56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56507a84f07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56507a92aafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56507a92abdd] pthread_create.c:0(start_thread)[0x7f5a203e6ea5] /lib64/libc.so.6(clone+0x6d)[0x7f5a1e7878cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f59a800fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 181 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:05:46 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:05:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:05:46 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:05:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:05:46 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:05:46 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:05:46 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:05:46 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:05:46 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:05:46 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136702049 2020-01-31 23:05:46 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:05:46 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:05:46 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:05:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:05:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:05:46 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:05:47 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:05:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136702058; transaction id 5902469 2020-01-31 23:05:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:05:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:05:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:05:47 0 [Note] Starting crash recovery... 2020-01-31 23:05:47 0 [Note] Crash recovery finished. 2020-01-31 23:05:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:05:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:05:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:05:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000009' at position 698689 2020-01-31 23:05:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000009' at position 698689, relay log './dbsvc-528-relay-bin.000060' position: 57671 2020-01-31 23:05:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:05:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000009' at position 698689 2020-01-31 23:05:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:05:48 2020-01-31 23:05:53 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:05:53 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:05:53 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:05:53 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 23:06:10 11 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 23:06:10 11 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 23:06:10 11 [Warning] itstraining.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:06:10 11 [Note] itstraining.wp_options: Table is fixed 2020-01-31 23:06:14 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:06:16 13 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:06:16 13 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:06:30 22 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 23:06:30 22 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 23:06:32 24 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 23:06:32 24 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 23:06:34 27 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:06:34 27 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 23:06:36 27 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity' is marked as crashed and should be repaired 2020-01-31 23:06:36 27 [Warning] Checking table: './ksarn/wp_learndash_user_activity' 2020-01-31 23:06:36 27 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity_meta' is marked as crashed and should be repaired 2020-01-31 23:06:36 27 [Warning] Checking table: './ksarn/wp_learndash_user_activity_meta' 2020-01-31 23:06:37 30 [ERROR] mysqld: Table './ksarn/wp_bp_activity' is marked as crashed and should be repaired 2020-01-31 23:06:37 30 [Warning] Checking table: './ksarn/wp_bp_activity' 2020-01-31 23:06:41 34 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000030, 3067492) 2020-01-31 23:07:01 41 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 23:07:01 41 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 23:07:26 54 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:07:26 54 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:07:26 55 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:07:26 55 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:07:27 56 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 23:07:27 56 [Warning] Checking table: './wpms/wp_options' 2020-01-31 23:08:14 108 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 23:08:14 108 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 23:08:16 110 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 23:08:16 110 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 23:08:39 120 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 23:08:39 120 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 23:09:19 143 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:09:19 143 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:09:19 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=9 max_threads=502 thread_count=18 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x560a81464e38 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0204078cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560a7c631f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560a7c0c83af] sigaction.c:0(__restore_rt)[0x7f021e60b630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x560a7bff6151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x560a7c134545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x560a7beeaccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x560a7bee0d4b] /usr/sbin/mysqld(+0x6ca792)[0x560a7beeb792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x560a7bf136da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560a7bf1565d] /usr/sbin/mysqld(+0x65e3f3)[0x560a7be7f3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x560a7be7ecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x560a7bf12b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560a7bf1565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x560a7bf15761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x560a7bf16377] /usr/sbin/mysqld(+0x58fd84)[0x560a7bdb0d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x560a7bebdb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x560a7bed5b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x560a7bed5cd2] /usr/sbin/mysqld(+0x6b55e7)[0x560a7bed65e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x560a7bed6677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x560a7bec556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x560a7bec707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x560a7bfa2afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560a7bfa2bdd] pthread_create.c:0(start_thread)[0x7f021e603ea5] /lib64/libc.so.6(clone+0x6d)[0x7f021c9a48cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x560a81473100): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 144 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:09:25 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:09:25 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:09:25 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:09:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:09:25 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:09:25 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:09:25 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:09:25 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:09:25 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:09:25 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136729889 2020-01-31 23:09:25 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:09:25 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:09:25 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:09:25 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:09:25 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:09:25 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:09:25 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:09:26 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136729898; transaction id 5902561 2020-01-31 23:09:26 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:09:26 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:09:26 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:09:26 0 [Note] Starting crash recovery... 2020-01-31 23:09:26 0 [Note] Crash recovery finished. 2020-01-31 23:09:26 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:09:26 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:09:26 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:09:26 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000009' at position 826497 2020-01-31 23:09:26 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000009' at position 826497, relay log './dbsvc-528-relay-bin.000062' position: 83318 2020-01-31 23:09:26 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:09:26 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000009' at position 826497 2020-01-31 23:09:26 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:09:26 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:09:26 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:09:26 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 23:09:28 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 23:09:28 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 23:09:28 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:09:28 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 23:09:29 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:09:29 2020-01-31 23:09:36 11 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 23:09:36 11 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 23:09:36 11 [Warning] shockeripe.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:09:36 11 [Note] shockeripe.wp_options: Table is fixed 2020-01-31 23:09:54 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:10:16 17 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:10:16 17 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 23:10:18 17 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity' is marked as crashed and should be repaired 2020-01-31 23:10:18 17 [Warning] Checking table: './ksarn/wp_learndash_user_activity' 2020-01-31 23:10:18 17 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity_meta' is marked as crashed and should be repaired 2020-01-31 23:10:18 17 [Warning] Checking table: './ksarn/wp_learndash_user_activity_meta' 2020-01-31 23:10:19 19 [ERROR] mysqld: Table './ksarn/wp_bp_activity' is marked as crashed and should be repaired 2020-01-31 23:10:19 19 [Warning] Checking table: './ksarn/wp_bp_activity' 2020-01-31 23:10:20 21 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000031, 945386) 2020-01-31 23:10:40 29 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:10:40 29 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:10:40 30 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:10:40 30 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:10:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f27d0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f27dc4f5cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5614ef476f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5614eef0d3af] sigaction.c:0(__restore_rt)[0x7f2848f36630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5614eee3b151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5614eef79545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5614eed2fccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5614eed25d4b] /usr/sbin/mysqld(+0x6ca792)[0x5614eed30792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5614eed586da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5614eed5a65d] /usr/sbin/mysqld(+0x65e3f3)[0x5614eecc43f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5614eecc3cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5614eed57b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5614eed5a65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5614eed5a761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5614eed5b377] /usr/sbin/mysqld(+0x58fd84)[0x5614eebf5d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5614eed02b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5614eed1ab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5614eed1acd2] /usr/sbin/mysqld(+0x6b55e7)[0x5614eed1b5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5614eed1b677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5614eed0a56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5614eed0c07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5614eede7afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5614eede7bdd] pthread_create.c:0(start_thread)[0x7f2848f2eea5] /lib64/libc.so.6(clone+0x6d)[0x7f28472cf8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f27d000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 31 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:10:46 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:10:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:10:46 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:10:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:10:46 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:10:46 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:10:46 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:10:46 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:10:46 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:10:46 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136742150 2020-01-31 23:10:46 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:10:46 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:10:46 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:10:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:10:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:10:46 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:10:46 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:10:46 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136742159; transaction id 5902612 2020-01-31 23:10:46 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:10:46 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:10:46 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:10:46 0 [Note] Starting crash recovery... 2020-01-31 23:10:46 0 [Note] Crash recovery finished. 2020-01-31 23:10:46 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:10:46 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:10:46 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:10:46 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000009' at position 930116 2020-01-31 23:10:46 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000009' at position 930116, relay log './dbsvc-528-relay-bin.000064' position: 67895 2020-01-31 23:10:46 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:10:46 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000009' at position 930116 2020-01-31 23:10:46 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:10:46 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:10:46 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:10:46 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 23:10:47 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:10:47 2020-01-31 23:11:11 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:11:11 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:11:11 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:11:11 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:11:12 11 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:11:12 11 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 23:11:12 11 [Warning] wpwsutv.wp_usermeta: 1 client is using or hasn't closed the table properly 2020-01-31 23:11:12 11 [Note] wpwsutv.wp_usermeta: Table is fixed 2020-01-31 23:11:14 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:11:29 16 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 23:11:29 16 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 23:11:37 21 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:11:37 21 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 23:11:37 21 [ERROR] mysqld: Table './ksarn/wp_bp_activity' is marked as crashed and should be repaired 2020-01-31 23:11:37 21 [Warning] Checking table: './ksarn/wp_bp_activity' 2020-01-31 23:11:37 21 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity' is marked as crashed and should be repaired 2020-01-31 23:11:37 21 [Warning] Checking table: './ksarn/wp_learndash_user_activity' 2020-01-31 23:11:39 23 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 23:11:39 23 [Warning] Checking table: './wpms/wp_options' 2020-01-31 23:11:40 26 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000032, 172581) 2020-01-31 23:11:44 27 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:11:44 27 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:11:44 28 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:11:44 28 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:11:44 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f88b0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f890c078cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5562e7a11f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5562e74a83af] sigaction.c:0(__restore_rt)[0x7f8926bcf630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5562e73d6151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5562e7514545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5562e72caccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5562e72c0d4b] /usr/sbin/mysqld(+0x6ca792)[0x5562e72cb792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5562e72f36da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5562e72f565d] /usr/sbin/mysqld(+0x65e3f3)[0x5562e725f3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5562e725ecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5562e72f2b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5562e72f565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5562e72f5761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5562e72f6377] /usr/sbin/mysqld(+0x58fd84)[0x5562e7190d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5562e729db6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5562e72b5b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5562e72b5cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5562e72b65e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5562e72b6677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5562e72a556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5562e72a707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5562e7382afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5562e7382bdd] pthread_create.c:0(start_thread)[0x7f8926bc7ea5] /lib64/libc.so.6(clone+0x6d)[0x7f8924f688cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f88b000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 29 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:11:51 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:11:51 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:11:51 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:11:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:11:51 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:11:51 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:11:51 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:11:51 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:11:51 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:11:51 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136762727 2020-01-31 23:11:51 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:11:51 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:11:51 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:11:51 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:11:51 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:11:51 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:11:51 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:11:51 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136762736; transaction id 5902679 2020-01-31 23:11:51 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:11:51 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:11:51 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:11:51 0 [Note] Starting crash recovery... 2020-01-31 23:11:51 0 [Note] Crash recovery finished. 2020-01-31 23:11:51 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:11:51 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:11:51 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:11:51 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000009' at position 1036737 2020-01-31 23:11:51 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000009' at position 1036737, relay log './dbsvc-528-relay-bin.000066' position: 70002 2020-01-31 23:11:51 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:11:51 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000009' at position 1036737 2020-01-31 23:11:52 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:11:52 2020-01-31 23:11:53 11 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 23:11:53 11 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 23:11:53 11 [Warning] wpsem.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:11:53 11 [Note] wpsem.wp_options: Table is fixed 2020-01-31 23:12:01 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 23:12:01 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 23:12:01 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:12:01 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 23:12:15 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:12:15 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:12:15 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:12:15 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 23:12:17 11 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:12:17 11 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 23:12:17 11 [Warning] ksarn.wp_usermeta: 1 client is using or hasn't closed the table properly 2020-01-31 23:12:17 11 [Note] ksarn.wp_usermeta: Table is fixed 2020-01-31 23:12:20 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:12:25 19 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:12:25 19 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:12:26 20 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:12:26 20 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 23:12:45 30 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000033, 56225) 2020-01-31 23:12:52 34 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 23:12:52 34 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 23:13:16 50 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 23:13:16 50 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 23:14:05 79 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 23:14:05 79 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 23:14:41 97 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:14:41 97 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:14:41 98 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:14:41 98 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:14:41 98 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:14:41 98 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:14:42 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=502 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f5814000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f57c82a4cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e2856b2f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e2851493af] sigaction.c:0(__restore_rt)[0x7f5833d59630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55e285077151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55e2851b5545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55e284f6bccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55e284f61d4b] /usr/sbin/mysqld(+0x6ca792)[0x55e284f6c792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55e284f946da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e284f9665d] /usr/sbin/mysqld(+0x65e3f3)[0x55e284f003f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55e284effcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55e284f93b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e284f9665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55e284f96761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55e284f97377] /usr/sbin/mysqld(+0x58fd84)[0x55e284e31d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55e284f3eb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55e284f56b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55e284f56cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55e284f575e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55e284f57677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55e284f4656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55e284f4807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55e285023afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e285023bdd] pthread_create.c:0(start_thread)[0x7f5833d51ea5] /lib64/libc.so.6(clone+0x6d)[0x7f58320f28cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f581400fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 100 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:14:49 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:14:49 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:14:49 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:14:49 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:14:49 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:14:49 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:14:49 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:14:49 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:14:49 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:14:49 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136784816 2020-01-31 23:14:49 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:14:49 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:14:49 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:14:49 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:14:49 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:14:49 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:14:49 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:14:49 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136784825; transaction id 5902769 2020-01-31 23:14:49 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:14:49 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:14:49 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:14:49 0 [Note] Starting crash recovery... 2020-01-31 23:14:49 0 [Note] Crash recovery finished. 2020-01-31 23:14:49 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:14:49 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:14:49 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:14:49 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000009' at position 1163293 2020-01-31 23:14:49 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000009' at position 1163293, relay log './dbsvc-528-relay-bin.000068' position: 83094 2020-01-31 23:14:50 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:14:50 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000009' at position 1163293 2020-01-31 23:14:50 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:14:50 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:14:50 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:14:50 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:14:51 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:14:51 2020-01-31 23:14:57 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:14:57 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:14:57 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:14:57 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 23:14:58 11 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:14:58 11 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 23:14:58 11 [Warning] ksarn.wp_usermeta: 1 client is using or hasn't closed the table properly 2020-01-31 23:14:58 11 [Note] ksarn.wp_usermeta: Table is fixed 2020-01-31 23:15:05 11 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 23:15:05 11 [Warning] Checking table: './wpms/wp_options' 2020-01-31 23:15:05 11 [Warning] wpms.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:15:05 11 [Note] wpms.wp_options: Table is fixed 2020-01-31 23:15:15 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:15:20 15 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 23:15:20 15 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 23:15:24 17 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 23:15:24 17 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 23:15:33 23 [ERROR] mysqld: Table './wpwsutv/wp_postmeta' is marked as crashed and should be repaired 2020-01-31 23:15:33 23 [Warning] Checking table: './wpwsutv/wp_postmeta' 2020-01-31 23:15:40 27 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:15:40 27 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:15:41 28 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:15:41 28 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:15:41 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f8dc4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f8e15ca2cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d147f3df2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55d1479d43af] sigaction.c:0(__restore_rt)[0x7f8e38e2f630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55d147902151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55d147a40545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55d1477f6ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55d1477ecd4b] /usr/sbin/mysqld(+0x6ca792)[0x55d1477f7792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55d14781f6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d14782165d] /usr/sbin/mysqld(+0x65e3f3)[0x55d14778b3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55d14778acb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55d14781eb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d14782165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55d147821761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55d147822377] /usr/sbin/mysqld(+0x58fd84)[0x55d1476bcd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55d1477c9b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55d1477e1b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55d1477e1cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55d1477e25e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55d1477e2677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55d1477d156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55d1477d307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55d1478aeafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d1478aebdd] pthread_create.c:0(start_thread)[0x7f8e38e27ea5] /lib64/libc.so.6(clone+0x6d)[0x7f8e371c88cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f8dc400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 29 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:15:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:15:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:15:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:15:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:15:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:15:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:15:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:15:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:15:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:15:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136802473 2020-01-31 23:15:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:15:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:15:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:15:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:15:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:15:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:15:47 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:15:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136802482; transaction id 5902845 2020-01-31 23:15:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:15:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:15:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:15:47 0 [Note] Starting crash recovery... 2020-01-31 23:15:47 0 [Note] Crash recovery finished. 2020-01-31 23:15:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:15:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:15:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:15:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000009' at position 1224696 2020-01-31 23:15:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000009' at position 1224696, relay log './dbsvc-528-relay-bin.000070' position: 40907 2020-01-31 23:15:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:15:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000009' at position 1224696 2020-01-31 23:15:49 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:15:49 2020-01-31 23:15:54 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:15:54 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:15:54 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:15:54 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 23:16:07 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:16:07 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:16:07 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:16:07 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:16:15 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:16:18 17 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 23:16:18 17 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 23:16:26 23 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:16:26 23 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 23:16:28 24 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity_meta' is marked as crashed and should be repaired 2020-01-31 23:16:28 24 [Warning] Checking table: './ksarn/wp_learndash_user_activity_meta' 2020-01-31 23:16:29 24 [ERROR] mysqld: Table './ksarn/wp_bp_activity' is marked as crashed and should be repaired 2020-01-31 23:16:29 24 [Warning] Checking table: './ksarn/wp_bp_activity' 2020-01-31 23:16:32 29 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 23:16:32 29 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 23:16:43 38 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000034, 552692) 2020-01-31 23:16:45 39 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 23:16:45 39 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 23:16:56 46 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 23:16:56 46 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 23:17:09 63 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:17:09 63 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:17:09 64 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:17:09 64 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:17:10 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f1610000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f16759efcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55f7e4722f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55f7e41b93af] sigaction.c:0(__restore_rt)[0x7f1690b55630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55f7e40e7151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55f7e4225545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55f7e3fdbccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55f7e3fd1d4b] /usr/sbin/mysqld(+0x6ca792)[0x55f7e3fdc792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55f7e40046da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55f7e400665d] /usr/sbin/mysqld(+0x65e3f3)[0x55f7e3f703f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55f7e3f6fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55f7e4003b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55f7e400665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55f7e4006761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55f7e4007377] /usr/sbin/mysqld(+0x58fd84)[0x55f7e3ea1d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55f7e3faeb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55f7e3fc6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55f7e3fc6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55f7e3fc75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55f7e3fc7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55f7e3fb656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55f7e3fb807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55f7e4093afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55f7e4093bdd] pthread_create.c:0(start_thread)[0x7f1690b4dea5] /lib64/libc.so.6(clone+0x6d)[0x7f168eeee8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f161000fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 65 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:17:16 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:17:16 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:17:16 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:17:16 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:17:16 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:17:16 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:17:16 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:17:16 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:17:16 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:17:17 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136823002 2020-01-31 23:17:17 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:17:17 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:17:17 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:17:17 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:17:17 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:17:17 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:17:17 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:17:17 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136823011; transaction id 5902906 2020-01-31 23:17:17 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:17:17 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:17:17 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:17:17 0 [Note] Starting crash recovery... 2020-01-31 23:17:17 0 [Note] Crash recovery finished. 2020-01-31 23:17:17 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:17:17 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:17:17 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:17:17 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000009' at position 1278705 2020-01-31 23:17:17 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000009' at position 1278705, relay log './dbsvc-528-relay-bin.000072' position: 35766 2020-01-31 23:17:17 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:17:17 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000009' at position 1278705 2020-01-31 23:17:17 11 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:17:17 11 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:17:17 11 [Warning] ksarn.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:17:17 11 [Note] ksarn.wp_options: Table is fixed 2020-01-31 23:17:19 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:17:19 2020-01-31 23:17:22 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:17:22 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:17:22 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:17:22 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:17:23 11 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:17:23 11 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 23:17:23 11 [Warning] ksarn.wp_usermeta: 1 client is using or hasn't closed the table properly 2020-01-31 23:17:23 11 [Note] ksarn.wp_usermeta: Table is fixed 2020-01-31 23:17:23 11 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity' is marked as crashed and should be repaired 2020-01-31 23:17:23 11 [Warning] Checking table: './ksarn/wp_learndash_user_activity' 2020-01-31 23:17:23 11 [Warning] ksarn.wp_learndash_user_activity: 1 client is using or hasn't closed the table properly 2020-01-31 23:17:23 11 [Note] ksarn.wp_learndash_user_activity: Table is fixed 2020-01-31 23:17:31 11 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity_meta' is marked as crashed and should be repaired 2020-01-31 23:17:31 11 [Warning] Checking table: './ksarn/wp_learndash_user_activity_meta' 2020-01-31 23:17:31 11 [Warning] ksarn.wp_learndash_user_activity_meta: 1 client is using or hasn't closed the table properly 2020-01-31 23:17:31 11 [Note] ksarn.wp_learndash_user_activity_meta: Table is fixed 2020-01-31 23:17:33 11 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 23:17:33 11 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 23:17:33 11 [Warning] wpcomm306.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:17:33 11 [Note] wpcomm306.wp_options: Table is fixed 2020-01-31 23:17:46 17 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:18:01 30 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 23:18:01 30 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 23:18:11 33 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000036, 612885) 2020-01-31 23:18:16 40 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 23:18:16 40 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 23:18:39 51 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 23:18:39 51 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 23:18:47 57 [ERROR] mysqld: Table './ksarn/wp_bp_activity' is marked as crashed and should be repaired 2020-01-31 23:18:47 57 [Warning] Checking table: './ksarn/wp_bp_activity' 2020-01-31 23:19:03 66 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 23:19:03 66 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 23:19:27 77 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 23:19:27 77 [Warning] Checking table: './wpms/wp_options' 2020-01-31 23:19:29 78 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 23:19:29 78 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 23:19:50 90 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 23:19:50 90 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 23:20:39 106 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 23:20:39 106 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 23:20:40 108 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:20:40 108 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:20:40 109 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:20:40 109 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:20:41 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7faa60006c18 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7faac4653cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d4981e4f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55d497c7b3af] sigaction.c:0(__restore_rt)[0x7faae09a8630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55d497ba9151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55d497ce7545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55d497a9dccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55d497a93d4b] /usr/sbin/mysqld(+0x6ca792)[0x55d497a9e792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55d497ac66da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d497ac865d] /usr/sbin/mysqld(+0x65e3f3)[0x55d497a323f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55d497a31cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55d497ac5b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d497ac865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55d497ac8761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55d497ac9377] /usr/sbin/mysqld(+0x58fd84)[0x55d497963d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55d497a70b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55d497a88b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55d497a88cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55d497a895e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55d497a89677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55d497a7856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55d497a7a07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55d497b55afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d497b55bdd] pthread_create.c:0(start_thread)[0x7faae09a0ea5] /lib64/libc.so.6(clone+0x6d)[0x7faaded418cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7faa60010ec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 110 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:20:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:20:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:20:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:20:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:20:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:20:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:20:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:20:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:20:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:20:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136847846 2020-01-31 23:20:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:20:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:20:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:20:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:20:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:20:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:20:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136847855; transaction id 5903018 2020-01-31 23:20:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:20:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:20:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:20:47 0 [Note] Starting crash recovery... 2020-01-31 23:20:47 0 [Note] Crash recovery finished. 2020-01-31 23:20:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:20:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:20:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:20:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000009' at position 1409506 2020-01-31 23:20:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000009' at position 1409506, relay log './dbsvc-528-relay-bin.000074' position: 85592 2020-01-31 23:20:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:20:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000009' at position 1409506 2020-01-31 23:20:49 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:20:49 2020-01-31 23:21:03 11 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 23:21:03 11 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 23:21:03 11 [Warning] wpstart.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:21:03 11 [Note] wpstart.wp_options: Table is fixed 2020-01-31 23:21:09 11 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:21:09 11 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:21:09 11 [Warning] ojsdata.sessions: 1 client is using or hasn't closed the table properly 2020-01-31 23:21:09 11 [Note] ojsdata.sessions: Table is fixed 2020-01-31 23:21:10 11 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:21:10 11 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:21:10 11 [Warning] ojsdata.scheduled_tasks: 1 client is using or hasn't closed the table properly 2020-01-31 23:21:10 11 [Note] ojsdata.scheduled_tasks: Table is fixed 2020-01-31 23:21:12 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:21:12 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000009' at position 1599338 2020-01-31 23:21:12 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:21:16 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:21:16 13 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:21:16 13 [Warning] Checking table: './wpwsutv/wp_options' 200131 23:21:17 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f59e0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f5a30101cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d3987a0f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55d3982373af] sigaction.c:0(__restore_rt)[0x7f5a4d88c630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55d398165151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55d3982a3545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55d398059ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55d39804fd4b] /usr/sbin/mysqld(+0x6ca792)[0x55d39805a792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55d3980826da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d39808465d] /usr/sbin/mysqld(+0x65e3f3)[0x55d397fee3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55d397fedcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55d398081b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d39808465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55d398084761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55d398085377] /usr/sbin/mysqld(+0x58fd84)[0x55d397f1fd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55d39802cb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55d398044b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55d398044cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55d3980455e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55d398045677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55d39803456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55d39803607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55d398111afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d398111bdd] pthread_create.c:0(start_thread)[0x7f5a4d884ea5] /lib64/libc.so.6(clone+0x6d)[0x7f5a4bc258cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f59e000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:21:23 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:21:23 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:21:23 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:21:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:21:23 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:21:23 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:21:23 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:21:23 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:21:23 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:21:23 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136865923 2020-01-31 23:21:23 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:21:23 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:21:23 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:21:23 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:21:23 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:21:23 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:21:23 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:21:23 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136865932; transaction id 5903085 2020-01-31 23:21:23 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:21:23 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:21:23 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:21:23 0 [Note] Starting crash recovery... 2020-01-31 23:21:23 0 [Note] Crash recovery finished. 2020-01-31 23:21:23 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:21:23 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:21:23 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:21:23 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000009' at position 1599338 2020-01-31 23:21:23 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000009' at position 1599338, relay log './dbsvc-528-relay-bin.000076' position: 124882 2020-01-31 23:21:23 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:21:23 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000009' at position 1599338 2020-01-31 23:21:25 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:21:25 2020-01-31 23:21:46 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:21:46 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:21:46 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:21:46 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:21:46 11 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Duplicate entry '_transient_doing_cron' for key 'option_name', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log dbsvc-527-bin.000010, end_log_pos 862, Gtid 0-1-2496625, Internal MariaDB error code: 1062 2020-01-31 23:21:46 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:21:46 11 [Warning] Slave: Duplicate entry '_transient_doing_cron' for key 'option_name' Error_code: 1062 2020-01-31 23:21:46 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 346 2020-01-31 23:21:46 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 346 2020-01-31 23:21:51 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:22:11 17 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:22:11 17 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 23:22:13 19 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 23:22:13 19 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 23:22:18 27 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000037, 365289) 2020-01-31 23:22:20 28 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 23:22:20 28 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 23:22:32 34 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:22:32 34 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:22:33 34 [ERROR] mysqld: Table './ksarn/wp_postmeta' is marked as crashed and should be repaired 2020-01-31 23:22:33 34 [Warning] Checking table: './ksarn/wp_postmeta' 2020-01-31 23:23:15 55 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 23:23:15 55 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 23:23:15 56 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 23:23:15 56 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 23:23:22 60 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 23:23:22 60 [Warning] Checking table: './wpms/wp_options' 2020-01-31 23:23:37 70 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:23:37 70 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 23:23:39 70 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity' is marked as crashed and should be repaired 2020-01-31 23:23:39 70 [Warning] Checking table: './ksarn/wp_learndash_user_activity' 2020-01-31 23:24:03 85 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity_meta' is marked as crashed and should be repaired 2020-01-31 23:24:03 85 [Warning] Checking table: './ksarn/wp_learndash_user_activity_meta' 2020-01-31 23:24:59 114 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:24:59 114 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:24:59 115 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:24:59 115 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:24:59 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f33b00009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f341c0fccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d64799cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55d6474333af] sigaction.c:0(__restore_rt)[0x7f34360b1630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55d647361151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55d64749f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55d647255ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55d64724bd4b] /usr/sbin/mysqld(+0x6ca792)[0x55d647256792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55d64727e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d64728065d] /usr/sbin/mysqld(+0x65e3f3)[0x55d6471ea3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55d6471e9cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55d64727db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d64728065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55d647280761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55d647281377] /usr/sbin/mysqld(+0x58fd84)[0x55d64711bd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55d647228b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55d647240b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55d647240cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55d6472415e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55d647241677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55d64723056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55d64723207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55d64730dafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d64730dbdd] pthread_create.c:0(start_thread)[0x7f34360a9ea5] /lib64/libc.so.6(clone+0x6d)[0x7f343444a8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f33b00109f0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 116 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:25:05 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:25:05 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:25:05 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:25:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:25:05 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:25:05 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:25:05 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:25:05 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:25:05 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:25:05 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136874122 2020-01-31 23:25:05 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:25:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:25:05 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:25:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:25:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:25:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:25:05 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136874131; transaction id 5903110 2020-01-31 23:25:05 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:25:05 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:25:05 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:25:05 0 [Note] Starting crash recovery... 2020-01-31 23:25:05 0 [Note] Crash recovery finished. 2020-01-31 23:25:05 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:25:05 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:25:05 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:25:05 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000010' at position 24548 2020-01-31 23:25:05 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 346, relay log './dbsvc-528-relay-bin.000079' position: 649 2020-01-31 23:25:05 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:25:05 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000010' at position 24548 2020-01-31 23:25:05 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:25:05 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:25:05 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:25:05 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:25:05 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:25:05 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:25:05 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:25:05 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:25:05 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:25:06 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:25:06 2020-01-31 23:25:32 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:25:40 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:25:40 14 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:25:40 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:25:40 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:25:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f56e0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f5744066cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561a87ef7f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x561a8798e3af] sigaction.c:0(__restore_rt)[0x7f575e2cb630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x561a878bc151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x561a879fa545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x561a877b0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x561a877a6d4b] /usr/sbin/mysqld(+0x6ca792)[0x561a877b1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x561a877d96da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x561a877db65d] /usr/sbin/mysqld(+0x65e3f3)[0x561a877453f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x561a87744cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x561a877d8b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x561a877db65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x561a877db761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x561a877dc377] /usr/sbin/mysqld(+0x58fd84)[0x561a87676d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x561a87783b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x561a8779bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x561a8779bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x561a8779c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x561a8779c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x561a8778b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x561a8778d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x561a87868afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x561a87868bdd] pthread_create.c:0(start_thread)[0x7f575e2c3ea5] /lib64/libc.so.6(clone+0x6d)[0x7f575c6648cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f56e000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:25:46 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:25:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:25:46 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:25:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:25:46 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:25:46 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:25:46 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:25:46 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:25:46 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:25:46 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136881369 2020-01-31 23:25:46 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:25:46 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:25:46 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:25:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:25:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:25:46 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:25:46 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:25:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136881378; transaction id 5903137 2020-01-31 23:25:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:25:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:25:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:25:47 0 [Note] Starting crash recovery... 2020-01-31 23:25:47 0 [Note] Crash recovery finished. 2020-01-31 23:25:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:25:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:25:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:25:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000010' at position 237831 2020-01-31 23:25:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:25:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:25:47 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:25:47 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:25:47 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:25:47 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:25:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000010' at position 237831 2020-01-31 23:25:47 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:25:47 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:25:47 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:25:47 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:25:47 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:25:47 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:25:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:25:48 2020-01-31 23:25:59 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000039, 648523) 2020-01-31 23:26:12 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:26:17 15 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:26:17 15 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 23:26:18 18 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 23:26:18 18 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 23:26:28 23 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 23:26:28 23 [Warning] Checking table: './wpms/wp_options' 2020-01-31 23:26:55 40 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:26:55 40 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:26:56 40 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:26:56 40 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 23:26:57 40 [ERROR] mysqld: Table './ksarn/wp_bp_activity' is marked as crashed and should be repaired 2020-01-31 23:26:57 40 [Warning] Checking table: './ksarn/wp_bp_activity' 2020-01-31 23:26:57 40 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity' is marked as crashed and should be repaired 2020-01-31 23:26:57 40 [Warning] Checking table: './ksarn/wp_learndash_user_activity' 2020-01-31 23:27:58 69 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 23:27:58 69 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 23:28:15 76 [ERROR] mysqld: Table './ksarn/wp_wp_pro_quiz_statistic_ref' is marked as crashed and should be repaired 2020-01-31 23:28:15 76 [Warning] Checking table: './ksarn/wp_wp_pro_quiz_statistic_ref' 2020-01-31 23:28:15 76 [ERROR] mysqld: Table './ksarn/wp_wp_pro_quiz_statistic' is marked as crashed and should be repaired 2020-01-31 23:28:15 76 [Warning] Checking table: './ksarn/wp_wp_pro_quiz_statistic' 2020-01-31 23:28:15 76 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity_meta' is marked as crashed and should be repaired 2020-01-31 23:28:15 76 [Warning] Checking table: './ksarn/wp_learndash_user_activity_meta' 2020-01-31 23:28:16 78 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 23:28:16 78 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 23:28:16 76 [ERROR] mysqld: Table './ksarn/wp_bp_activity_meta' is marked as crashed and should be repaired 2020-01-31 23:28:16 76 [Warning] Checking table: './ksarn/wp_bp_activity_meta' 2020-01-31 23:28:28 94 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:28:28 94 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:28:28 95 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:28:28 95 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:28:28 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9724000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f96e1a4dcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5647dbe34f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5647db8cb3af] sigaction.c:0(__restore_rt)[0x7f9742c95630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5647db7f9151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5647db937545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5647db6edccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5647db6e3d4b] /usr/sbin/mysqld(+0x6ca792)[0x5647db6ee792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5647db7166da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5647db71865d] /usr/sbin/mysqld(+0x65e3f3)[0x5647db6823f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5647db681cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5647db715b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5647db71865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5647db718761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5647db719377] /usr/sbin/mysqld(+0x58fd84)[0x5647db5b3d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5647db6c0b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5647db6d8b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5647db6d8cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5647db6d95e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5647db6d9677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5647db6c856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5647db6ca07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5647db7a5afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5647db7a5bdd] pthread_create.c:0(start_thread)[0x7f9742c8dea5] /lib64/libc.so.6(clone+0x6d)[0x7f974102e8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f972400ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 96 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:28:34 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:28:34 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:28:34 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:28:34 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:28:34 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:28:34 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:28:34 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:28:34 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:28:34 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:28:34 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136887446 2020-01-31 23:28:34 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:28:34 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:28:34 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:28:34 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:28:34 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:28:34 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:28:34 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136887455; transaction id 5903163 2020-01-31 23:28:34 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:28:35 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:28:35 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:28:35 0 [Note] Starting crash recovery... 2020-01-31 23:28:35 0 [Note] Crash recovery finished. 2020-01-31 23:28:35 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:28:35 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:28:35 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:28:35 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000010' at position 347379 2020-01-31 23:28:35 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:28:35 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:28:35 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000010' at position 347379 2020-01-31 23:28:35 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:28:35 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:28:35 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:28:35 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:28:35 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:28:35 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:28:35 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:28:35 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:28:35 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:28:35 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:28:36 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:28:36 2020-01-31 23:28:57 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:28:57 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000010' at position 351440 2020-01-31 23:28:57 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:29:03 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:29:03 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000041, 484159) 2020-01-31 23:29:06 14 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-01-31 23:29:06 14 [Warning] Checking table: './idtgroup/wp_options' 2020-01-31 23:29:07 18 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:29:07 18 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:29:07 19 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:29:07 19 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:29:07 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=9 max_threads=502 thread_count=17 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f29f8000f18 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2a3c0a1cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x558717ba8f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55871763f3af] sigaction.c:0(__restore_rt)[0x7f2aa8237630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55871756d151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5587176ab545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x558717461ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x558717457d4b] /usr/sbin/mysqld(+0x6ca792)[0x558717462792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55871748a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55871748c65d] /usr/sbin/mysqld(+0x65e3f3)[0x5587173f63f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5587173f5cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x558717489b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55871748c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55871748c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55871748d377] /usr/sbin/mysqld(+0x58fd84)[0x558717327d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x558717434b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55871744cb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55871744ccd2] /usr/sbin/mysqld(+0x6b55e7)[0x55871744d5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55871744d677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55871743c56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55871743e07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x558717519afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x558717519bdd] pthread_create.c:0(start_thread)[0x7f2aa822fea5] /lib64/libc.so.6(clone+0x6d)[0x7f2aa65d08cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f29f8010370): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 21 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:29:14 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:29:14 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:29:14 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:29:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:29:14 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:29:14 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:29:14 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:29:14 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:29:14 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:29:14 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136890272 2020-01-31 23:29:14 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:29:14 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:29:14 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:29:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:29:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:29:14 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:29:14 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:29:15 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136890281; transaction id 5903174 2020-01-31 23:29:15 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:29:15 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:29:15 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:29:15 0 [Note] Starting crash recovery... 2020-01-31 23:29:15 0 [Note] Crash recovery finished. 2020-01-31 23:29:15 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:29:15 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:29:15 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:29:15 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000010' at position 351440 2020-01-31 23:29:15 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:29:15 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:29:15 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000010' at position 351440 2020-01-31 23:29:15 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:29:15 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:29:15 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:29:15 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:29:15 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:29:16 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:29:16 2020-01-31 23:29:43 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:29:43 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000011' at position 274456 2020-01-31 23:29:43 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:29:43 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:29:46 18 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:29:46 18 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:29:46 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:29:46 20 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:29:46 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:29:46 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:29:46 21 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:29:46 21 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:29:47 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fab10000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fab7499acf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e99f8a7f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e99f33e3af] sigaction.c:0(__restore_rt)[0x7fab91aef630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55e99f26c151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55e99f3aa545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55e99f160ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55e99f156d4b] /usr/sbin/mysqld(+0x6ca792)[0x55e99f161792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55e99f1896da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e99f18b65d] /usr/sbin/mysqld(+0x65e3f3)[0x55e99f0f53f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55e99f0f4cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55e99f188b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e99f18b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55e99f18b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55e99f18c377] /usr/sbin/mysqld(+0x58fd84)[0x55e99f026d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55e99f133b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55e99f14bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55e99f14bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55e99f14c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55e99f14c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55e99f13b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55e99f13d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55e99f218afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e99f218bdd] pthread_create.c:0(start_thread)[0x7fab91ae7ea5] /lib64/libc.so.6(clone+0x6d)[0x7fab8fe888cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fab1000fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:29:54 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:29:54 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:29:54 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:29:54 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:29:54 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:29:54 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:29:54 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:29:54 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:29:54 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:29:54 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136893102 2020-01-31 23:29:54 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:29:54 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:29:54 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:29:54 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:29:54 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:29:54 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:29:54 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:29:54 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136893111; transaction id 5903185 2020-01-31 23:29:54 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:29:54 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:29:54 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:29:54 0 [Note] Starting crash recovery... 2020-01-31 23:29:54 0 [Note] Crash recovery finished. 2020-01-31 23:29:54 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:29:54 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:29:54 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:29:54 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000011' at position 274456 2020-01-31 23:29:54 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:29:54 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:29:54 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000011' at position 274456 2020-01-31 23:29:54 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:29:54 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:29:54 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:29:54 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:29:54 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:29:55 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:29:55 2020-01-31 23:30:23 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:30:23 13 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:30:23 13 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:30:25 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:30:25 16 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:30:25 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:30:25 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:30:25 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff19c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff1f4732cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55f1d1ad0f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55f1d15673af] sigaction.c:0(__restore_rt)[0x7ff2118f3630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55f1d1495151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55f1d15d3545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55f1d1389ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55f1d137fd4b] /usr/sbin/mysqld(+0x6ca792)[0x55f1d138a792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55f1d13b26da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55f1d13b465d] /usr/sbin/mysqld(+0x65e3f3)[0x55f1d131e3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55f1d131dcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55f1d13b1b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55f1d13b465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55f1d13b4761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55f1d13b5377] /usr/sbin/mysqld(+0x58fd84)[0x55f1d124fd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55f1d135cb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55f1d1374b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55f1d1374cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55f1d13755e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55f1d1375677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55f1d136456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55f1d136607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55f1d1441afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55f1d1441bdd] pthread_create.c:0(start_thread)[0x7ff2118ebea5] /lib64/libc.so.6(clone+0x6d)[0x7ff20fc8c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff19c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:30:31 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:30:31 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:30:31 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:30:31 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:30:31 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:30:31 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:30:31 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:30:31 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:30:31 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:30:31 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136895949 2020-01-31 23:30:31 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:30:31 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:30:31 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:30:31 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:30:31 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:30:31 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:30:31 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:30:32 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136895958; transaction id 5903196 2020-01-31 23:30:32 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:30:32 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:30:32 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:30:32 0 [Note] Starting crash recovery... 2020-01-31 23:30:32 0 [Note] Crash recovery finished. 2020-01-31 23:30:32 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:30:32 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:30:32 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:30:32 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000012' at position 49882 2020-01-31 23:30:32 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:30:32 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000012' at position 49882 2020-01-31 23:30:32 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:30:32 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:30:32 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:30:32 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:30:32 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:30:32 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:30:33 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:30:33 2020-01-31 23:30:41 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:30:41 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000012' at position 65415 2020-01-31 23:30:41 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:30:47 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000042, 1464) 2020-01-31 23:30:58 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:31:01 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:31:01 14 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:31:17 23 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:31:17 23 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:31:17 24 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:31:17 24 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:31:17 24 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:31:17 24 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:31:18 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f30c8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f3124940cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a332d12f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a3327a93af] sigaction.c:0(__restore_rt)[0x7f313fad8630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a3326d7151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a332815545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a3325cbccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a3325c1d4b] /usr/sbin/mysqld(+0x6ca792)[0x55a3325cc792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a3325f46da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a3325f665d] /usr/sbin/mysqld(+0x65e3f3)[0x55a3325603f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a33255fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a3325f3b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a3325f665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a3325f6761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a3325f7377] /usr/sbin/mysqld(+0x58fd84)[0x55a332491d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a33259eb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a3325b6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a3325b6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a3325b75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a3325b7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a3325a656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a3325a807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a332683afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a332683bdd] pthread_create.c:0(start_thread)[0x7f313fad0ea5] /lib64/libc.so.6(clone+0x6d)[0x7f313de718cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f30c800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 25 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:31:23 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:31:23 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:31:23 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:31:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:31:23 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:31:23 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:31:23 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:31:23 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:31:23 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:31:23 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136898812 2020-01-31 23:31:23 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:31:24 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:31:24 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:31:24 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:31:24 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:31:24 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:31:24 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:31:24 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136898821; transaction id 5903207 2020-01-31 23:31:24 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:31:24 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:31:24 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:31:24 0 [Note] Starting crash recovery... 2020-01-31 23:31:24 0 [Note] Crash recovery finished. 2020-01-31 23:31:24 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:31:24 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:31:24 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:31:24 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000012' at position 65415 2020-01-31 23:31:24 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:31:24 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:31:24 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:31:24 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:31:24 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:31:24 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:31:24 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000012' at position 65415 2020-01-31 23:31:24 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:31:24 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:31:24 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:31:24 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:31:24 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:31:24 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:31:25 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:31:25 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000013' at position 1464 2020-01-31 23:31:25 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:31:25 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:31:25 2020-01-31 23:31:31 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000045, 66837) 2020-01-31 23:31:49 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:32:03 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:32:03 16 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:32:03 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:32:03 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:32:03 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7efef0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7eff4c15bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563d049fbf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x563d044923af] sigaction.c:0(__restore_rt)[0x7eff67529630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x563d043c0151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x563d044fe545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x563d042b4ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x563d042aad4b] /usr/sbin/mysqld(+0x6ca792)[0x563d042b5792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x563d042dd6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563d042df65d] /usr/sbin/mysqld(+0x65e3f3)[0x563d042493f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x563d04248cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x563d042dcb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563d042df65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x563d042df761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x563d042e0377] /usr/sbin/mysqld(+0x58fd84)[0x563d0417ad84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x563d04287b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x563d0429fb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x563d0429fcd2] /usr/sbin/mysqld(+0x6b55e7)[0x563d042a05e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x563d042a0677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x563d0428f56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x563d0429107c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x563d0436cafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x563d0436cbdd] pthread_create.c:0(start_thread)[0x7eff67521ea5] /lib64/libc.so.6(clone+0x6d)[0x7eff658c28cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7efef000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:32:09 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:32:09 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:32:09 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:32:09 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:32:09 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:32:09 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:32:09 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:32:09 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:32:09 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:32:09 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136901675 2020-01-31 23:32:09 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:32:09 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:32:09 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:32:09 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:32:09 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:32:09 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:32:09 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:32:09 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136901684; transaction id 5903218 2020-01-31 23:32:09 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:32:09 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:32:09 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:32:09 0 [Note] Starting crash recovery... 2020-01-31 23:32:09 0 [Note] Crash recovery finished. 2020-01-31 23:32:09 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:32:09 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:32:09 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:32:09 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000013' at position 1464 2020-01-31 23:32:09 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:32:09 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:32:09 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:32:09 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:32:09 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:32:09 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:32:09 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:32:09 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000013' at position 1464 2020-01-31 23:32:10 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:32:10 2020-01-31 23:32:34 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:32:37 18 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:32:37 18 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:32:37 19 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:32:37 19 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:32:37 19 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:32:37 19 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:32:37 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc078000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc0c83b8cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x557671a90f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5576715273af] sigaction.c:0(__restore_rt)[0x7fc0e54e7630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x557671455151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x557671593545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x557671349ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55767133fd4b] /usr/sbin/mysqld(+0x6ca792)[0x55767134a792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5576713726da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55767137465d] /usr/sbin/mysqld(+0x65e3f3)[0x5576712de3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5576712ddcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x557671371b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55767137465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x557671374761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x557671375377] /usr/sbin/mysqld(+0x58fd84)[0x55767120fd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55767131cb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x557671334b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x557671334cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5576713355e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x557671335677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55767132456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55767132607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x557671401afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x557671401bdd] pthread_create.c:0(start_thread)[0x7fc0e54dfea5] /lib64/libc.so.6(clone+0x6d)[0x7fc0e38808cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc07800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 20 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:32:43 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:32:43 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:32:43 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:32:43 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:32:43 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:32:43 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:32:43 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:32:43 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:32:43 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:32:43 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136904522 2020-01-31 23:32:43 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:32:43 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:32:43 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:32:43 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:32:43 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:32:43 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:32:43 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:32:44 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136904531; transaction id 5903229 2020-01-31 23:32:44 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:32:44 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:32:44 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:32:44 0 [Note] Starting crash recovery... 2020-01-31 23:32:44 0 [Note] Crash recovery finished. 2020-01-31 23:32:44 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:32:44 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:32:44 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:32:44 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000014' at position 158603 2020-01-31 23:32:44 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:32:44 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:32:44 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000014' at position 158603 2020-01-31 23:32:44 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:32:44 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:32:44 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:32:44 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:32:44 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:32:45 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:32:45 2020-01-31 23:32:53 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:32:53 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000014' at position 159721 2020-01-31 23:32:53 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:32:59 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000046, 1464) 2020-01-31 23:33:09 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:33:09 15 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:33:09 15 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:33:11 15 [ERROR] mysqld: Table './ksarn/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:33:11 15 [Warning] Checking table: './ksarn/wp_usermeta' 2020-01-31 23:33:12 17 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 23:33:12 17 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 23:33:15 22 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 23:33:15 22 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 23:33:53 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000014' at position 159721 2020-01-31 23:34:21 52 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:34:21 52 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:34:21 53 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:34:21 53 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:34:21 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2e0c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2e640bfcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55aaba6a6f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55aaba13d3af] sigaction.c:0(__restore_rt)[0x7f2e81c1e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55aaba06b151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55aaba1a9545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55aab9f5fccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55aab9f55d4b] /usr/sbin/mysqld(+0x6ca792)[0x55aab9f60792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55aab9f886da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55aab9f8a65d] /usr/sbin/mysqld(+0x65e3f3)[0x55aab9ef43f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55aab9ef3cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55aab9f87b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55aab9f8a65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55aab9f8a761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55aab9f8b377] /usr/sbin/mysqld(+0x58fd84)[0x55aab9e25d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55aab9f32b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55aab9f4ab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55aab9f4acd2] /usr/sbin/mysqld(+0x6b55e7)[0x55aab9f4b5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55aab9f4b677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55aab9f3a56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55aab9f3c07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55aaba017afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55aaba017bdd] pthread_create.c:0(start_thread)[0x7f2e81c16ea5] /lib64/libc.so.6(clone+0x6d)[0x7f2e7ffb78cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f2e0c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 54 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:34:28 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:34:28 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:34:28 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:34:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:34:28 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:34:28 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:34:28 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:34:28 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:34:28 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:34:28 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136907696 2020-01-31 23:34:28 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:34:28 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:34:28 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:34:28 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:34:28 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:34:28 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:34:28 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:34:28 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136907705; transaction id 5903249 2020-01-31 23:34:28 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:34:28 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:34:28 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:34:28 0 [Note] Starting crash recovery... 2020-01-31 23:34:28 0 [Note] Crash recovery finished. 2020-01-31 23:34:28 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:34:28 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:34:28 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:34:28 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000015' at position 346 2020-01-31 23:34:28 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:34:28 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:34:28 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000015' at position 346 2020-01-31 23:34:28 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:34:28 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:34:28 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:34:28 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:34:28 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:34:28 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:34:28 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:34:28 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:34:28 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:34:28 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:34:29 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:34:29 2020-01-31 23:34:54 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:35:06 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:35:06 17 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:35:06 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:35:06 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:35:06 18 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:35:06 18 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:35:07 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fcae8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fcb48b6fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x564049160f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x564048bf73af] sigaction.c:0(__restore_rt)[0x7fcb63c9b630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564048b25151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564048c63545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564048a19ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x564048a0fd4b] /usr/sbin/mysqld(+0x6ca792)[0x564048a1a792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x564048a426da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564048a4465d] /usr/sbin/mysqld(+0x65e3f3)[0x5640489ae3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5640489adcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x564048a41b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564048a4465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x564048a44761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x564048a45377] /usr/sbin/mysqld(+0x58fd84)[0x5640488dfd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5640489ecb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564048a04b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564048a04cd2] /usr/sbin/mysqld(+0x6b55e7)[0x564048a055e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564048a05677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5640489f456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5640489f607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x564048ad1afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x564048ad1bdd] pthread_create.c:0(start_thread)[0x7fcb63c93ea5] /lib64/libc.so.6(clone+0x6d)[0x7fcb620348cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fcae800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:35:12 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:35:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:35:12 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:35:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:35:12 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:35:12 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:35:12 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:35:13 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:35:13 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:35:13 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136910715 2020-01-31 23:35:13 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:35:13 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:35:13 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:35:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:35:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:35:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:35:13 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:35:13 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136910724; transaction id 5903260 2020-01-31 23:35:13 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:35:13 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:35:13 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:35:13 0 [Note] Starting crash recovery... 2020-01-31 23:35:13 0 [Note] Crash recovery finished. 2020-01-31 23:35:13 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:35:13 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:35:13 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:35:13 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000015' at position 346 2020-01-31 23:35:13 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:35:13 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:35:13 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:35:13 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:35:13 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:35:13 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:35:13 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:35:13 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000015' at position 346 2020-01-31 23:35:14 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:35:14 2020-01-31 23:35:22 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000048, 542019) 2020-01-31 23:35:40 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:35:40 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:35:40 15 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:35:40 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:35:40 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:35:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f724c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f72a0078cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5576984fcf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x557697f933af] sigaction.c:0(__restore_rt)[0x7f72ba537630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x557697ec1151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x557697fff545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x557697db5ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x557697dabd4b] /usr/sbin/mysqld(+0x6ca792)[0x557697db6792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x557697dde6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557697de065d] /usr/sbin/mysqld(+0x65e3f3)[0x557697d4a3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x557697d49cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x557697dddb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557697de065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x557697de0761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x557697de1377] /usr/sbin/mysqld(+0x58fd84)[0x557697c7bd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x557697d88b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x557697da0b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x557697da0cd2] /usr/sbin/mysqld(+0x6b55e7)[0x557697da15e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x557697da1677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x557697d9056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x557697d9207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x557697e6dafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x557697e6dbdd] pthread_create.c:0(start_thread)[0x7f72ba52fea5] /lib64/libc.so.6(clone+0x6d)[0x7f72b88d08cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f724c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:35:46 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:35:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:35:46 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:35:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:35:46 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:35:46 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:35:46 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:35:46 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:35:46 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:35:46 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136913644 2020-01-31 23:35:46 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:35:46 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:35:46 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:35:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:35:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:35:46 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:35:46 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:35:46 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136913653; transaction id 5903272 2020-01-31 23:35:46 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:35:46 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:35:46 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:35:46 0 [Note] Starting crash recovery... 2020-01-31 23:35:46 0 [Note] Crash recovery finished. 2020-01-31 23:35:46 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:35:46 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:35:46 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:35:46 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000015' at position 226350 2020-01-31 23:35:46 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:35:46 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:35:46 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000015' at position 226350 2020-01-31 23:35:46 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:35:46 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:35:46 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:35:46 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:35:46 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:35:47 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:35:47 2020-01-31 23:36:15 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:36:17 14 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:36:17 14 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 23:36:18 15 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:36:18 15 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-01-31 23:36:18 17 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 23:36:18 17 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 23:36:20 19 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:36:20 19 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:36:23 21 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:36:23 21 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:36:23 22 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:36:23 22 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:36:23 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe7fc000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe86018acf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5606f032cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5606efdc33af] sigaction.c:0(__restore_rt)[0x7fe87b093630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5606efcf1151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5606efe2f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5606efbe5ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5606efbdbd4b] /usr/sbin/mysqld(+0x6ca792)[0x5606efbe6792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5606efc0e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5606efc1065d] /usr/sbin/mysqld(+0x65e3f3)[0x5606efb7a3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5606efb79cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5606efc0db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5606efc1065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5606efc10761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5606efc11377] /usr/sbin/mysqld(+0x58fd84)[0x5606efaabd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5606efbb8b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5606efbd0b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5606efbd0cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5606efbd15e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5606efbd1677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5606efbc056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5606efbc207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5606efc9dafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5606efc9dbdd] pthread_create.c:0(start_thread)[0x7fe87b08bea5] /lib64/libc.so.6(clone+0x6d)[0x7fe87942c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe7fc00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 23 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:36:29 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:36:29 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:36:29 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:36:29 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:36:29 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:36:29 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:36:29 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:36:29 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:36:29 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:36:29 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136917298 2020-01-31 23:36:29 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:36:29 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:36:29 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:36:29 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:36:29 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:36:29 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:36:29 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:36:30 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136917307; transaction id 5903283 2020-01-31 23:36:30 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:36:30 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:36:30 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:36:30 0 [Note] Starting crash recovery... 2020-01-31 23:36:30 0 [Note] Crash recovery finished. 2020-01-31 23:36:30 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:36:30 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:36:30 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:36:30 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000015' at position 315853 2020-01-31 23:36:30 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:36:30 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:36:30 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000015' at position 315853 2020-01-31 23:36:30 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:36:30 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:36:30 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:36:30 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:36:30 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:36:30 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:36:30 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:36:30 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:36:30 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:36:30 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:36:31 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:36:31 2020-01-31 23:36:41 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000050, 1454) 2020-01-31 23:36:55 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:37:06 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:37:06 16 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:37:06 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:37:06 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:37:06 17 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:37:06 17 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:37:06 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fec88000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fece4078cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55efc11a8f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55efc0c3f3af] sigaction.c:0(__restore_rt)[0x7fecfe768630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55efc0b6d151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55efc0cab545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55efc0a61ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55efc0a57d4b] /usr/sbin/mysqld(+0x6ca792)[0x55efc0a62792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55efc0a8a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55efc0a8c65d] /usr/sbin/mysqld(+0x65e3f3)[0x55efc09f63f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55efc09f5cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55efc0a89b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55efc0a8c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55efc0a8c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55efc0a8d377] /usr/sbin/mysqld(+0x58fd84)[0x55efc0927d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55efc0a34b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55efc0a4cb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55efc0a4ccd2] /usr/sbin/mysqld(+0x6b55e7)[0x55efc0a4d5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55efc0a4d677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55efc0a3c56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55efc0a3e07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55efc0b19afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55efc0b19bdd] pthread_create.c:0(start_thread)[0x7fecfe760ea5] /lib64/libc.so.6(clone+0x6d)[0x7fecfcb018cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fec8800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:37:13 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:37:13 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:37:13 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:37:13 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:37:13 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:37:13 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:37:13 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:37:13 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:37:13 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:37:13 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136920277 2020-01-31 23:37:13 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:37:13 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:37:13 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:37:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:37:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:37:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:37:13 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:37:13 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136920286; transaction id 5903294 2020-01-31 23:37:13 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:37:13 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:37:13 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:37:13 0 [Note] Starting crash recovery... 2020-01-31 23:37:13 0 [Note] Crash recovery finished. 2020-01-31 23:37:13 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:37:13 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:37:13 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:37:13 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000015' at position 720744 2020-01-31 23:37:13 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:37:13 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:37:13 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:37:13 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:37:13 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:37:13 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:37:13 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:37:13 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000015' at position 720744 2020-01-31 23:37:14 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:37:14 2020-01-31 23:37:40 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:37:42 13 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 23:37:42 13 [Warning] Checking table: './wpms/wp_options' 2020-01-31 23:37:56 18 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:37:56 18 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:37:59 19 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:37:59 19 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:37:59 20 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:37:59 20 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:37:59 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fae10000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fae68de6cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5614e0755f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5614e01ec3af] sigaction.c:0(__restore_rt)[0x7fae85f94630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5614e011a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5614e0258545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5614e000eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5614e0004d4b] /usr/sbin/mysqld(+0x6ca792)[0x5614e000f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5614e00376da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5614e003965d] /usr/sbin/mysqld(+0x65e3f3)[0x5614dffa33f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5614dffa2cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5614e0036b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5614e003965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5614e0039761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5614e003a377] /usr/sbin/mysqld(+0x58fd84)[0x5614dfed4d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5614dffe1b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5614dfff9b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5614dfff9cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5614dfffa5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5614dfffa677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5614dffe956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5614dffeb07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5614e00c6afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5614e00c6bdd] pthread_create.c:0(start_thread)[0x7fae85f8cea5] /lib64/libc.so.6(clone+0x6d)[0x7fae8432d8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fae1000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 21 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:38:05 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:38:05 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:38:05 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:38:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:38:05 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:38:05 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:38:05 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:38:05 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:38:05 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:38:05 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136923149 2020-01-31 23:38:05 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:38:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:38:05 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:38:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:38:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:38:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:38:05 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:38:05 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136923158; transaction id 5903305 2020-01-31 23:38:05 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:38:05 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:38:05 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:38:05 0 [Note] Starting crash recovery... 2020-01-31 23:38:05 0 [Note] Crash recovery finished. 2020-01-31 23:38:05 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:38:05 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:38:05 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:38:05 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000015' at position 849940 2020-01-31 23:38:05 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:38:05 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:38:05 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000015' at position 849940 2020-01-31 23:38:05 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:38:05 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:38:05 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:38:05 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:38:05 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:38:06 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:38:06 2020-01-31 23:38:07 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000052, 2513) 2020-01-31 23:38:30 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:38:30 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000015' at position 1008372 2020-01-31 23:38:30 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:38:31 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:38:31 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:38:31 14 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:38:31 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:38:31 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:38:31 12 [Warning] Aborted connection 12 to db: 'unconnected' user: 'replication' host: 'dbsvc-527.wichita.edu' (Got an error writing communication packets) 200131 23:38:31 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9644006bf8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f96b0d8ccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5579a88e6f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5579a837d3af] sigaction.c:0(__restore_rt)[0x7f96cbe29630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5579a82ab151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5579a83e9545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5579a819fccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5579a8195d4b] /usr/sbin/mysqld(+0x6ca792)[0x5579a81a0792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5579a81c86da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5579a81ca65d] /usr/sbin/mysqld(+0x65e3f3)[0x5579a81343f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5579a8133cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5579a81c7b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5579a81ca65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5579a81ca761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5579a81cb377] /usr/sbin/mysqld(+0x58fd84)[0x5579a8065d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5579a8172b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5579a818ab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5579a818acd2] /usr/sbin/mysqld(+0x6b55e7)[0x5579a818b5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5579a818b677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5579a817a56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5579a817c07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5579a8257afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5579a8257bdd] pthread_create.c:0(start_thread)[0x7f96cbe21ea5] /lib64/libc.so.6(clone+0x6d)[0x7f96ca1c28cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f9644010ea0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:38:37 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:38:37 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:38:37 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:38:37 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:38:37 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:38:37 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:38:37 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:38:37 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:38:37 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:38:37 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136925963 2020-01-31 23:38:37 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:38:37 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:38:37 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:38:37 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:38:37 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:38:37 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:38:37 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:38:37 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136925972; transaction id 5903316 2020-01-31 23:38:37 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:38:37 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:38:37 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:38:37 0 [Note] Starting crash recovery... 2020-01-31 23:38:37 0 [Note] Crash recovery finished. 2020-01-31 23:38:37 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:38:37 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:38:37 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:38:37 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000015' at position 1008372 2020-01-31 23:38:37 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:38:37 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000015' at position 1008372 2020-01-31 23:38:37 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:38:37 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:38:37 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:38:37 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:38:37 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:38:37 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:38:38 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:38:38 2020-01-31 23:39:06 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:39:10 14 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 23:39:10 14 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 23:39:16 17 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:39:16 17 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:39:29 25 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-01-31 23:39:29 25 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-01-31 23:39:36 32 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000054, 346) 2020-01-31 23:39:36 34 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 23:39:36 34 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 23:39:47 43 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:39:47 43 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:39:47 44 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:39:47 44 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:39:47 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fb7c4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fb83805acf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5611151eef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x561114c853af] sigaction.c:0(__restore_rt)[0x7fb83e186630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x561114bb3151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x561114cf1545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x561114aa7ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x561114a9dd4b] /usr/sbin/mysqld(+0x6ca792)[0x561114aa8792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x561114ad06da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x561114ad265d] /usr/sbin/mysqld(+0x65e3f3)[0x561114a3c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x561114a3bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x561114acfb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x561114ad265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x561114ad2761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x561114ad3377] /usr/sbin/mysqld(+0x58fd84)[0x56111496dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x561114a7ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x561114a92b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x561114a92cd2] /usr/sbin/mysqld(+0x6b55e7)[0x561114a935e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x561114a93677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x561114a8256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x561114a8407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x561114b5fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x561114b5fbdd] pthread_create.c:0(start_thread)[0x7fb83e17eea5] /lib64/libc.so.6(clone+0x6d)[0x7fb83c51f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fb7c400ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 45 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:39:53 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:39:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:39:53 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:39:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:39:53 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:39:53 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:39:53 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:39:53 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:39:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:39:53 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136928827 2020-01-31 23:39:53 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:39:53 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:39:53 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:39:53 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:39:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:39:53 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:39:53 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:39:53 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136928836; transaction id 5903327 2020-01-31 23:39:53 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:39:53 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:39:53 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:39:53 0 [Note] Starting crash recovery... 2020-01-31 23:39:53 0 [Note] Crash recovery finished. 2020-01-31 23:39:53 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:39:53 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:39:53 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:39:53 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000016' at position 44684 2020-01-31 23:39:53 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:39:53 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:39:53 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000016' at position 44684 2020-01-31 23:39:53 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:39:53 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:39:53 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:39:53 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:39:53 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:39:53 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:39:53 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:39:53 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:39:53 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:39:53 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:39:54 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:39:54 2020-01-31 23:40:00 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:40:00 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000016' at position 45802 2020-01-31 23:40:00 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:40:06 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000055, 402934) 2020-01-31 23:40:21 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:40:21 15 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:40:21 15 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:40:25 19 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:40:25 19 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:40:25 20 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:40:25 20 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:40:25 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f1540000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f154c132cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5567b49cff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5567b44663af] sigaction.c:0(__restore_rt)[0x7f15b6ebf630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5567b4394151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5567b44d2545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5567b4288ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5567b427ed4b] /usr/sbin/mysqld(+0x6ca792)[0x5567b4289792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5567b42b16da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5567b42b365d] /usr/sbin/mysqld(+0x65e3f3)[0x5567b421d3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5567b421ccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5567b42b0b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5567b42b365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5567b42b3761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5567b42b4377] /usr/sbin/mysqld(+0x58fd84)[0x5567b414ed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5567b425bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5567b4273b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5567b4273cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5567b42745e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5567b4274677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5567b426356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5567b426507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5567b4340afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5567b4340bdd] pthread_create.c:0(start_thread)[0x7f15b6eb7ea5] /lib64/libc.so.6(clone+0x6d)[0x7f15b52588cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f154000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 21 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:40:31 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:40:31 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:40:31 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:40:31 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:40:31 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:40:31 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:40:31 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:40:31 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:40:31 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:40:31 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136931641 2020-01-31 23:40:31 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:40:31 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:40:31 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:40:31 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:40:31 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:40:31 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:40:31 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136931650; transaction id 5903338 2020-01-31 23:40:31 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:40:31 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:40:31 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:40:31 0 [Note] Starting crash recovery... 2020-01-31 23:40:31 0 [Note] Crash recovery finished. 2020-01-31 23:40:31 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:40:31 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:40:31 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:40:31 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000016' at position 45802 2020-01-31 23:40:31 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:40:31 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:40:31 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000016' at position 45802 2020-01-31 23:40:31 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:40:31 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:40:31 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:40:31 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:40:31 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:40:32 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:40:32 2020-01-31 23:40:40 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:40:40 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000017' at position 120862 2020-01-31 23:40:40 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:40:46 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000056, 46744) 2020-01-31 23:40:56 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:40:57 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:40:57 14 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:41:20 32 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:41:20 32 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:41:20 33 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:41:20 33 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:41:20 33 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:41:20 33 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:41:21 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7feb0c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7feb6413fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b890c27f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55b8906be3af] sigaction.c:0(__restore_rt)[0x7feb7f231630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55b8905ec151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55b89072a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55b8904e0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55b8904d6d4b] /usr/sbin/mysqld(+0x6ca792)[0x55b8904e1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55b8905096da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b89050b65d] /usr/sbin/mysqld(+0x65e3f3)[0x55b8904753f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55b890474cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55b890508b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b89050b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55b89050b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55b89050c377] /usr/sbin/mysqld(+0x58fd84)[0x55b8903a6d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55b8904b3b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55b8904cbb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55b8904cbcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55b8904cc5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55b8904cc677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55b8904bb56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55b8904bd07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55b890598afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55b890598bdd] pthread_create.c:0(start_thread)[0x7feb7f229ea5] /lib64/libc.so.6(clone+0x6d)[0x7feb7d5ca8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7feb0c010a30): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 34 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:41:27 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:41:27 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:41:27 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:41:27 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:41:27 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:41:27 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:41:27 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:41:27 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:41:27 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:41:27 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136934504 2020-01-31 23:41:27 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:41:27 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:41:27 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:41:27 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:41:27 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:41:27 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:41:27 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:41:27 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136934513; transaction id 5903349 2020-01-31 23:41:27 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:41:27 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:41:27 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:41:27 0 [Note] Starting crash recovery... 2020-01-31 23:41:27 0 [Note] Crash recovery finished. 2020-01-31 23:41:27 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:41:27 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:41:27 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:41:27 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000017' at position 120862 2020-01-31 23:41:27 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:41:27 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:41:27 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000017' at position 120862 2020-01-31 23:41:27 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:41:27 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:41:27 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:41:27 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:41:27 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:41:27 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:41:27 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:41:27 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:41:27 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:41:27 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:41:28 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:41:28 2020-01-31 23:41:34 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:41:34 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000018' at position 1464 2020-01-31 23:41:34 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:41:39 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000057, 326697) 2020-01-31 23:41:57 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:41:57 18 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:41:57 18 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:42:34 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000018' at position 1464 2020-01-31 23:42:47 58 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:42:47 58 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:42:47 59 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:42:47 59 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:42:48 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc880000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc88dffecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55befa110f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55bef9ba73af] sigaction.c:0(__restore_rt)[0x7fc8ef18a630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55bef9ad5151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55bef9c13545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55bef99c9ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55bef99bfd4b] /usr/sbin/mysqld(+0x6ca792)[0x55bef99ca792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55bef99f26da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55bef99f465d] /usr/sbin/mysqld(+0x65e3f3)[0x55bef995e3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55bef995dcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55bef99f1b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55bef99f465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55bef99f4761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55bef99f5377] /usr/sbin/mysqld(+0x58fd84)[0x55bef988fd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55bef999cb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55bef99b4b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55bef99b4cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55bef99b55e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55bef99b5677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55bef99a456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55bef99a607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55bef9a81afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55bef9a81bdd] pthread_create.c:0(start_thread)[0x7fc8ef182ea5] /lib64/libc.so.6(clone+0x6d)[0x7fc8ed5238cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc88000feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 60 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:42:54 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:42:54 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:42:54 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:42:54 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:42:54 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:42:54 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:42:54 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:42:54 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:42:54 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:42:55 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136940745 2020-01-31 23:42:55 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:42:55 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:42:55 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:42:55 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:42:55 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:42:55 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:42:55 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:42:55 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136940754; transaction id 5903365 2020-01-31 23:42:55 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:42:55 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:42:55 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:42:55 0 [Note] Starting crash recovery... 2020-01-31 23:42:55 0 [Note] Crash recovery finished. 2020-01-31 23:42:55 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:42:55 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:42:55 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:42:55 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000019' at position 346 2020-01-31 23:42:55 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:42:55 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:42:55 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:42:55 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:42:55 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:42:55 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:42:55 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:42:55 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:42:55 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:42:55 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:42:55 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:42:55 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:42:55 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000019' at position 346 2020-01-31 23:42:57 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:42:57 2020-01-31 23:43:22 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:43:32 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:43:32 14 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:43:43 19 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-01-31 23:43:43 19 [Warning] Checking table: './wpcomm303/wp_options' 2020-01-31 23:43:48 22 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:43:48 22 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:43:48 23 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:43:48 23 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:43:49 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9c6c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f9cc4cf6cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560ad6b40f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560ad65d73af] sigaction.c:0(__restore_rt)[0x7f9cdfe14630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x560ad6505151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x560ad6643545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x560ad63f9ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x560ad63efd4b] /usr/sbin/mysqld(+0x6ca792)[0x560ad63fa792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x560ad64226da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560ad642465d] /usr/sbin/mysqld(+0x65e3f3)[0x560ad638e3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x560ad638dcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x560ad6421b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560ad642465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x560ad6424761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x560ad6425377] /usr/sbin/mysqld(+0x58fd84)[0x560ad62bfd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x560ad63ccb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x560ad63e4b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x560ad63e4cd2] /usr/sbin/mysqld(+0x6b55e7)[0x560ad63e55e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x560ad63e5677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x560ad63d456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x560ad63d607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x560ad64b1afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560ad64b1bdd] pthread_create.c:0(start_thread)[0x7f9cdfe0cea5] /lib64/libc.so.6(clone+0x6d)[0x7f9cde1ad8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f9c6c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_extern2020-01-31 23:43:49 25 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000058, 258451) al_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 24 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:43:55 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:43:55 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:43:55 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:43:55 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:43:55 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:43:55 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:43:55 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:43:55 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:43:55 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:43:55 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136943762 2020-01-31 23:43:55 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:43:55 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:43:55 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:43:55 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:43:55 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:43:55 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:43:55 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:43:55 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136943771; transaction id 5903376 2020-01-31 23:43:55 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:43:55 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:43:55 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:43:55 0 [Note] Starting crash recovery... 2020-01-31 23:43:55 0 [Note] Crash recovery finished. 2020-01-31 23:43:55 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:43:55 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:43:55 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:43:55 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000019' at position 154133 2020-01-31 23:43:55 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:43:55 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:43:55 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000019' at position 154133 2020-01-31 23:43:55 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:43:55 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:43:55 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:43:55 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:43:55 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:43:56 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:43:56 2020-01-31 23:43:58 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:43:58 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000019' at position 155251 2020-01-31 23:43:58 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:44:04 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000059, 49156) 2020-01-31 23:44:22 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:44:27 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:44:27 16 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:44:27 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:44:27 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:44:27 17 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:44:27 17 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:44:27 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f3164000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f31b8078cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c9a3ed5f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c9a396c3af] sigaction.c:0(__restore_rt)[0x7f31d25b4630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c9a389a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c9a39d8545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c9a378eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c9a3784d4b] /usr/sbin/mysqld(+0x6ca792)[0x55c9a378f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c9a37b76da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c9a37b965d] /usr/sbin/mysqld(+0x65e3f3)[0x55c9a37233f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c9a3722cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c9a37b6b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c9a37b965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c9a37b9761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c9a37ba377] /usr/sbin/mysqld(+0x58fd84)[0x55c9a3654d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c9a3761b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c9a3779b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c9a3779cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c9a377a5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c9a377a677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c9a376956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c9a376b07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c9a3846afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c9a3846bdd] pthread_create.c:0(start_thread)[0x7f31d25acea5] /lib64/libc.so.6(clone+0x6d)[0x7f31d094d8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f316400ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:44:34 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:44:34 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:44:34 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:44:34 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:44:34 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:44:34 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:44:34 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:44:34 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:44:34 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:44:34 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136947456 2020-01-31 23:44:34 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:44:34 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:44:34 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:44:34 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:44:34 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:44:34 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:44:34 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:44:34 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136947465; transaction id 5903387 2020-01-31 23:44:34 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:44:34 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:44:34 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:44:34 0 [Note] Starting crash recovery... 2020-01-31 23:44:34 0 [Note] Crash recovery finished. 2020-01-31 23:44:34 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:44:34 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:44:34 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:44:34 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000019' at position 155251 2020-01-31 23:44:34 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:44:34 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:44:34 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000019' at position 155251 2020-01-31 23:44:34 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:44:34 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:44:34 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:44:34 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:44:34 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:44:35 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:44:35 2020-01-31 23:44:59 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:44:59 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000020' at position 1464 2020-01-31 23:44:59 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:45:02 15 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:45:04 18 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 23:45:04 18 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 23:45:04 19 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000060, 1464) 2020-01-31 23:45:05 17 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:45:05 17 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 23:45:24 27 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 23:45:24 27 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 23:45:28 31 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 23:45:28 31 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 23:45:29 33 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:45:29 33 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:45:40 39 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:45:40 39 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:45:40 40 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:45:40 40 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:45:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9804000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f985885fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x557b5c950f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x557b5c3e73af] sigaction.c:0(__restore_rt)[0x7f9873a4e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x557b5c315151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x557b5c453545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x557b5c209ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x557b5c1ffd4b] /usr/sbin/mysqld(+0x6ca792)[0x557b5c20a792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x557b5c2326da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557b5c23465d] /usr/sbin/mysqld(+0x65e3f3)[0x557b5c19e3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x557b5c19dcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x557b5c231b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557b5c23465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x557b5c234761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x557b5c235377] /usr/sbin/mysqld(+0x58fd84)[0x557b5c0cfd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x557b5c1dcb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x557b5c1f4b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x557b5c1f4cd2] /usr/sbin/mysqld(+0x6b55e7)[0x557b5c1f55e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x557b5c1f5677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x557b5c1e456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x557b5c1e607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x557b5c2c1afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x557b5c2c1bdd] pthread_create.c:0(start_thread)[0x7f9873a46ea5] /lib64/libc.so.6(clone+0x6d)[0x7f9871de78cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f980400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 42 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:45:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:45:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:45:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:45:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:45:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:45:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:45:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:45:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:45:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:45:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136950285 2020-01-31 23:45:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:45:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:45:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:45:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:45:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:45:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:45:47 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:45:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136950294; transaction id 5903398 2020-01-31 23:45:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:45:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:45:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:45:47 0 [Note] Starting crash recovery... 2020-01-31 23:45:47 0 [Note] Crash recovery finished. 2020-01-31 23:45:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:45:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:45:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:45:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000020' at position 1464 2020-01-31 23:45:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:45:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:45:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000020' at position 1464 2020-01-31 23:45:47 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:45:47 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:45:47 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:45:47 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:45:47 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:45:47 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:45:47 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:45:47 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:45:47 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:45:47 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:45:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:45:48 2020-01-31 23:45:48 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:45:48 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000021' at position 1464 2020-01-31 23:45:48 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:45:54 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000061, 375566) 2020-01-31 23:46:13 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:46:17 17 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:46:17 17 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 23:46:18 20 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 23:46:18 20 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 23:46:24 23 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:46:24 23 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:46:24 24 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:46:24 24 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:46:25 25 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:46:25 25 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:46:25 25 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:46:25 25 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:46:25 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fb664000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fb6bc904cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5588eb24af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5588eace13af] sigaction.c:0(__restore_rt)[0x7fb6d9ac5630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5588eac0f151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5588ead4d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5588eab03ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5588eaaf9d4b] /usr/sbin/mysqld(+0x6ca792)[0x5588eab04792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5588eab2c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5588eab2e65d] /usr/sbin/mysqld(+0x65e3f3)[0x5588eaa983f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5588eaa97cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5588eab2bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5588eab2e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5588eab2e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5588eab2f377] /usr/sbin/mysqld(+0x58fd84)[0x5588ea9c9d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5588eaad6b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5588eaaeeb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5588eaaeecd2] /usr/sbin/mysqld(+0x6b55e7)[0x5588eaaef5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5588eaaef677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5588eaade56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5588eaae007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5588eabbbafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5588eabbbbdd] pthread_create.c:0(start_thread)[0x7fb6d9abdea5] /lib64/libc.so.6(clone+0x6d)[0x7fb6d7e5e8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fb66400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 26 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:46:31 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:46:31 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:46:31 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:46:31 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:46:31 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:46:31 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:46:31 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:46:31 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:46:31 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:46:31 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136953099 2020-01-31 23:46:31 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:46:31 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:46:31 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:46:31 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:46:31 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:46:31 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:46:31 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:46:31 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136953108; transaction id 5903409 2020-01-31 23:46:31 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:46:31 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:46:31 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:46:31 0 [Note] Starting crash recovery... 2020-01-31 23:46:31 0 [Note] Crash recovery finished. 2020-01-31 23:46:31 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:46:31 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:46:31 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:46:31 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000021' at position 1464 2020-01-31 23:46:31 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:46:31 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:46:31 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:46:31 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:46:31 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:46:31 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:46:31 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:46:31 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:46:31 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:46:31 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:46:31 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:46:31 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:46:31 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:46:32 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:46:32 2020-01-31 23:46:36 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000062, 34508) 2020-01-31 23:46:58 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:47:05 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:47:05 17 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:47:05 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:47:05 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:47:05 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f5244000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f52a413fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563cebddff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x563ceb8763af] sigaction.c:0(__restore_rt)[0x7f52bf000630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x563ceb7a4151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x563ceb8e2545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x563ceb698ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x563ceb68ed4b] /usr/sbin/mysqld(+0x6ca792)[0x563ceb699792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x563ceb6c16da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563ceb6c365d] /usr/sbin/mysqld(+0x65e3f3)[0x563ceb62d3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x563ceb62ccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x563ceb6c0b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563ceb6c365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x563ceb6c3761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x563ceb6c4377] /usr/sbin/mysqld(+0x58fd84)[0x563ceb55ed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x563ceb66bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x563ceb683b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x563ceb683cd2] /usr/sbin/mysqld(+0x6b55e7)[0x563ceb6845e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x563ceb684677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x563ceb67356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x563ceb67507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x563ceb750afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x563ceb750bdd] pthread_create.c:0(start_thread)[0x7f52beff8ea5] /lib64/libc.so.6(clone+0x6d)[0x7f52bd3998cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f5244010a30): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:47:11 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:47:11 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:47:11 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:47:11 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:47:11 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:47:11 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:47:11 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:47:11 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:47:11 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:47:11 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136953505 2020-01-31 23:47:11 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:47:11 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:47:11 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:47:11 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:47:11 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:47:11 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:47:11 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:47:11 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136953514; transaction id 5903414 2020-01-31 23:47:11 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:47:11 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:47:11 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:47:11 0 [Note] Starting crash recovery... 2020-01-31 23:47:11 0 [Note] Crash recovery finished. 2020-01-31 23:47:11 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:47:11 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:47:11 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:47:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000021' at position 1464 2020-01-31 23:47:11 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:47:11 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:47:11 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000021' at position 1464 2020-01-31 23:47:11 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:47:11 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:47:11 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:47:11 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:47:11 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:47:11 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:47:11 2020-01-31 23:47:13 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:47:13 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000023' at position 1970 2020-01-31 23:47:13 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:47:18 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000063, 1464) 2020-01-31 23:47:38 19 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:47:45 27 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 23:47:45 27 [Warning] Checking table: './wpms/wp_options' 2020-01-31 23:47:49 33 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-01-31 23:47:49 33 [Warning] Checking table: './itstraining/wp_options' 2020-01-31 23:47:50 34 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:47:50 34 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:47:50 35 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:47:50 35 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:47:50 35 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:47:50 35 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:47:50 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fb8d8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fb8c80c5cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x562e8a832f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x562e8a2c93af] sigaction.c:0(__restore_rt)[0x7fb8e3525630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x562e8a1f7151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x562e8a335545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x562e8a0ebccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x562e8a0e1d4b] /usr/sbin/mysqld(+0x6ca792)[0x562e8a0ec792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x562e8a1146da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x562e8a11665d] /usr/sbin/mysqld(+0x65e3f3)[0x562e8a0803f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x562e8a07fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x562e8a113b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x562e8a11665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x562e8a116761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x562e8a117377] /usr/sbin/mysqld(+0x58fd84)[0x562e89fb1d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x562e8a0beb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x562e8a0d6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x562e8a0d6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x562e8a0d75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x562e8a0d7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x562e8a0c656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x562e8a0c807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x562e8a1a3afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x562e8a1a3bdd] pthread_create.c:0(start_thread)[0x7fb8e351dea5] /lib64/libc.so.6(clone+0x6d)[0x7fb8e18be8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fb8d800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 36 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:47:56 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:47:56 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:47:56 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:47:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:47:56 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:47:56 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:47:56 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:47:56 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:47:56 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:47:56 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136953832 2020-01-31 23:47:56 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:47:56 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:47:56 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:47:56 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:47:56 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:47:56 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:47:56 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:47:56 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136953841; transaction id 5903419 2020-01-31 23:47:56 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:47:56 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:47:56 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:47:56 0 [Note] Starting crash recovery... 2020-01-31 23:47:56 0 [Note] Crash recovery finished. 2020-01-31 23:47:56 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:47:56 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:47:56 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:47:56 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000023' at position 1970 2020-01-31 23:47:56 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:47:56 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:47:56 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000023' at position 1970 2020-01-31 23:47:56 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:47:56 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:47:56 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:47:56 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:47:56 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:47:56 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:47:56 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:47:56 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:47:56 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:47:56 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:47:58 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:47:58 2020-01-31 23:48:16 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:48:16 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000024' at position 90708 2020-01-31 23:48:16 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:48:22 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000064, 77450) 2020-01-31 23:48:24 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:48:24 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:48:24 14 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:48:25 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:48:25 15 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:48:25 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:48:25 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:48:26 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff1ec000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff24462acf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55896529ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x558964d353af] sigaction.c:0(__restore_rt)[0x7ff2617a6630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x558964c63151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x558964da1545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x558964b57ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x558964b4dd4b] /usr/sbin/mysqld(+0x6ca792)[0x558964b58792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x558964b806da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x558964b8265d] /usr/sbin/mysqld(+0x65e3f3)[0x558964aec3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x558964aebcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x558964b7fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x558964b8265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x558964b82761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x558964b83377] /usr/sbin/mysqld(+0x58fd84)[0x558964a1dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x558964b2ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x558964b42b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x558964b42cd2] /usr/sbin/mysqld(+0x6b55e7)[0x558964b435e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x558964b43677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x558964b3256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x558964b3407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x558964c0fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x558964c0fbdd] pthread_create.c:0(start_thread)[0x7ff26179eea5] /lib64/libc.so.6(clone+0x6d)[0x7ff25fb3f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff1ec00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:48:32 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:48:32 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:48:32 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:48:32 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:48:32 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:48:32 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:48:32 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:48:32 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:48:32 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:48:32 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136956688 2020-01-31 23:48:32 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:48:32 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:48:32 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:48:32 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:48:32 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:48:32 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:48:32 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:48:32 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136956697; transaction id 5903431 2020-01-31 23:48:32 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:48:32 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:48:32 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:48:32 0 [Note] Starting crash recovery... 2020-01-31 23:48:32 0 [Note] Crash recovery finished. 2020-01-31 23:48:32 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:48:32 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:48:32 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:48:32 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000024' at position 90708 2020-01-31 23:48:32 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:48:32 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:48:32 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000024' at position 90708 2020-01-31 23:48:32 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:48:32 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:48:32 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:48:32 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:48:32 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:48:33 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:48:33 2020-01-31 23:48:59 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:49:05 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:49:05 15 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:49:05 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:49:05 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:49:05 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fcb50000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fcbb51c6cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b1fd494f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55b1fcf2b3af] sigaction.c:0(__restore_rt)[0x7fcbd02f3630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55b1fce59151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55b1fcf97545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55b1fcd4dccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55b1fcd43d4b] /usr/sbin/mysqld(+0x6ca792)[0x55b1fcd4e792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55b1fcd766da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b1fcd7865d] /usr/sbin/mysqld(+0x65e3f3)[0x55b1fcce23f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55b1fcce1cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55b1fcd75b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b1fcd7865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55b1fcd78761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55b1fcd79377] /usr/sbin/mysqld(+0x58fd84)[0x55b1fcc13d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55b1fcd20b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55b1fcd38b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55b1fcd38cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55b1fcd395e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55b1fcd39677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55b1fcd2856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55b1fcd2a07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55b1fce05afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55b1fce05bdd] pthread_create.c:0(start_thread)[0x7fcbd02ebea5] /lib64/libc.so.6(clone+0x6d)[0x7fcbce68c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fcb5000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:49:11 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:49:11 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:49:11 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:49:11 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:49:11 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:49:11 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:49:11 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:49:11 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:49:11 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:49:11 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136960342 2020-01-31 23:49:11 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:49:11 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:49:11 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:49:11 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:49:11 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:49:11 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:49:11 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:49:11 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136960351; transaction id 5903442 2020-01-31 23:49:11 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:49:11 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:49:11 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:49:11 0 [Note] Starting crash recovery... 2020-01-31 23:49:11 0 [Note] Crash recovery finished. 2020-01-31 23:49:11 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:49:11 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:49:11 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:49:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000025' at position 146364 2020-01-31 23:49:11 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:49:11 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:49:11 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000025' at position 146364 2020-01-31 23:49:11 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:49:11 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:49:11 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:49:11 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:49:11 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:49:12 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:49:12 2020-01-31 23:49:13 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:49:13 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000025' at position 192001 2020-01-31 23:49:13 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:49:19 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000065, 46289) 2020-01-31 23:49:39 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:49:39 15 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-01-31 23:49:39 15 [Warning] Checking table: './shockeripe/wp_options' 2020-01-31 23:49:41 18 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:49:41 18 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:49:41 19 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:49:41 19 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:49:41 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fa4600009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fa4bcf13cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x564b6c1d1f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x564b6bc683af] sigaction.c:0(__restore_rt)[0x7fa4d808e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564b6bb96151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564b6bcd4545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564b6ba8accf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x564b6ba80d4b] /usr/sbin/mysqld(+0x6ca792)[0x564b6ba8b792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x564b6bab36da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564b6bab565d] /usr/sbin/mysqld(+0x65e3f3)[0x564b6ba1f3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x564b6ba1ecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x564b6bab2b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564b6bab565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x564b6bab5761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x564b6bab6377] /usr/sbin/mysqld(+0x58fd84)[0x564b6b950d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x564b6ba5db6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564b6ba75b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564b6ba75cd2] /usr/sbin/mysqld(+0x6b55e7)[0x564b6ba765e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564b6ba76677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x564b6ba6556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x564b6ba6707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x564b6bb42afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x564b6bb42bdd] pthread_create.c:0(start_thread)[0x7fa4d8086ea5] /lib64/libc.so.6(clone+0x6d)[0x7fa4d64278cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fa46000fdd0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 20 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:49:47 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:49:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:49:47 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:49:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:49:47 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:49:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:49:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:49:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:49:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:49:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136963204 2020-01-31 23:49:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:49:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:49:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:49:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:49:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:49:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:49:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136963213; transaction id 5903453 2020-01-31 23:49:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:49:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:49:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:49:47 0 [Note] Starting crash recovery... 2020-01-31 23:49:47 0 [Note] Crash recovery finished. 2020-01-31 23:49:47 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:49:47 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:49:47 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:49:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000025' at position 192001 2020-01-31 23:49:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:49:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:49:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000025' at position 192001 2020-01-31 23:49:47 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:49:47 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:49:47 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:49:47 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:49:47 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:49:48 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:49:48 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000026' at position 27730 2020-01-31 23:49:48 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:49:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:49:48 2020-01-31 23:49:54 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000067, 58690) 2020-01-31 23:50:14 15 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:50:14 16 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:50:14 16 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:50:17 20 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 23:50:17 20 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 23:50:29 25 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:50:29 25 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:50:29 26 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:50:29 26 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:50:29 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f1978000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f19cc066cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d4fa8b8f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55d4fa34f3af] sigaction.c:0(__restore_rt)[0x7f19e6304630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55d4fa27d151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55d4fa3bb545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55d4fa171ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55d4fa167d4b] /usr/sbin/mysqld(+0x6ca792)[0x55d4fa172792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55d4fa19a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d4fa19c65d] /usr/sbin/mysqld(+0x65e3f3)[0x55d4fa1063f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55d4fa105cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55d4fa199b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d4fa19c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55d4fa19c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55d4fa19d377] /usr/sbin/mysqld(+0x58fd84)[0x55d4fa037d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55d4fa144b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55d4fa15cb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55d4fa15ccd2] /usr/sbin/mysqld(+0x6b55e7)[0x55d4fa15d5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55d4fa15d677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55d4fa14c56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55d4fa14e07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55d4fa229afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d4fa229bdd] pthread_create.c:0(start_thread)[0x7f19e62fcea5] /lib64/libc.so.6(clone+0x6d)[0x7f19e469d8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f197800fe80): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 27 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:50:35 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:50:35 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:50:35 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:50:35 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:50:35 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:50:35 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:50:35 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:50:35 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:50:35 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:50:35 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136966184 2020-01-31 23:50:35 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:50:35 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:50:35 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:50:35 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:50:35 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:50:35 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:50:35 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:50:35 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136966193; transaction id 5903464 2020-01-31 23:50:35 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:50:35 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:50:35 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:50:35 0 [Note] Starting crash recovery... 2020-01-31 23:50:35 0 [Note] Crash recovery finished. 2020-01-31 23:50:35 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:50:35 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:50:35 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:50:35 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000026' at position 27730 2020-01-31 23:50:35 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:50:35 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:50:35 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:50:35 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:50:35 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:50:35 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:50:35 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:50:35 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:50:35 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:50:35 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:50:35 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:50:35 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:50:35 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000026' at position 27730 2020-01-31 23:50:36 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:50:36 2020-01-31 23:50:40 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:50:40 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000027' at position 1454 2020-01-31 23:50:40 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:50:46 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000068, 222882) 2020-01-31 23:51:04 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:51:06 16 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:51:06 16 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:51:09 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:51:09 17 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:51:09 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:51:09 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:51:09 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f92a0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f92bff68cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e4db3e0f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e4dae773af] sigaction.c:0(__restore_rt)[0x7f9321138630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55e4dada5151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55e4daee3545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55e4dac99ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55e4dac8fd4b] /usr/sbin/mysqld(+0x6ca792)[0x55e4dac9a792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55e4dacc26da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e4dacc465d] /usr/sbin/mysqld(+0x65e3f3)[0x55e4dac2e3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55e4dac2dcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55e4dacc1b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e4dacc465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55e4dacc4761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55e4dacc5377] /usr/sbin/mysqld(+0x58fd84)[0x55e4dab5fd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55e4dac6cb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55e4dac84b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55e4dac84cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55e4dac855e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55e4dac85677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55e4dac7456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55e4dac7607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55e4dad51afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e4dad51bdd] pthread_create.c:0(start_thread)[0x7f9321130ea5] /lib64/libc.so.6(clone+0x6d)[0x7f931f4d18cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f92a000fe70): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:51:15 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:51:15 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:51:15 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:51:15 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:51:15 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:51:15 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:51:15 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:51:15 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:51:15 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:51:15 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136969057 2020-01-31 23:51:15 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:51:15 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:51:15 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:51:15 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:51:15 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:51:15 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:51:15 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:51:15 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136969066; transaction id 5903475 2020-01-31 23:51:15 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:51:15 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:51:15 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:51:15 0 [Note] Starting crash recovery... 2020-01-31 23:51:15 0 [Note] Crash recovery finished. 2020-01-31 23:51:15 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:51:15 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:51:15 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:51:15 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000027' at position 1454 2020-01-31 23:51:15 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:51:15 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:51:15 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000027' at position 1454 2020-01-31 23:51:15 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:51:15 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:51:15 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:51:15 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:51:15 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:51:15 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:51:15 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:51:15 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:51:15 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:51:15 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:51:16 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:51:16 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000028' at position 3655 2020-01-31 23:51:16 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:51:16 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:51:16 2020-01-31 23:51:21 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000069, 88651) 2020-01-31 23:51:45 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:51:45 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:51:45 14 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:51:57 34 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-01-31 23:51:57 34 [Warning] Checking table: './wpms/wp_options' 2020-01-31 23:51:59 36 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:51:59 36 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:51:59 37 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:51:59 37 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:51:59 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f663c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f665052acf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563a15de7f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x563a1587e3af] sigaction.c:0(__restore_rt)[0x7f66b28b3630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x563a157ac151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x563a158ea545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x563a156a0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x563a15696d4b] /usr/sbin/mysqld(+0x6ca792)[0x563a156a1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x563a156c96da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563a156cb65d] /usr/sbin/mysqld(+0x65e3f3)[0x563a156353f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x563a15634cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x563a156c8b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563a156cb65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x563a156cb761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x563a156cc377] /usr/sbin/mysqld(+0x58fd84)[0x563a15566d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x563a15673b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x563a1568bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x563a1568bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x563a1568c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x563a1568c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x563a1567b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x563a1567d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x563a15758afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x563a15758bdd] pthread_create.c:0(start_thread)[0x7f66b28abea5] /lib64/libc.so.6(clone+0x6d)[0x7f66b0c4c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f663c00feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 38 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:52:05 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:52:05 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:52:05 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:52:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:52:05 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:52:05 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:52:05 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:52:05 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:52:05 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:52:05 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136972711 2020-01-31 23:52:05 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:52:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:52:05 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:52:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:52:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:52:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:52:05 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:52:06 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136972720; transaction id 5903486 2020-01-31 23:52:06 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:52:06 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:52:06 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:52:06 0 [Note] Starting crash recovery... 2020-01-31 23:52:06 0 [Note] Crash recovery finished. 2020-01-31 23:52:06 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:52:06 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:52:06 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:52:06 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000028' at position 3655 2020-01-31 23:52:06 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:52:06 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:52:06 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000028' at position 3655 2020-01-31 23:52:06 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:52:06 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:52:06 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:52:06 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:52:06 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:52:07 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:52:07 2020-01-31 23:52:13 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:52:13 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000029' at position 45870 2020-01-31 23:52:13 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:52:18 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000070, 380223) 2020-01-31 23:52:35 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:52:38 15 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:52:38 15 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:52:42 20 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-01-31 23:52:42 20 [Warning] Checking table: './wpstart/wp_options' 2020-01-31 23:52:48 24 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:52:48 24 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:52:48 25 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:52:48 25 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:52:49 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff35c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff3b88f5cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b2b2dbdf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55b2b28543af] sigaction.c:0(__restore_rt)[0x7ff3d3a9e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55b2b2782151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55b2b28c0545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55b2b2676ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55b2b266cd4b] /usr/sbin/mysqld(+0x6ca792)[0x55b2b2677792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55b2b269f6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b2b26a165d] /usr/sbin/mysqld(+0x65e3f3)[0x55b2b260b3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55b2b260acb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55b2b269eb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b2b26a165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55b2b26a1761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55b2b26a2377] /usr/sbin/mysqld(+0x58fd84)[0x55b2b253cd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55b2b2649b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55b2b2661b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55b2b2661cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55b2b26625e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55b2b2662677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55b2b265156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55b2b265307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55b2b272eafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55b2b272ebdd] pthread_create.c:0(start_thread)[0x7ff3d3a96ea5] /lib64/libc.so.6(clone+0x6d)[0x7ff3d1e378cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff35c00feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 26 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:52:55 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:52:55 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:52:55 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:52:55 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:52:55 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:52:55 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:52:55 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:52:55 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:52:55 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:52:55 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136975540 2020-01-31 23:52:55 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:52:55 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:52:55 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:52:55 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:52:55 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:52:55 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:52:55 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:52:55 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136975549; transaction id 5903497 2020-01-31 23:52:55 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:52:55 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:52:55 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:52:55 0 [Note] Starting crash recovery... 2020-01-31 23:52:55 0 [Note] Crash recovery finished. 2020-01-31 23:52:55 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:52:55 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:52:55 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:52:55 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000029' at position 45870 2020-01-31 23:52:55 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:52:55 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:52:55 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000029' at position 45870 2020-01-31 23:52:55 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:52:55 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:52:55 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:52:55 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:52:55 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:52:55 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:52:55 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:52:55 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:52:55 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:52:55 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:52:56 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:52:56 2020-01-31 23:53:10 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:53:10 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000030' at position 142080 2020-01-31 23:53:10 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:53:16 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000071, 330759) 2020-01-31 23:53:20 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:53:21 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:53:21 14 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:53:21 15 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-01-31 23:53:21 15 [Warning] Checking table: './wpcomm305/comm305_options' 2020-01-31 23:53:21 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:53:21 17 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:53:21 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:53:21 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:53:22 18 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:53:22 18 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:53:22 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc6e0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc69cfb1cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560ea6399f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560ea5e303af] sigaction.c:0(__restore_rt)[0x7fc6ff796630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x560ea5d5e151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x560ea5e9c545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x560ea5c52ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x560ea5c48d4b] /usr/sbin/mysqld(+0x6ca792)[0x560ea5c53792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x560ea5c7b6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560ea5c7d65d] /usr/sbin/mysqld(+0x65e3f3)[0x560ea5be73f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x560ea5be6cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x560ea5c7ab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560ea5c7d65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x560ea5c7d761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x560ea5c7e377] /usr/sbin/mysqld(+0x58fd84)[0x560ea5b18d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x560ea5c25b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x560ea5c3db16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x560ea5c3dcd2] /usr/sbin/mysqld(+0x6b55e7)[0x560ea5c3e5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x560ea5c3e677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x560ea5c2d56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x560ea5c2f07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x560ea5d0aafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560ea5d0abdd] pthread_create.c:0(start_thread)[0x7fc6ff78eea5] /lib64/libc.so.6(clone+0x6d)[0x7fc6fdb2f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc6e000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:53:27 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:53:27 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:53:27 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:53:27 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:53:27 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:53:27 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:53:27 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:53:27 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:53:27 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:53:27 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136978380 2020-01-31 23:53:27 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:53:28 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:53:28 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:53:28 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:53:28 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:53:28 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:53:28 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:53:28 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136978389; transaction id 5903508 2020-01-31 23:53:28 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:53:28 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:53:28 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:53:28 0 [Note] Starting crash recovery... 2020-01-31 23:53:28 0 [Note] Crash recovery finished. 2020-01-31 23:53:28 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:53:28 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:53:28 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:53:28 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000030' at position 142080 2020-01-31 23:53:28 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:53:28 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:53:28 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000030' at position 142080 2020-01-31 23:53:28 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:53:28 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:53:28 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:53:28 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:53:28 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:53:29 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:53:29 2020-01-31 23:53:47 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:53:47 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000031' at position 140325 2020-01-31 23:53:47 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:53:53 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000072, 43666) 2020-01-31 23:53:55 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:53:55 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:53:55 14 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:53:55 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:53:55 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:53:56 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7effb4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7effb809ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x559bf3003f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x559bf2a9a3af] sigaction.c:0(__restore_rt)[0x7f0024aa7630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x559bf29c8151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x559bf2b06545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x559bf28bcccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x559bf28b2d4b] /usr/sbin/mysqld(+0x6ca792)[0x559bf28bd792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x559bf28e56da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559bf28e765d] /usr/sbin/mysqld(+0x65e3f3)[0x559bf28513f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x559bf2850cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x559bf28e4b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559bf28e765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x559bf28e7761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x559bf28e8377] /usr/sbin/mysqld(+0x58fd84)[0x559bf2782d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x559bf288fb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x559bf28a7b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x559bf28a7cd2] /usr/sbin/mysqld(+0x6b55e7)[0x559bf28a85e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x559bf28a8677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x559bf289756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x559bf289907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x559bf2974afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x559bf2974bdd] pthread_create.c:0(start_thread)[0x7f0024a9fea5] /lib64/libc.so.6(clone+0x6d)[0x7f0022e408cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7effb400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:54:02 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:54:02 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:54:02 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:54:02 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:54:02 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:54:02 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:54:02 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:54:02 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:54:02 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:54:02 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136982050 2020-01-31 23:54:02 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:54:02 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:54:02 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:54:02 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:54:02 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:54:02 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:54:02 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:54:02 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136982059; transaction id 5903519 2020-01-31 23:54:02 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:54:02 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:54:02 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:54:02 0 [Note] Starting crash recovery... 2020-01-31 23:54:02 0 [Note] Crash recovery finished. 2020-01-31 23:54:02 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:54:02 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:54:02 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:54:02 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000031' at position 140325 2020-01-31 23:54:02 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:54:02 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:54:02 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000031' at position 140325 2020-01-31 23:54:02 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:54:02 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:54:02 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:54:02 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:54:02 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:54:03 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:54:03 2020-01-31 23:54:30 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:54:31 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:54:31 14 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:54:33 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:54:33 17 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:54:33 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:54:33 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:54:33 18 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:54:33 18 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:54:33 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7eff24000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7eff7909acf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ace8552f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ace7fe93af] sigaction.c:0(__restore_rt)[0x7eff94240630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55ace7f17151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55ace8055545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55ace7e0bccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55ace7e01d4b] /usr/sbin/mysqld(+0x6ca792)[0x55ace7e0c792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55ace7e346da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ace7e3665d] /usr/sbin/mysqld(+0x65e3f3)[0x55ace7da03f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55ace7d9fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55ace7e33b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ace7e3665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55ace7e36761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55ace7e37377] /usr/sbin/mysqld(+0x58fd84)[0x55ace7cd1d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55ace7ddeb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55ace7df6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55ace7df6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55ace7df75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55ace7df7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55ace7de656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55ace7de807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55ace7ec3afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55ace7ec3bdd] pthread_create.c:0(start_thread)[0x7eff94238ea5] /lib64/libc.so.6(clone+0x6d)[0x7eff925d98cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7eff2400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 20 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:54:40 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:54:40 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:54:40 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:54:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:54:40 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:54:40 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:54:40 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:54:40 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:54:40 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:54:40 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136984896 2020-01-31 23:54:40 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:54:40 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:54:40 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:54:40 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:54:40 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:54:40 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:54:40 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:54:40 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136984905; transaction id 5903530 2020-01-31 23:54:40 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:54:40 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:54:40 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:54:40 0 [Note] Starting crash recovery... 2020-01-31 23:54:40 0 [Note] Crash recovery finished. 2020-01-31 23:54:40 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:54:40 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:54:40 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:54:40 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000032' at position 66468 2020-01-31 23:54:40 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:54:40 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:54:40 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000032' at position 66468 2020-01-31 23:54:40 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:54:40 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:54:40 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:54:40 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:54:40 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:54:40 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:54:40 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:54:40 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:54:40 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:54:40 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:54:41 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:54:41 2020-01-31 23:54:42 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:54:42 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000032' at position 68591 2020-01-31 23:54:42 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:54:48 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000073, 1464) 2020-01-31 23:55:06 16 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:55:06 18 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:55:06 18 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:55:07 17 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-01-31 23:55:07 17 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-01-31 23:55:17 24 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-01-31 23:55:17 24 [Warning] Checking table: './openrecords/wp_options' 2020-01-31 23:55:19 29 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:55:19 29 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:55:19 30 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:55:19 30 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:55:19 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f82b0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f830c078cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d34d3a3f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55d34ce3a3af] sigaction.c:0(__restore_rt)[0x7f83267c5630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55d34cd68151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55d34cea6545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55d34cc5cccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55d34cc52d4b] /usr/sbin/mysqld(+0x6ca792)[0x55d34cc5d792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55d34cc856da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d34cc8765d] /usr/sbin/mysqld(+0x65e3f3)[0x55d34cbf13f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55d34cbf0cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55d34cc84b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d34cc8765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55d34cc87761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55d34cc88377] /usr/sbin/mysqld(+0x58fd84)[0x55d34cb22d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55d34cc2fb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55d34cc47b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55d34cc47cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55d34cc485e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55d34cc48677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55d34cc3756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55d34cc3907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55d34cd14afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d34cd14bdd] pthread_create.c:0(start_thread)[0x7f83267bdea5] /lib64/libc.so.6(clone+0x6d)[0x7f8324b5e8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f82b000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 32 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:55:25 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:55:25 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:55:25 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:55:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:55:25 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:55:25 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:55:25 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:55:25 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:55:25 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:55:26 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136987759 2020-01-31 23:55:26 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:55:27 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:55:27 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:55:27 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:55:27 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:55:27 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:55:27 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:55:27 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136987768; transaction id 5903541 2020-01-31 23:55:27 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:55:27 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:55:27 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:55:27 0 [Note] Starting crash recovery... 2020-01-31 23:55:27 0 [Note] Crash recovery finished. 2020-01-31 23:55:27 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:55:27 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:55:27 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:55:27 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000032' at position 68591 2020-01-31 23:55:27 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:55:27 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:55:27 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000032' at position 68591 2020-01-31 23:55:27 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:55:27 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:55:27 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:55:27 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:55:27 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:55:27 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:55:27 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:55:27 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:55:27 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:55:27 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:55:28 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:55:28 2020-01-31 23:55:37 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:55:37 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000033' at position 138030 2020-01-31 23:55:37 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:55:43 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000075, 163717) 2020-01-31 23:55:51 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:55:51 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:55:51 14 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:56:02 19 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:56:02 19 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:56:02 20 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:56:02 20 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:56:02 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2d5c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2dbc7c8cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5574555bdf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5574550543af] sigaction.c:0(__restore_rt)[0x7f2dd9959630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x557454f82151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5574550c0545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x557454e76ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x557454e6cd4b] /usr/sbin/mysqld(+0x6ca792)[0x557454e77792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x557454e9f6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557454ea165d] /usr/sbin/mysqld(+0x65e3f3)[0x557454e0b3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x557454e0acb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x557454e9eb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557454ea165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x557454ea1761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x557454ea2377] /usr/sbin/mysqld(+0x58fd84)[0x557454d3cd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x557454e49b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x557454e61b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x557454e61cd2] /usr/sbin/mysqld(+0x6b55e7)[0x557454e625e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x557454e62677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x557454e5156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x557454e5307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x557454f2eafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x557454f2ebdd] pthread_create.c:0(start_thread)[0x7f2dd9951ea5] /lib64/libc.so.6(clone+0x6d)[0x7f2dd7cf28cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f2d5c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 21 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:56:08 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:56:08 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:56:08 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:56:08 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:56:08 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:56:08 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:56:08 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:56:08 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:56:08 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:56:08 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136990606 2020-01-31 23:56:08 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:56:08 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:56:08 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:56:08 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:56:08 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:56:08 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:56:08 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:56:08 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136990615; transaction id 5903552 2020-01-31 23:56:08 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:56:08 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:56:08 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:56:08 0 [Note] Starting crash recovery... 2020-01-31 23:56:08 0 [Note] Crash recovery finished. 2020-01-31 23:56:08 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:56:08 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:56:08 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:56:08 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000033' at position 138030 2020-01-31 23:56:08 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:56:08 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:56:08 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:56:08 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:56:08 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:56:08 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:56:08 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:56:08 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000033' at position 138030 2020-01-31 23:56:09 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:56:09 2020-01-31 23:56:15 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:56:15 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000034' at position 2513 2020-01-31 23:56:15 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:56:20 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000076, 45671) 2020-01-31 23:56:36 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:56:37 16 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-01-31 23:56:37 16 [Warning] Checking table: './pizzahut/wp_options' 2020-01-31 23:56:38 20 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-01-31 23:56:38 20 [Warning] Checking table: './wpsem/wp_options' 2020-01-31 23:56:38 22 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:56:38 22 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:56:49 27 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:56:49 27 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:56:49 28 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:56:49 28 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:56:49 28 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:56:49 28 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:56:49 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fd6e8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fd705350cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ac1e05df2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ac1daf43af] sigaction.c:0(__restore_rt)[0x7fd766564630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55ac1da22151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55ac1db60545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55ac1d916ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55ac1d90cd4b] /usr/sbin/mysqld(+0x6ca792)[0x55ac1d917792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55ac1d93f6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ac1d94165d] /usr/sbin/mysqld(+0x65e3f3)[0x55ac1d8ab3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55ac1d8aacb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55ac1d93eb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ac1d94165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55ac1d941761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55ac1d942377] /usr/sbin/mysqld(+0x58fd84)[0x55ac1d7dcd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55ac1d8e9b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55ac1d901b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55ac1d901cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55ac1d9025e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55ac1d902677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55ac1d8f156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55ac1d8f307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55ac1d9ceafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55ac1d9cebdd] pthread_create.c:0(start_thread)[0x7fd76655cea5] /lib64/libc.so.6(clone+0x6d)[0x7fd7648fd8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fd6e800feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 29 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:56:55 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:56:55 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:56:55 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:56:55 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:56:55 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:56:55 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:56:55 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:56:55 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:56:55 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:56:55 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136993495 2020-01-31 23:56:55 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:56:55 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:56:55 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:56:55 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:56:55 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:56:55 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:56:55 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:56:55 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136993504; transaction id 5903564 2020-01-31 23:56:55 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:56:55 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:56:55 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:56:55 0 [Note] Starting crash recovery... 2020-01-31 23:56:55 0 [Note] Crash recovery finished. 2020-01-31 23:56:55 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:56:55 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:56:55 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:56:55 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000034' at position 2513 2020-01-31 23:56:55 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:56:55 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:56:55 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-01-31 23:56:55 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-01-31 23:56:55 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-01-31 23:56:55 11 [Note] wpwsutv.wp_options: Table is fixed 2020-01-31 23:56:55 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000034' at position 2513 2020-01-31 23:56:55 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:56:55 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:56:55 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-01-31 23:56:55 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:56:55 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:56:55 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:56:56 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:56:56 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000035' at position 1464 2020-01-31 23:56:56 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:56:56 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:56:56 2020-01-31 23:57:02 17 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000077, 175269) 2020-01-31 23:57:22 18 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:57:26 21 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:57:26 21 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:57:31 24 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-01-31 23:57:31 24 [Warning] Checking table: './wpcomm306/wp_options' 2020-01-31 23:57:32 27 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:57:32 27 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:57:32 28 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:57:32 28 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:57:33 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f4e90000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4ef564bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5630d593ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5630d53d63af] sigaction.c:0(__restore_rt)[0x7f4f107d8630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5630d5304151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5630d5442545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5630d51f8ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5630d51eed4b] /usr/sbin/mysqld(+0x6ca792)[0x5630d51f9792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5630d52216da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5630d522365d] /usr/sbin/mysqld(+0x65e3f3)[0x5630d518d3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5630d518ccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5630d5220b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5630d522365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5630d5223761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5630d5224377] /usr/sbin/mysqld(+0x58fd84)[0x5630d50bed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5630d51cbb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5630d51e3b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5630d51e3cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5630d51e45e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5630d51e4677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5630d51d356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5630d51d507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5630d52b0afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5630d52b0bdd] pthread_create.c:0(start_thread)[0x7f4f107d0ea5] /lib64/libc.so.6(clone+0x6d)[0x7f4f0eb718cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f4e9000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 29 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:57:39 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:57:39 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:57:39 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:57:39 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:57:39 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:57:39 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:57:39 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:57:39 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:57:39 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:57:39 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5136997149 2020-01-31 23:57:39 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:57:39 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:57:39 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:57:39 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:57:39 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:57:39 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:57:39 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:57:39 0 [Note] InnoDB: 10.4.10 started; log sequence number 5136997158; transaction id 5903575 2020-01-31 23:57:39 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:57:39 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:57:39 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:57:39 0 [Note] Starting crash recovery... 2020-01-31 23:57:39 0 [Note] Crash recovery finished. 2020-01-31 23:57:39 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:57:39 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:57:39 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:57:39 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000035' at position 1464 2020-01-31 23:57:39 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:57:39 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:57:39 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000035' at position 1464 2020-01-31 23:57:39 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:57:39 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:57:39 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:57:39 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:57:39 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:57:40 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:57:40 2020-01-31 23:57:49 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:57:49 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000036' at position 90273 2020-01-31 23:57:49 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:57:56 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000078, 92993) 2020-01-31 23:58:07 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:58:07 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:58:07 14 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:58:07 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:58:07 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:58:07 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f92b0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f93080f4cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x564082dedf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5640828843af] sigaction.c:0(__restore_rt)[0x7f9323269630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5640827b2151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5640828f0545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5640826a6ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56408269cd4b] /usr/sbin/mysqld(+0x6ca792)[0x5640826a7792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5640826cf6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5640826d165d] /usr/sbin/mysqld(+0x65e3f3)[0x56408263b3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56408263acb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5640826ceb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5640826d165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5640826d1761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5640826d2377] /usr/sbin/mysqld(+0x58fd84)[0x56408256cd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x564082679b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564082691b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564082691cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5640826925e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564082692677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56408268156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56408268307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56408275eafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56408275ebdd] pthread_create.c:0(start_thread)[0x7f9323261ea5] /lib64/libc.so.6(clone+0x6d)[0x7f93216028cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f92b000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:58:14 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:58:14 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:58:14 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:58:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:58:14 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:58:14 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:58:14 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:58:14 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:58:14 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:58:14 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137000011 2020-01-31 23:58:14 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:58:14 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:58:14 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:58:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:58:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:58:14 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:58:14 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137000020; transaction id 5903586 2020-01-31 23:58:14 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:58:14 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:58:14 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:58:14 0 [Note] Starting crash recovery... 2020-01-31 23:58:14 0 [Note] Crash recovery finished. 2020-01-31 23:58:14 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:58:14 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:58:14 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:58:14 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000036' at position 90273 2020-01-31 23:58:14 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:58:14 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:58:14 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:58:14 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:58:14 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:58:14 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:58:14 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:58:14 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000036' at position 90273 2020-01-31 23:58:15 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:58:15 2020-01-31 23:58:28 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:58:28 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000037' at position 49814 2020-01-31 23:58:28 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:58:34 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000079, 1464) 2020-01-31 23:58:42 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:58:42 14 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-01-31 23:58:42 14 [Warning] Checking table: './ulrich/wp_options' 2020-01-31 23:58:43 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:58:43 16 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:58:43 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:58:43 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:58:43 17 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:58:43 17 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:58:43 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9938000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f99280ddcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5645a6aa4f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5645a653b3af] sigaction.c:0(__restore_rt)[0x7f9942ea1630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5645a6469151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5645a65a7545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5645a635dccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5645a6353d4b] /usr/sbin/mysqld(+0x6ca792)[0x5645a635e792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5645a63866da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5645a638865d] /usr/sbin/mysqld(+0x65e3f3)[0x5645a62f23f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5645a62f1cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5645a6385b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5645a638865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5645a6388761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5645a6389377] /usr/sbin/mysqld(+0x58fd84)[0x5645a6223d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5645a6330b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5645a6348b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5645a6348cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5645a63495e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5645a6349677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5645a633856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5645a633a07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5645a6415afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5645a6415bdd] pthread_create.c:0(start_thread)[0x7f9942e99ea5] /lib64/libc.so.6(clone+0x6d)[0x7f994123a8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f993800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:58:49 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:58:49 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:58:49 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:58:49 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:58:49 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:58:49 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:58:49 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:58:49 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:58:49 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:58:49 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137002825 2020-01-31 23:58:49 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:58:49 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:58:49 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:58:49 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:58:49 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:58:49 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:58:49 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:58:49 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137002834; transaction id 5903597 2020-01-31 23:58:49 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:58:49 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:58:49 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:58:49 0 [Note] Starting crash recovery... 2020-01-31 23:58:49 0 [Note] Crash recovery finished. 2020-01-31 23:58:49 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:58:49 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:58:49 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:58:49 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000037' at position 49814 2020-01-31 23:58:49 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:58:49 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:58:49 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000037' at position 49814 2020-01-31 23:58:49 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:58:49 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:58:49 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:58:49 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:58:49 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:58:50 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:58:50 2020-01-31 23:59:05 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:59:05 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000038' at position 45665 2020-01-31 23:59:05 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:59:11 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000080, 2059) 2020-01-31 23:59:17 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:59:18 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:59:18 14 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:59:24 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:59:24 17 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:59:24 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:59:24 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 200131 23:59:24 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f33e4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f3444b7ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x564b01853f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x564b012ea3af] sigaction.c:0(__restore_rt)[0x7f3461d07630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564b01218151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564b01356545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564b0110cccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x564b01102d4b] /usr/sbin/mysqld(+0x6ca792)[0x564b0110d792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x564b011356da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564b0113765d] /usr/sbin/mysqld(+0x65e3f3)[0x564b010a13f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x564b010a0cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x564b01134b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564b0113765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x564b01137761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x564b01138377] /usr/sbin/mysqld(+0x58fd84)[0x564b00fd2d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x564b010dfb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564b010f7b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564b010f7cd2] /usr/sbin/mysqld(+0x6b55e7)[0x564b010f85e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564b010f8677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x564b010e756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x564b010e907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x564b011c4afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x564b011c4bdd] pthread_create.c:0(start_thread)[0x7f3461cffea5] /lib64/libc.so.6(clone+0x6d)[0x7f34600a08cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f33e400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-01-31 23:59:30 0 [Note] InnoDB: Using Linux native AIO 2020-01-31 23:59:30 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-01-31 23:59:30 0 [Note] InnoDB: Uses event mutexes 2020-01-31 23:59:30 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-01-31 23:59:30 0 [Note] InnoDB: Number of pools: 1 2020-01-31 23:59:30 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-01-31 23:59:30 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-01-31 23:59:30 0 [Note] InnoDB: Completed initialization of buffer pool 2020-01-31 23:59:30 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-01-31 23:59:30 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137005688 2020-01-31 23:59:30 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-01-31 23:59:30 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-01-31 23:59:30 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-01-31 23:59:30 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-01-31 23:59:30 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-01-31 23:59:30 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-01-31 23:59:30 0 [Note] InnoDB: Waiting for purge to start 2020-01-31 23:59:31 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137005697; transaction id 5903608 2020-01-31 23:59:31 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-01-31 23:59:31 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-01-31 23:59:31 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-01-31 23:59:31 0 [Note] Starting crash recovery... 2020-01-31 23:59:31 0 [Note] Crash recovery finished. 2020-01-31 23:59:31 0 [Note] Server socket created on IP: '::'. 2020-01-31 23:59:31 0 [Note] Reading of all Master_info entries succeeded 2020-01-31 23:59:31 0 [Note] Added new Master_info '' to hash table 2020-01-31 23:59:31 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000038' at position 45665 2020-01-31 23:59:31 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-01-31 23:59:31 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-01-31 23:59:31 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000038' at position 45665 2020-01-31 23:59:31 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-01-31 23:59:31 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-01-31 23:59:31 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-01-31 23:59:31 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-01-31 23:59:31 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-01-31 23:59:32 0 [Note] InnoDB: Buffer pool(s) load completed at 200131 23:59:32 2020-01-31 23:59:45 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-01-31 23:59:45 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000039' at position 69315 2020-01-31 23:59:45 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-01-31 23:59:51 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000081, 89673) 2020-01-31 23:59:57 15 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-01-31 23:59:58 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-01-31 23:59:58 16 [Warning] Checking table: './ojsdata/sessions' 2020-01-31 23:59:58 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-01-31 23:59:58 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-01-31 23:59:58 18 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-01-31 23:59:58 18 [Warning] Checking table: './ksarn/wp_options' 2020-01-31 23:59:58 17 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-01-31 23:59:58 17 [Warning] Checking table: './ojsdata/controlled_vocabs' 200131 23:59:58 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc738000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc788594cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x556e00574f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x556e0000b3af] sigaction.c:0(__restore_rt)[0x7fc7a5783630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x556dfff39151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x556e00077545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x556dffe2dccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x556dffe23d4b] /usr/sbin/mysqld(+0x6ca792)[0x556dffe2e792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x556dffe566da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556dffe5865d] /usr/sbin/mysqld(+0x65e3f3)[0x556dffdc23f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x556dffdc1cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x556dffe55b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556dffe5865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x556dffe58761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x556dffe59377] /usr/sbin/mysqld(+0x58fd84)[0x556dffcf3d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x556dffe00b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x556dffe18b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x556dffe18cd2] /usr/sbin/mysqld(+0x6b55e7)[0x556dffe195e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x556dffe19677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x556dffe0856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x556dffe0a07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x556dffee5afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x556dffee5bdd] pthread_create.c:0(start_thread)[0x7fc7a577bea5] /lib64/libc.so.6(clone+0x6d)[0x7fc7a3b1c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc73800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:00:05 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:00:05 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:00:05 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:00:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:00:05 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:00:05 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:00:05 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:00:05 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:00:05 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:00:05 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137008535 2020-02-01 0:00:05 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:00:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:00:05 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:00:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:00:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:00:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:00:05 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:00:05 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137008544; transaction id 5903619 2020-02-01 0:00:05 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:00:05 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:00:05 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:00:05 0 [Note] Starting crash recovery... 2020-02-01 0:00:05 0 [Note] Crash recovery finished. 2020-02-01 0:00:05 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:00:05 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:00:05 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:00:05 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000039' at position 69315 2020-02-01 0:00:05 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:00:05 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:00:05 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:00:05 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:00:05 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:00:05 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:00:05 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:00:05 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000039' at position 69315 2020-02-01 0:00:06 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:00:06 2020-02-01 0:00:21 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:00:21 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000040' at position 6602 2020-02-01 0:00:21 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:00:27 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000082, 2513) 2020-02-01 0:00:33 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:00:33 14 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 0:00:33 14 [Warning] Checking table: './wpcomm306/wp_options' 2020-02-01 0:00:34 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:00:34 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:00:34 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:00:34 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:00:34 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc2d8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc2e05d6cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561b26b11f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x561b265a83af] sigaction.c:0(__restore_rt)[0x7fc34cfb1630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x561b264d6151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x561b26614545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x561b263caccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x561b263c0d4b] /usr/sbin/mysqld(+0x6ca792)[0x561b263cb792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x561b263f36da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x561b263f565d] /usr/sbin/mysqld(+0x65e3f3)[0x561b2635f3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x561b2635ecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x561b263f2b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x561b263f565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x561b263f5761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x561b263f6377] /usr/sbin/mysqld(+0x58fd84)[0x561b26290d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x561b2639db6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x561b263b5b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x561b263b5cd2] /usr/sbin/mysqld(+0x6b55e7)[0x561b263b65e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x561b263b6677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x561b263a556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x561b263a707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x561b26482afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x561b26482bdd] pthread_create.c:0(start_thread)[0x7fc34cfa9ea5] /lib64/libc.so.6(clone+0x6d)[0x7fc34b34a8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc2d800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:00:40 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:00:40 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:00:40 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:00:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:00:40 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:00:40 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:00:40 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:00:40 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:00:40 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:00:40 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137011365 2020-02-01 0:00:40 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:00:40 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:00:40 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:00:40 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:00:40 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:00:40 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:00:40 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:00:41 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137011374; transaction id 5903630 2020-02-01 0:00:41 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:00:41 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:00:41 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:00:41 0 [Note] Starting crash recovery... 2020-02-01 0:00:41 0 [Note] Crash recovery finished. 2020-02-01 0:00:41 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:00:41 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:00:41 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:00:41 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000040' at position 6602 2020-02-01 0:00:41 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:00:41 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:00:41 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000040' at position 6602 2020-02-01 0:00:41 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:00:41 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:00:41 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:00:41 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:00:41 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:00:42 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:00:42 2020-02-01 0:00:56 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:00:56 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000041' at position 46517 2020-02-01 0:00:56 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:01:02 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000083, 1464) 2020-02-01 0:01:08 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:01:13 18 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-02-01 0:01:13 18 [Warning] Checking table: './wpcomm305/comm305_options' 2020-02-01 0:01:15 20 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-02-01 0:01:15 20 [Warning] Checking table: './wpstart/wp_options' 2020-02-01 0:01:15 23 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:01:15 23 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:01:15 24 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:01:15 24 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:01:16 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=11 max_threads=502 thread_count=19 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fd050000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fd0040e7cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c7b0b7ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c7b06163af] sigaction.c:0(__restore_rt)[0x7fd06edab630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c7b0544151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c7b0682545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c7b0438ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c7b042ed4b] /usr/sbin/mysqld(+0x6ca792)[0x55c7b0439792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c7b04616da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c7b046365d] /usr/sbin/mysqld(+0x65e3f3)[0x55c7b03cd3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c7b03cccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c7b0460b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c7b046365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c7b0463761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c7b0464377] /usr/sbin/mysqld(+0x58fd84)[0x55c7b02fed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c7b040bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c7b0423b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c7b0423cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c7b04245e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c7b0424677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c7b041356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c7b041507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c7b04f0afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c7b04f0bdd] pthread_create.c:0(start_thread)[0x7fd06eda3ea5] /lib64/libc.so.6(clone+0x6d)[0x7fd06d1448cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fd05000feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 27 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:01:22 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:01:22 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:01:22 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:01:22 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:01:22 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:01:22 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:01:22 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:01:22 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:01:22 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:01:22 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137014179 2020-02-01 0:01:22 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:01:23 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:01:23 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:01:23 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:01:23 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:01:23 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:01:23 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:01:23 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137014188; transaction id 5903641 2020-02-01 0:01:23 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:01:23 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:01:23 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:01:23 0 [Note] Starting crash recovery... 2020-02-01 0:01:23 0 [Note] Crash recovery finished. 2020-02-01 0:01:23 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:01:23 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:01:23 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:01:23 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000041' at position 46517 2020-02-01 0:01:23 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:01:23 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:01:23 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:01:23 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:01:23 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:01:23 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:01:23 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000041' at position 46517 2020-02-01 0:01:23 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:01:23 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:01:23 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:01:23 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:01:23 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:01:23 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:01:24 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:01:24 2020-02-01 0:01:30 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:01:30 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000042' at position 1464 2020-02-01 0:01:30 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:01:36 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000084, 101544) 2020-02-01 0:01:48 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:01:50 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:01:50 16 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:01:50 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:01:50 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:01:50 17 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:01:50 17 [Warning] Checking table: './ojsdata/controlled_vocabs' 2020-02-01 0:01:50 20 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:01:50 20 [Warning] Checking table: './ksarn/wp_options' 200201 0:01:50 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f3068000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f30c40ddcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ee405aaf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ee400413af] sigaction.c:0(__restore_rt)[0x7f30dee84630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55ee3ff6f151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55ee400ad545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55ee3fe63ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55ee3fe59d4b] /usr/sbin/mysqld(+0x6ca792)[0x55ee3fe64792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55ee3fe8c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ee3fe8e65d] /usr/sbin/mysqld(+0x65e3f3)[0x55ee3fdf83f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55ee3fdf7cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55ee3fe8bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ee3fe8e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55ee3fe8e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55ee3fe8f377] /usr/sbin/mysqld(+0x58fd84)[0x55ee3fd29d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55ee3fe36b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55ee3fe4eb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55ee3fe4ecd2] /usr/sbin/mysqld(+0x6b55e7)[0x55ee3fe4f5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55ee3fe4f677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55ee3fe3e56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55ee3fe4007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55ee3ff1bafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55ee3ff1bbdd] pthread_create.c:0(start_thread)[0x7f30dee7cea5] /lib64/libc.so.6(clone+0x6d)[0x7f30dd21d8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f306800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:01:57 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:01:57 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:01:57 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:01:57 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:01:57 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:01:57 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:01:57 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:01:57 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:01:57 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:01:57 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137017042 2020-02-01 0:01:57 0 [Note] InnoDB: Starting final batch to recover 1 pages from redo log. 2020-02-01 0:01:57 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:01:57 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:01:57 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:01:57 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:01:57 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:01:57 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:01:57 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:01:57 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137017504; transaction id 5903657 2020-02-01 0:01:57 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:01:57 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:01:57 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:01:57 0 [Note] Starting crash recovery... 2020-02-01 0:01:57 0 [Note] Crash recovery finished. 2020-02-01 0:01:57 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:01:57 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:01:57 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:01:57 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000042' at position 1464 2020-02-01 0:01:57 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:01:57 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:01:57 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:01:57 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:01:57 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:01:57 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:01:57 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:01:57 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000042' at position 1464 2020-02-01 0:01:58 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:01:58 2020-02-01 0:02:12 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:02:12 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000043' at position 102219 2020-02-01 0:02:12 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:02:18 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000085, 23382) 2020-02-01 0:02:23 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:02:24 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:02:24 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:02:24 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:02:24 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:02:24 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f1d08000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f1d6da3acf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561d55d10f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x561d557a73af] sigaction.c:0(__restore_rt)[0x7f1d88b5c630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x561d556d5151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x561d55813545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x561d555c9ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x561d555bfd4b] /usr/sbin/mysqld(+0x6ca792)[0x561d555ca792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x561d555f26da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x561d555f465d] /usr/sbin/mysqld(+0x65e3f3)[0x561d5555e3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x561d5555dcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x561d555f1b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x561d555f465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x561d555f4761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x561d555f5377] /usr/sbin/mysqld(+0x58fd84)[0x561d5548fd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x561d5559cb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x561d555b4b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x561d555b4cd2] /usr/sbin/mysqld(+0x6b55e7)[0x561d555b55e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x561d555b5677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x561d555a456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x561d555a607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x561d55681afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x561d55681bdd] pthread_create.c:0(start_thread)[0x7f1d88b54ea5] /lib64/libc.so.6(clone+0x6d)[0x7f1d86ef58cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f1d0800ff10): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:02:31 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:02:31 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:02:31 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:02:31 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:02:31 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:02:31 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:02:31 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:02:31 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:02:31 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:02:31 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137020446 2020-02-01 0:02:31 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:02:31 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:02:31 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:02:31 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:02:31 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:02:31 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:02:31 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:02:31 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137020455; transaction id 5903668 2020-02-01 0:02:31 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:02:31 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:02:31 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:02:31 0 [Note] Starting crash recovery... 2020-02-01 0:02:31 0 [Note] Crash recovery finished. 2020-02-01 0:02:31 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:02:31 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:02:31 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:02:31 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000043' at position 102219 2020-02-01 0:02:31 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:02:31 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:02:31 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000043' at position 102219 2020-02-01 0:02:31 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:02:31 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:02:31 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:02:31 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:02:31 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:02:32 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:02:32 2020-02-01 0:02:52 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:02:52 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000044' at position 425093 2020-02-01 0:02:52 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:02:58 17 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000086, 1464) 2020-02-01 0:02:59 18 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:03:00 19 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:03:00 19 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:03:00 20 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:03:00 20 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:03:01 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ffaf40009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ffb60ca0cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55fda16b7f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55fda114e3af] sigaction.c:0(__restore_rt)[0x7ffb7d020630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55fda107c151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55fda11ba545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55fda0f70ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55fda0f66d4b] /usr/sbin/mysqld(+0x6ca792)[0x55fda0f71792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55fda0f996da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55fda0f9b65d] /usr/sbin/mysqld(+0x65e3f3)[0x55fda0f053f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55fda0f04cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55fda0f98b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55fda0f9b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55fda0f9b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55fda0f9c377] /usr/sbin/mysqld(+0x58fd84)[0x55fda0e36d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55fda0f43b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55fda0f5bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55fda0f5bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55fda0f5c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55fda0f5c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55fda0f4b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55fda0f4d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55fda1028afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55fda1028bdd] pthread_create.c:0(start_thread)[0x7ffb7d018ea5] /lib64/libc.so.6(clone+0x6d)[0x7ffb7b3b98cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ffaf400fe00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 21 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:03:07 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:03:07 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:03:07 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:03:07 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:03:07 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:03:07 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:03:07 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:03:07 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:03:07 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:03:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137023293 2020-02-01 0:03:07 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:03:07 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:03:07 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:03:07 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:03:07 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:03:07 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:03:07 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:03:07 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137023302; transaction id 5903679 2020-02-01 0:03:07 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:03:07 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:03:07 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:03:07 0 [Note] Starting crash recovery... 2020-02-01 0:03:07 0 [Note] Crash recovery finished. 2020-02-01 0:03:07 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:03:07 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:03:07 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:03:07 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000044' at position 425093 2020-02-01 0:03:07 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:03:07 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:03:07 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:03:07 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:03:07 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:03:07 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:03:07 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:03:07 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000044' at position 425093 2020-02-01 0:03:08 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:03:08 2020-02-01 0:03:25 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:03:25 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000045' at position 1766 2020-02-01 0:03:25 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:03:30 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000087, 1464) 2020-02-01 0:03:34 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:03:37 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:03:37 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:03:37 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:03:37 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:03:37 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f71e4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f724807acf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5562224ddf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x556221f743af] sigaction.c:0(__restore_rt)[0x7f72633f2630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x556221ea2151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x556221fe0545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x556221d96ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x556221d8cd4b] /usr/sbin/mysqld(+0x6ca792)[0x556221d97792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x556221dbf6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556221dc165d] /usr/sbin/mysqld(+0x65e3f3)[0x556221d2b3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x556221d2acb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x556221dbeb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556221dc165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x556221dc1761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x556221dc2377] /usr/sbin/mysqld(+0x58fd84)[0x556221c5cd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x556221d69b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x556221d81b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x556221d81cd2] /usr/sbin/mysqld(+0x6b55e7)[0x556221d825e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x556221d82677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x556221d7156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x556221d7307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x556221e4eafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x556221e4ebdd] pthread_create.c:0(start_thread)[0x7f72633eaea5] /lib64/libc.so.6(clone+0x6d)[0x7f726178b8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f71e400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:03:43 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:03:43 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:03:43 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:03:43 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:03:43 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:03:43 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:03:43 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:03:43 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:03:43 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:03:43 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137026156 2020-02-01 0:03:43 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:03:43 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:03:43 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:03:43 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:03:43 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:03:43 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:03:43 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:03:43 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137026165; transaction id 5903690 2020-02-01 0:03:43 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:03:43 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:03:43 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:03:43 0 [Note] Starting crash recovery... 2020-02-01 0:03:43 0 [Note] Crash recovery finished. 2020-02-01 0:03:43 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:03:43 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:03:43 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:03:43 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000045' at position 1766 2020-02-01 0:03:43 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:03:43 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:03:43 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000045' at position 1766 2020-02-01 0:03:43 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:03:43 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:03:43 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:03:43 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:03:43 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:03:44 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:03:44 2020-02-01 0:04:09 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:04:13 13 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:04:13 13 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:04:14 14 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:04:14 14 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:04:14 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=502 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f4ae8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4b01b2ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b478bfff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55b4786963af] sigaction.c:0(__restore_rt)[0x7f4b62c8d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55b4785c4151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55b478702545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55b4784b8ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55b4784aed4b] /usr/sbin/mysqld(+0x6ca792)[0x55b4784b9792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55b4784e16da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b4784e365d] /usr/sbin/mysqld(+0x65e3f3)[0x55b47844d3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55b47844ccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55b4784e0b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b4784e365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55b4784e3761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55b4784e4377] /usr/sbin/mysqld(+0x58fd84)[0x55b47837ed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55b47848bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55b4784a3b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55b4784a3cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55b4784a45e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55b4784a4677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55b47849356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55b47849507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55b478570afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55b478570bdd] pthread_create.c:0(start_thread)[0x7f4b62c85ea5] /lib64/libc.so.6(clone+0x6d)[0x7f4b610268cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f4ae800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 15 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:04:20 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:04:20 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:04:20 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:04:20 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:04:20 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:04:20 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:04:20 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:04:20 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:04:20 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:04:20 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137029029 2020-02-01 0:04:20 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:04:20 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:04:20 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:04:20 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:04:20 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:04:20 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:04:20 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:04:20 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137029038; transaction id 5903702 2020-02-01 0:04:20 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:04:20 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:04:20 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:04:20 0 [Note] Starting crash recovery... 2020-02-01 0:04:20 0 [Note] Crash recovery finished. 2020-02-01 0:04:20 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:04:20 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:04:20 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:04:20 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000046' at position 87236 2020-02-01 0:04:20 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:04:20 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:04:20 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000046' at position 87236 2020-02-01 0:04:20 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:04:20 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:04:20 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:04:20 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:04:20 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:04:21 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:04:21 2020-02-01 0:04:31 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:04:31 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000046' at position 178750 2020-02-01 0:04:31 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:04:36 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000088, 45217) 2020-02-01 0:04:49 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:04:51 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:04:51 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:04:51 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:04:51 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:04:51 16 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:04:51 16 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:04:51 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fbd2c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fbd8813fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b5b6e7cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55b5b69133af] sigaction.c:0(__restore_rt)[0x7fbda30a6630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55b5b6841151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55b5b697f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55b5b6735ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55b5b672bd4b] /usr/sbin/mysqld(+0x6ca792)[0x55b5b6736792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55b5b675e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b5b676065d] /usr/sbin/mysqld(+0x65e3f3)[0x55b5b66ca3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55b5b66c9cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55b5b675db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b5b676065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55b5b6760761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55b5b6761377] /usr/sbin/mysqld(+0x58fd84)[0x55b5b65fbd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55b5b6708b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55b5b6720b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55b5b6720cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55b5b67215e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55b5b6721677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55b5b671056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55b5b671207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55b5b67edafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55b5b67edbdd] pthread_create.c:0(start_thread)[0x7fbda309eea5] /lib64/libc.so.6(clone+0x6d)[0x7fbda143f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fbd2c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:04:57 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:04:57 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:04:57 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:04:57 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:04:57 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:04:57 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:04:57 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:04:57 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:04:57 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:04:58 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137032683 2020-02-01 0:04:58 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:04:58 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:04:58 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:04:58 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:04:58 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:04:58 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:04:58 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:04:58 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137032692; transaction id 5903713 2020-02-01 0:04:58 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:04:58 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:04:58 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:04:58 0 [Note] Starting crash recovery... 2020-02-01 0:04:58 0 [Note] Crash recovery finished. 2020-02-01 0:04:58 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:04:58 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:04:58 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:04:58 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000046' at position 178750 2020-02-01 0:04:58 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:04:58 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:04:58 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000046' at position 178750 2020-02-01 0:04:58 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:04:58 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:04:58 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:04:58 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:04:58 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:05:00 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:05:00 2020-02-01 0:05:05 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:05:05 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000047' at position 1464 2020-02-01 0:05:05 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:05:11 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000090, 1464) 2020-02-01 0:05:25 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:05:25 14 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-02-01 0:05:25 14 [Warning] Checking table: './wpsem/wp_options' 2020-02-01 0:05:26 18 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-02-01 0:05:26 18 [Warning] Checking table: './wpcomm305/comm305_options' 2020-02-01 0:05:26 19 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:05:26 19 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:05:27 23 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:05:27 23 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:05:27 24 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:05:27 24 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:05:27 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=10 max_threads=502 thread_count=18 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f7e24000e78 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f7e5871bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d8b2036f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55d8b1acd3af] sigaction.c:0(__restore_rt)[0x7f7ecab18630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55d8b19fb151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55d8b1b39545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55d8b18efccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55d8b18e5d4b] /usr/sbin/mysqld(+0x6ca792)[0x55d8b18f0792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55d8b19186da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d8b191a65d] /usr/sbin/mysqld(+0x65e3f3)[0x55d8b18843f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55d8b1883cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55d8b1917b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d8b191a65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55d8b191a761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55d8b191b377] /usr/sbin/mysqld(+0x58fd84)[0x55d8b17b5d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55d8b18c2b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55d8b18dab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55d8b18dacd2] /usr/sbin/mysqld(+0x6b55e7)[0x55d8b18db5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55d8b18db677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55d8b18ca56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55d8b18cc07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55d8b19a7afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d8b19a7bdd] pthread_create.c:0(start_thread)[0x7f7ecab10ea5] /lib64/libc.so.6(clone+0x6d)[0x7f7ec8eb18cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f7e2400f140): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 26 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:05:34 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:05:34 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:05:34 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:05:34 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:05:34 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:05:34 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:05:34 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:05:34 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:05:34 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:05:34 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137035545 2020-02-01 0:05:34 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:05:34 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:05:34 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:05:34 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:05:34 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:05:34 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:05:34 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:05:34 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137035554; transaction id 5903724 2020-02-01 0:05:34 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:05:34 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:05:34 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:05:34 0 [Note] Starting crash recovery... 2020-02-01 0:05:34 0 [Note] Crash recovery finished. 2020-02-01 0:05:34 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:05:34 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:05:34 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:05:34 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000047' at position 1464 2020-02-01 0:05:34 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:05:34 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:05:34 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000047' at position 1464 2020-02-01 0:05:34 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:05:34 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:05:34 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:05:34 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:05:34 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:05:35 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:05:35 2020-02-01 0:05:47 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:05:47 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000048' at position 401381 2020-02-01 0:05:47 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:05:53 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000091, 148792) 2020-02-01 0:06:00 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:06:00 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:06:00 14 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:06:05 20 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-02-01 0:06:05 20 [Warning] Checking table: './wpcomm305/comm305_options' 2020-02-01 0:06:05 20 [ERROR] Got an error from unknown thread, /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.10/storage/myisam/ha_myisam.cc:1080 2020-02-01 0:06:05 20 [Warning] Recovering table: './wpcomm305/comm305_options' 2020-02-01 0:06:05 20 [Note] Retrying repair of: './wpcomm305/comm305_options' including modifying data file 2020-02-01 0:06:05 20 [Note] mysqld: Making backup of data file ./wpcomm305/comm305_options.MYD with extension '-200201000605.BAK' 2020-02-01 0:06:05 20 [Note] Found 283 of 282 rows when repairing './wpcomm305/comm305_options' 2020-02-01 0:06:07 22 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-02-01 0:06:07 22 [Warning] Checking table: './wpcomm303/wp_options' 2020-02-01 0:06:07 24 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:06:07 24 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:06:07 25 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:06:07 25 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:06:08 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=8 max_threads=502 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f70a4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f70bbf1dcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x564b7d83ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x564b7d2d63af] sigaction.c:0(__restore_rt)[0x7f711d186630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564b7d204151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564b7d342545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564b7d0f8ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x564b7d0eed4b] /usr/sbin/mysqld(+0x6ca792)[0x564b7d0f9792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x564b7d1216da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564b7d12365d] /usr/sbin/mysqld(+0x65e3f3)[0x564b7d08d3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x564b7d08ccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x564b7d120b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564b7d12365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x564b7d123761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x564b7d124377] /usr/sbin/mysqld(+0x58fd84)[0x564b7cfbed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x564b7d0cbb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564b7d0e3b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564b7d0e3cd2] /usr/sbin/mysqld(+0x6b55e7)[0x564b7d0e45e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564b7d0e4677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x564b7d0d356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x564b7d0d507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x564b7d1b0afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x564b7d1b0bdd] pthread_create.c:0(start_thread)[0x7f711d17eea5] /lib64/libc.so.6(clone+0x6d)[0x7f711b51f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f70a400fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 26 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:06:14 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:06:14 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:06:14 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:06:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:06:14 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:06:14 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:06:14 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:06:14 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:06:14 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:06:14 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137038408 2020-02-01 0:06:14 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:06:14 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:06:14 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:06:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:06:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:06:14 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:06:15 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137038417; transaction id 5903735 2020-02-01 0:06:15 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:06:15 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:06:15 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:06:15 0 [Note] Starting crash recovery... 2020-02-01 0:06:15 0 [Note] Crash recovery finished. 2020-02-01 0:06:15 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:06:15 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:06:15 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:06:15 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000048' at position 401381 2020-02-01 0:06:15 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:06:15 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:06:15 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000048' at position 401381 2020-02-01 0:06:15 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:06:15 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:06:15 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:06:15 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:06:15 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:06:16 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:06:16 2020-02-01 0:06:24 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:06:24 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000049' at position 44758 2020-02-01 0:06:24 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:06:29 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000092, 1180840) 2020-02-01 0:06:40 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:06:41 14 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-02-01 0:06:41 14 [Warning] Checking table: './pizzahut/wp_options' 2020-02-01 0:06:41 15 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-02-01 0:06:41 15 [Warning] Checking table: './openrecords/wp_options' 2020-02-01 0:06:42 18 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:06:42 18 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:06:42 19 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:06:42 19 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:06:43 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9718000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f9731bc4cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5598cc083f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5598cbb1a3af] sigaction.c:0(__restore_rt)[0x7f9792cdd630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5598cba48151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5598cbb86545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5598cb93cccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5598cb932d4b] /usr/sbin/mysqld(+0x6ca792)[0x5598cb93d792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5598cb9656da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5598cb96765d] /usr/sbin/mysqld(+0x65e3f3)[0x5598cb8d13f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5598cb8d0cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5598cb964b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5598cb96765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5598cb967761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5598cb968377] /usr/sbin/mysqld(+0x58fd84)[0x5598cb802d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5598cb90fb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5598cb927b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5598cb927cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5598cb9285e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5598cb928677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5598cb91756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5598cb91907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5598cb9f4afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5598cb9f4bdd] pthread_create.c:0(start_thread)[0x7f9792cd5ea5] /lib64/libc.so.6(clone+0x6d)[0x7f97910768cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f9718010a30): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 20 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:06:49 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:06:49 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:06:49 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:06:49 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:06:49 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:06:49 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:06:49 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:06:49 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:06:49 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:06:49 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137041255 2020-02-01 0:06:49 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:06:49 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:06:49 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:06:49 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:06:49 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:06:49 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:06:49 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:06:49 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137041264; transaction id 5903746 2020-02-01 0:06:49 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:06:49 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:06:49 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:06:49 0 [Note] Starting crash recovery... 2020-02-01 0:06:49 0 [Note] Crash recovery finished. 2020-02-01 0:06:49 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:06:49 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:06:49 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:06:49 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000049' at position 44758 2020-02-01 0:06:49 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:06:49 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:06:49 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:06:49 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:06:49 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:06:49 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:06:49 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:06:49 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000049' at position 44758 2020-02-01 0:06:51 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:06:51 2020-02-01 0:06:59 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:06:59 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000050' at position 32443 2020-02-01 0:07:00 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:07:05 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000093, 34846) 2020-02-01 0:07:15 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:07:18 15 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:07:18 15 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:07:18 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:07:18 17 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:07:18 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:07:18 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:07:19 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f7c3c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f7c90a8ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560b654f6f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560b64f8d3af] sigaction.c:0(__restore_rt)[0x7f7cabcc8630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x560b64ebb151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x560b64ff9545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x560b64dafccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x560b64da5d4b] /usr/sbin/mysqld(+0x6ca792)[0x560b64db0792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x560b64dd86da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560b64dda65d] /usr/sbin/mysqld(+0x65e3f3)[0x560b64d443f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x560b64d43cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x560b64dd7b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560b64dda65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x560b64dda761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x560b64ddb377] /usr/sbin/mysqld(+0x58fd84)[0x560b64c75d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x560b64d82b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x560b64d9ab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x560b64d9acd2] /usr/sbin/mysqld(+0x6b55e7)[0x560b64d9b5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x560b64d9b677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x560b64d8a56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x560b64d8c07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x560b64e67afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560b64e67bdd] pthread_create.c:0(start_thread)[0x7f7cabcc0ea5] /lib64/libc.so.6(clone+0x6d)[0x7f7caa0618cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f7c3c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:07:25 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:07:25 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:07:25 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:07:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:07:25 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:07:25 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:07:25 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:07:25 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:07:25 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:07:25 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137044235 2020-02-01 0:07:25 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:07:25 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:07:25 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:07:25 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:07:25 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:07:25 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:07:25 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:07:25 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137044244; transaction id 5903757 2020-02-01 0:07:25 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:07:25 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:07:25 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:07:25 0 [Note] Starting crash recovery... 2020-02-01 0:07:25 0 [Note] Crash recovery finished. 2020-02-01 0:07:25 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:07:25 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:07:25 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:07:25 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000050' at position 32443 2020-02-01 0:07:25 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:07:25 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:07:25 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000050' at position 32443 2020-02-01 0:07:25 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:07:25 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:07:25 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:07:25 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:07:25 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:07:26 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:07:26 2020-02-01 0:07:31 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:07:31 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000051' at position 1464 2020-02-01 0:07:31 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:07:37 17 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000094, 1464) 2020-02-01 0:07:50 18 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:07:51 20 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:07:51 20 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:07:51 21 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:07:51 21 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:07:51 22 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:07:51 22 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:07:51 22 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:07:51 22 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:07:51 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=9 max_threads=502 thread_count=17 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x556b97f38728 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f9c880a1cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x556b93ba5f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x556b9363c3af] sigaction.c:0(__restore_rt)[0x7f9cf4235630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x556b9356a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x556b936a8545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x556b9345eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x556b93454d4b] /usr/sbin/mysqld(+0x6ca792)[0x556b9345f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x556b934876da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556b9348965d] /usr/sbin/mysqld(+0x65e3f3)[0x556b933f33f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x556b933f2cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x556b93486b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556b9348965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x556b93489761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x556b9348a377] /usr/sbin/mysqld(+0x58fd84)[0x556b93324d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x556b93431b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x556b93449b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x556b93449cd2] /usr/sbin/mysqld(+0x6b55e7)[0x556b9344a5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x556b9344a677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x556b9343956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x556b9343b07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x556b93516afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x556b93516bdd] pthread_create.c:0(start_thread)[0x7f9cf422dea5] /lib64/libc.so.6(clone+0x6d)[0x7f9cf25ce8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x556b97f46e00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 26 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:07:58 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:07:58 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:07:58 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:07:58 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:07:58 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:07:58 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:07:58 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:07:58 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:07:58 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:07:58 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137047098 2020-02-01 0:07:58 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:07:58 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:07:58 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:07:58 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:07:58 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:07:58 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:07:58 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:07:58 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137047107; transaction id 5903768 2020-02-01 0:07:58 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:07:58 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:07:58 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:07:58 0 [Note] Starting crash recovery... 2020-02-01 0:07:58 0 [Note] Crash recovery finished. 2020-02-01 0:07:58 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:07:58 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:07:58 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:07:58 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000051' at position 1464 2020-02-01 0:07:58 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:07:58 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:07:58 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000051' at position 1464 2020-02-01 0:07:58 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:07:58 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:07:58 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:07:58 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:07:58 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:07:59 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:07:59 2020-02-01 0:08:13 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:08:13 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000052' at position 240730 2020-02-01 0:08:13 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:08:19 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000095, 3170) 2020-02-01 0:08:26 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:08:26 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:08:26 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:08:26 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:08:26 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:08:26 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff8f0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff95c04fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56257dbf6f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56257d68d3af] sigaction.c:0(__restore_rt)[0x7ff962211630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56257d5bb151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56257d6f9545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56257d4afccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56257d4a5d4b] /usr/sbin/mysqld(+0x6ca792)[0x56257d4b0792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56257d4d86da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56257d4da65d] /usr/sbin/mysqld(+0x65e3f3)[0x56257d4443f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56257d443cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56257d4d7b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56257d4da65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56257d4da761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56257d4db377] /usr/sbin/mysqld(+0x58fd84)[0x56257d375d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56257d482b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56257d49ab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56257d49acd2] /usr/sbin/mysqld(+0x6b55e7)[0x56257d49b5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56257d49b677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56257d48a56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56257d48c07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56257d567afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56257d567bdd] pthread_create.c:0(start_thread)[0x7ff962209ea5] /lib64/libc.so.6(clone+0x6d)[0x7ff9605aa8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff8f000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:08:32 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:08:32 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:08:32 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:08:32 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:08:32 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:08:32 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:08:32 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:08:32 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:08:32 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:08:32 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137049945 2020-02-01 0:08:32 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:08:32 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:08:32 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:08:32 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:08:32 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:08:32 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:08:32 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:08:33 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137049954; transaction id 5903779 2020-02-01 0:08:33 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:08:33 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:08:33 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:08:33 0 [Note] Starting crash recovery... 2020-02-01 0:08:33 0 [Note] Crash recovery finished. 2020-02-01 0:08:33 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:08:33 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:08:33 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:08:33 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000052' at position 240730 2020-02-01 0:08:33 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:08:33 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:08:33 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000052' at position 240730 2020-02-01 0:08:33 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:08:33 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:08:33 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:08:33 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:08:33 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:08:34 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:08:34 2020-02-01 0:08:53 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:08:53 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000053' at position 1703404 2020-02-01 0:08:53 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:08:58 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000096, 1464) 2020-02-01 0:09:01 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:09:03 17 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-02-01 0:09:03 17 [Warning] Checking table: './wpcomm305/comm305_options' 2020-02-01 0:09:04 15 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 0:09:04 15 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-02-01 0:09:05 19 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:09:05 19 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:09:05 20 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:09:05 20 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:09:05 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f3d14000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f3d64549cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55626db23f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55626d5ba3af] sigaction.c:0(__restore_rt)[0x7f3d81747630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55626d4e8151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55626d626545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55626d3dcccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55626d3d2d4b] /usr/sbin/mysqld(+0x6ca792)[0x55626d3dd792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55626d4056da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55626d40765d] /usr/sbin/mysqld(+0x65e3f3)[0x55626d3713f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55626d370cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55626d404b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55626d40765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55626d407761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55626d408377] /usr/sbin/mysqld(+0x58fd84)[0x55626d2a2d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55626d3afb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55626d3c7b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55626d3c7cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55626d3c85e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55626d3c8677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55626d3b756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55626d3b907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55626d494afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55626d494bdd] pthread_create.c:0(start_thread)[0x7f3d8173fea5] /lib64/libc.so.6(clone+0x6d)[0x7f3d7fae08cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f3d1400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 21 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:09:11 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:09:11 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:09:11 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:09:11 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:09:11 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:09:11 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:09:11 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:09:11 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:09:11 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:09:11 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137052808 2020-02-01 0:09:11 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:09:11 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:09:11 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:09:11 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:09:11 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:09:11 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:09:12 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:09:12 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137052817; transaction id 5903790 2020-02-01 0:09:12 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:09:12 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:09:12 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:09:12 0 [Note] Starting crash recovery... 2020-02-01 0:09:12 0 [Note] Crash recovery finished. 2020-02-01 0:09:12 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:09:12 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:09:12 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:09:12 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000053' at position 1703404 2020-02-01 0:09:12 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:09:12 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:09:12 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000053' at position 1703404 2020-02-01 0:09:12 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:09:12 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:09:12 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:09:12 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:09:12 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:09:12 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:09:12 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:09:12 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:09:12 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:09:12 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:09:13 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:09:13 2020-02-01 0:09:35 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:09:35 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000054' at position 2430 2020-02-01 0:09:35 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:09:36 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:09:41 14 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000097, 598933) 2020-02-01 0:09:41 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:09:41 17 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:09:41 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:09:41 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:09:41 18 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:09:41 18 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:09:42 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9790000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f97e5787cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a0a1ba8f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a0a163f3af] sigaction.c:0(__restore_rt)[0x7f980098c630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a0a156d151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a0a16ab545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a0a1461ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a0a1457d4b] /usr/sbin/mysqld(+0x6ca792)[0x55a0a1462792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a0a148a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a0a148c65d] /usr/sbin/mysqld(+0x65e3f3)[0x55a0a13f63f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a0a13f5cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a0a1489b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a0a148c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a0a148c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a0a148d377] /usr/sbin/mysqld(+0x58fd84)[0x55a0a1327d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a0a1434b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a0a144cb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a0a144ccd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a0a144d5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a0a144d677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a0a143c56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a0a143e07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a0a1519afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a0a1519bdd] pthread_create.c:0(start_thread)[0x7f9800984ea5] /lib64/libc.so.6(clone+0x6d)[0x7f97fed258cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f979000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:09:48 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:09:48 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:09:48 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:09:48 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:09:48 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:09:48 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:09:48 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:09:48 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:09:48 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:09:48 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137055655 2020-02-01 0:09:48 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:09:48 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:09:48 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:09:48 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:09:48 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:09:48 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:09:48 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:09:48 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137055664; transaction id 5903801 2020-02-01 0:09:48 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:09:48 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:09:48 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:09:48 0 [Note] Starting crash recovery... 2020-02-01 0:09:48 0 [Note] Crash recovery finished. 2020-02-01 0:09:48 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:09:48 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:09:48 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:09:48 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000054' at position 2430 2020-02-01 0:09:48 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:09:48 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:09:48 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:09:48 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:09:48 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:09:48 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:09:48 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:09:48 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000054' at position 2430 2020-02-01 0:09:49 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:09:49 2020-02-01 0:10:16 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:10:18 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:10:18 14 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:10:21 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:10:21 17 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:10:21 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:10:21 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:10:21 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f29b0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f29c814ccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c531cd4f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c53176b3af] sigaction.c:0(__restore_rt)[0x7f2a35386630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c531699151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c5317d7545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c53158dccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c531583d4b] /usr/sbin/mysqld(+0x6ca792)[0x55c53158e792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c5315b66da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c5315b865d] /usr/sbin/mysqld(+0x65e3f3)[0x55c5315223f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c531521cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c5315b5b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c5315b865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c5315b8761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c5315b9377] /usr/sbin/mysqld(+0x58fd84)[0x55c531453d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c531560b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c531578b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c531578cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c5315795e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c531579677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c53156856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c53156a07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c531645afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c531645bdd] pthread_create.c:0(start_thread)[0x7f2a3537eea5] /lib64/libc.so.6(clone+0x6d)[0x7f2a3371f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f29b000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 20 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:10:28 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:10:28 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:10:28 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:10:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:10:28 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:10:28 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:10:28 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:10:28 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:10:28 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:10:28 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137058518 2020-02-01 0:10:28 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:10:28 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:10:28 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:10:28 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:10:28 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:10:28 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:10:28 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:10:28 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137058527; transaction id 5903812 2020-02-01 0:10:28 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:10:28 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:10:28 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:10:28 0 [Note] Starting crash recovery... 2020-02-01 0:10:28 0 [Note] Crash recovery finished. 2020-02-01 0:10:28 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:10:28 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:10:28 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:10:28 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 346 2020-02-01 0:10:28 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:10:28 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:10:28 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 346 2020-02-01 0:10:28 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:10:28 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:10:28 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:10:28 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:10:28 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:10:29 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:10:29 2020-02-01 0:10:32 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:10:32 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000055' at position 1464 2020-02-01 0:10:32 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:10:38 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000098, 5045) 2020-02-01 0:10:57 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:11:00 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:11:00 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:11:00 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:11:00 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:11:00 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fdc10000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fdc6c62acf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5625530f4f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x562552b8b3af] sigaction.c:0(__restore_rt)[0x7fdc897a7630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x562552ab9151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x562552bf7545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5625529adccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5625529a3d4b] /usr/sbin/mysqld(+0x6ca792)[0x5625529ae792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5625529d66da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5625529d865d] /usr/sbin/mysqld(+0x65e3f3)[0x5625529423f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x562552941cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5625529d5b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5625529d865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5625529d8761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5625529d9377] /usr/sbin/mysqld(+0x58fd84)[0x562552873d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x562552980b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x562552998b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x562552998cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5625529995e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x562552999677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56255298856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56255298a07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x562552a65afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x562552a65bdd] pthread_create.c:0(start_thread)[0x7fdc8979fea5] /lib64/libc.so.6(clone+0x6d)[0x7fdc87b408cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fdc1000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:11:06 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:11:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:11:06 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:11:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:11:06 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:11:06 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:11:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:11:06 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:11:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:11:06 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137061390 2020-02-01 0:11:06 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:11:06 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:11:06 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:11:06 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:11:06 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:11:06 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:11:06 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137061399; transaction id 5903823 2020-02-01 0:11:06 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:11:06 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:11:06 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:11:06 0 [Note] Starting crash recovery... 2020-02-01 0:11:06 0 [Note] Crash recovery finished. 2020-02-01 0:11:06 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:11:06 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:11:06 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:11:06 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000055' at position 1464 2020-02-01 0:11:06 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:11:06 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:11:06 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:11:06 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:11:06 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:11:06 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:11:06 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:11:06 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000055' at position 1464 2020-02-01 0:11:08 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:11:08 2020-02-01 0:11:08 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:11:08 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000056' at position 2513 2020-02-01 0:11:08 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:11:13 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000100, 1464) 2020-02-01 0:11:32 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:11:34 15 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 0:11:34 15 [Warning] Checking table: './wpcomm306/wp_options' 2020-02-01 0:11:36 19 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:11:36 19 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:11:43 25 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:11:43 25 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:11:44 26 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:11:44 26 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:11:44 26 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:11:44 26 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:11:44 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7efeec006bf8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7eff506e7cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d2fb717f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55d2fb1ae3af] sigaction.c:0(__restore_rt)[0x7eff6d830630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55d2fb0dc151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55d2fb21a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55d2fafd0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55d2fafc6d4b] /usr/sbin/mysqld(+0x6ca792)[0x55d2fafd1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55d2faff96da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d2faffb65d] /usr/sbin/mysqld(+0x65e3f3)[0x55d2faf653f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55d2faf64cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55d2faff8b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d2faffb65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55d2faffb761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55d2faffc377] /usr/sbin/mysqld(+0x58fd84)[0x55d2fae96d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55d2fafa3b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55d2fafbbb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55d2fafbbcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55d2fafbc5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55d2fafbc677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55d2fafab56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55d2fafad07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55d2fb088afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d2fb088bdd] pthread_create.c:0(start_thread)[0x7eff6d828ea5] /lib64/libc.so.6(clone+0x6d)[0x7eff6bbc98cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7efeec010ea0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 27 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:11:50 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:11:50 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:11:50 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:11:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:11:50 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:11:50 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:11:50 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:11:50 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:11:50 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:11:50 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137064768 2020-02-01 0:11:50 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:11:50 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:11:50 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:11:50 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:11:50 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:11:50 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:11:50 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:11:50 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137064777; transaction id 5903839 2020-02-01 0:11:50 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:11:50 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:11:50 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:11:50 0 [Note] Starting crash recovery... 2020-02-01 0:11:50 0 [Note] Crash recovery finished. 2020-02-01 0:11:50 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:11:50 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:11:50 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:11:50 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000056' at position 2513 2020-02-01 0:11:50 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:11:50 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:11:50 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:11:50 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:11:50 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:11:50 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:11:50 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000056' at position 2513 2020-02-01 0:11:50 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:11:50 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:11:50 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:11:50 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:11:50 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:11:50 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:11:51 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:11:51 2020-02-01 0:11:51 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:11:51 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000057' at position 1464 2020-02-01 0:11:51 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:11:57 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000101, 81362) 2020-02-01 0:12:17 19 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:12:22 22 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 0:12:22 22 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-02-01 0:12:28 26 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:12:28 26 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:12:28 27 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:12:28 27 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:12:29 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc910000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc96cfa9cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b6be6bbf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55b6be1523af] sigaction.c:0(__restore_rt)[0x7fc988117630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55b6be080151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55b6be1be545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55b6bdf74ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55b6bdf6ad4b] /usr/sbin/mysqld(+0x6ca792)[0x55b6bdf75792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55b6bdf9d6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b6bdf9f65d] /usr/sbin/mysqld(+0x65e3f3)[0x55b6bdf093f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55b6bdf08cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55b6bdf9cb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b6bdf9f65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55b6bdf9f761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55b6bdfa0377] /usr/sbin/mysqld(+0x58fd84)[0x55b6bde3ad84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55b6bdf47b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55b6bdf5fb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55b6bdf5fcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55b6bdf605e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55b6bdf60677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55b6bdf4f56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55b6bdf5107c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55b6be02cafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55b6be02cbdd] pthread_create.c:0(start_thread)[0x7fc98810fea5] /lib64/libc.so.6(clone+0x6d)[0x7fc9864b08cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc91000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 28 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:12:35 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:12:35 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:12:35 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:12:35 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:12:35 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:12:35 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:12:35 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:12:35 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:12:35 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:12:35 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137068564 2020-02-01 0:12:35 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:12:35 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:12:35 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:12:35 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:12:35 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:12:35 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:12:35 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:12:35 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137068573; transaction id 5903850 2020-02-01 0:12:35 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:12:35 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:12:35 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:12:35 0 [Note] Starting crash recovery... 2020-02-01 0:12:35 0 [Note] Crash recovery finished. 2020-02-01 0:12:35 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:12:35 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:12:35 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:12:35 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000057' at position 1464 2020-02-01 0:12:35 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:12:35 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:12:35 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000057' at position 1464 2020-02-01 0:12:35 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:12:35 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:12:35 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:12:35 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:12:35 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:12:35 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:12:35 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:12:35 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:12:35 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:12:35 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:12:36 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:12:36 2020-02-01 0:12:38 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:12:38 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000058' at position 120017 2020-02-01 0:12:38 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:12:44 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000102, 23638) 2020-02-01 0:13:02 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:13:03 14 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-02-01 0:13:03 14 [Warning] Checking table: './wpcomm305/comm305_options' 2020-02-01 0:13:06 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:13:06 16 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:13:06 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:13:06 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:13:06 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f4778000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f47dc3b8cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560cdbe44f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560cdb8db3af] sigaction.c:0(__restore_rt)[0x7f47f954b630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x560cdb809151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x560cdb947545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x560cdb6fdccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x560cdb6f3d4b] /usr/sbin/mysqld(+0x6ca792)[0x560cdb6fe792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x560cdb7266da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560cdb72865d] /usr/sbin/mysqld(+0x65e3f3)[0x560cdb6923f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x560cdb691cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x560cdb725b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560cdb72865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x560cdb728761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x560cdb729377] /usr/sbin/mysqld(+0x58fd84)[0x560cdb5c3d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x560cdb6d0b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x560cdb6e8b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x560cdb6e8cd2] /usr/sbin/mysqld(+0x6b55e7)[0x560cdb6e95e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x560cdb6e9677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x560cdb6d856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x560cdb6da07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x560cdb7b5afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560cdb7b5bdd] pthread_create.c:0(start_thread)[0x7f47f9543ea5] /lib64/libc.so.6(clone+0x6d)[0x7f47f78e48cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f477800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:13:12 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:13:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:13:12 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:13:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:13:12 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:13:12 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:13:12 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:13:12 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:13:12 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:13:12 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137071410 2020-02-01 0:13:12 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:13:12 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:13:12 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:13:12 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:13:12 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:13:12 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:13:12 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:13:12 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137071419; transaction id 5903861 2020-02-01 0:13:12 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:13:12 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:13:12 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:13:12 0 [Note] Starting crash recovery... 2020-02-01 0:13:12 0 [Note] Crash recovery finished. 2020-02-01 0:13:12 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:13:12 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:13:12 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:13:12 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000058' at position 120017 2020-02-01 0:13:12 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:13:12 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:13:12 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000058' at position 120017 2020-02-01 0:13:12 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:13:12 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:13:12 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:13:12 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:13:12 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:13:13 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:13:13 2020-02-01 0:13:17 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:13:17 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000059' at position 41630 2020-02-01 0:13:17 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:13:23 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000103, 7083) 2020-02-01 0:13:38 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:13:38 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:13:38 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:13:39 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:13:39 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:13:39 16 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:13:39 16 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:13:39 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff9e8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ffa5404bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x558f9198cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x558f914233af] sigaction.c:0(__restore_rt)[0x7ffa5ac1a630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x558f91351151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x558f9148f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x558f91245ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x558f9123bd4b] /usr/sbin/mysqld(+0x6ca792)[0x558f91246792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x558f9126e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x558f9127065d] /usr/sbin/mysqld(+0x65e3f3)[0x558f911da3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x558f911d9cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x558f9126db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x558f9127065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x558f91270761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x558f91271377] /usr/sbin/mysqld(+0x58fd84)[0x558f9110bd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x558f91218b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x558f91230b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x558f91230cd2] /usr/sbin/mysqld(+0x6b55e7)[0x558f912315e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x558f91231677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x558f9122056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x558f9122207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x558f912fdafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x558f912fdbdd] pthread_create.c:0(start_thread)[0x7ffa5ac12ea5] /lib64/libc.so.6(clone+0x6d)[0x7ffa58fb38cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff9e800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:13:45 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:13:45 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:13:45 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:13:45 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:13:45 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:13:45 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:13:45 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:13:45 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:13:45 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:13:45 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137074273 2020-02-01 0:13:45 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:13:45 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:13:45 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:13:45 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:13:45 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:13:45 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:13:45 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:13:45 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137074282; transaction id 5903872 2020-02-01 0:13:45 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:13:45 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:13:45 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:13:45 0 [Note] Starting crash recovery... 2020-02-01 0:13:45 0 [Note] Crash recovery finished. 2020-02-01 0:13:45 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:13:45 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:13:45 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:13:45 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000059' at position 41630 2020-02-01 0:13:45 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:13:45 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:13:45 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000059' at position 41630 2020-02-01 0:13:45 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:13:45 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:13:45 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:13:45 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:13:45 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:13:46 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:13:46 2020-02-01 0:13:51 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:13:51 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000060' at position 115602 2020-02-01 0:13:51 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:13:57 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000104, 2513) 2020-02-01 0:14:13 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:14:15 14 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-02-01 0:14:15 14 [Warning] Checking table: './wpcomm303/wp_options' 2020-02-01 0:14:16 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:14:16 16 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:14:16 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:14:16 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:14:16 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fca14000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fca70904cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5562dd49ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5562dcf363af] sigaction.c:0(__restore_rt)[0x7fca8daaf630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5562dce64151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5562dcfa2545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5562dcd58ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5562dcd4ed4b] /usr/sbin/mysqld(+0x6ca792)[0x5562dcd59792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5562dcd816da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5562dcd8365d] /usr/sbin/mysqld(+0x65e3f3)[0x5562dcced3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5562dcceccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5562dcd80b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5562dcd8365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5562dcd83761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5562dcd84377] /usr/sbin/mysqld(+0x58fd84)[0x5562dcc1ed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5562dcd2bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5562dcd43b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5562dcd43cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5562dcd445e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5562dcd44677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5562dcd3356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5562dcd3507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5562dce10afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5562dce10bdd] pthread_create.c:0(start_thread)[0x7fca8daa7ea5] /lib64/libc.so.6(clone+0x6d)[0x7fca8be488cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fca1400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:14:22 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:14:22 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:14:22 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:14:22 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:14:22 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:14:22 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:14:22 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:14:22 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:14:22 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:14:22 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137077120 2020-02-01 0:14:22 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:14:23 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:14:23 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:14:23 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:14:23 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:14:23 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:14:23 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:14:23 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137077129; transaction id 5903883 2020-02-01 0:14:23 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:14:23 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:14:23 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:14:23 0 [Note] Starting crash recovery... 2020-02-01 0:14:23 0 [Note] Crash recovery finished. 2020-02-01 0:14:23 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:14:23 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:14:23 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:14:23 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000060' at position 115602 2020-02-01 0:14:23 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:14:23 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:14:23 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000060' at position 115602 2020-02-01 0:14:23 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:14:23 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:14:23 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:14:23 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:14:23 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:14:24 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:14:24 2020-02-01 0:14:36 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:14:36 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000061' at position 744460 2020-02-01 0:14:36 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:14:41 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000105, 6948) 2020-02-01 0:14:48 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:14:50 16 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:14:50 16 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:14:52 19 [ERROR] mysqld: Table './ksarn/wp_bp_activity' is marked as crashed and should be repaired 2020-02-01 0:14:52 19 [Warning] Checking table: './ksarn/wp_bp_activity' 2020-02-01 0:14:52 19 [ERROR] mysqld: Table './ksarn/wp_bp_activity_meta' is marked as crashed and should be repaired 2020-02-01 0:14:52 19 [Warning] Checking table: './ksarn/wp_bp_activity_meta' 2020-02-01 0:14:52 19 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity' is marked as crashed and should be repaired 2020-02-01 0:14:52 19 [Warning] Checking table: './ksarn/wp_learndash_user_activity' 2020-02-01 0:14:52 19 [ERROR] mysqld: Table './ksarn/wp_learndash_user_activity_meta' is marked as crashed and should be repaired 2020-02-01 0:14:52 19 [Warning] Checking table: './ksarn/wp_learndash_user_activity_meta' 2020-02-01 0:14:52 19 [ERROR] mysqld: Table './ksarn/wp_wp_pro_quiz_statistic' is marked as crashed and should be repaired 2020-02-01 0:14:52 19 [Warning] Checking table: './ksarn/wp_wp_pro_quiz_statistic' 2020-02-01 0:14:52 19 [ERROR] mysqld: Table './ksarn/wp_wp_pro_quiz_statistic_ref' is marked as crashed and should be repaired 2020-02-01 0:14:52 19 [Warning] Checking table: './ksarn/wp_wp_pro_quiz_statistic_ref' 2020-02-01 0:14:54 22 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-02-01 0:14:54 22 [Warning] Checking table: './openrecords/wp_options' 2020-02-01 0:15:07 45 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:15:07 45 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:15:07 46 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:15:07 46 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:15:07 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f99d4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f997df1dcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55996dba3f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55996d63a3af] sigaction.c:0(__restore_rt)[0x7f99df190630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55996d568151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55996d6a6545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55996d45cccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55996d452d4b] /usr/sbin/mysqld(+0x6ca792)[0x55996d45d792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55996d4856da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55996d48765d] /usr/sbin/mysqld(+0x65e3f3)[0x55996d3f13f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55996d3f0cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55996d484b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55996d48765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55996d487761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55996d488377] /usr/sbin/mysqld(+0x58fd84)[0x55996d322d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55996d42fb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55996d447b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55996d447cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55996d4485e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55996d448677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55996d43756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55996d43907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55996d514afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55996d514bdd] pthread_create.c:0(start_thread)[0x7f99df188ea5] /lib64/libc.so.6(clone+0x6d)[0x7f99dd5298cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f99d400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 48 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:15:14 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:15:14 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:15:14 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:15:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:15:14 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:15:14 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:15:14 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:15:14 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:15:14 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:15:14 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137088707 2020-02-01 0:15:14 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:15:14 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:15:14 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:15:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:15:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:15:14 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:15:14 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:15:14 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137088716; transaction id 5903894 2020-02-01 0:15:14 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:15:14 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:15:14 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:15:14 0 [Note] Starting crash recovery... 2020-02-01 0:15:14 0 [Note] Crash recovery finished. 2020-02-01 0:15:14 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:15:14 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:15:14 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:15:14 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000061' at position 744460 2020-02-01 0:15:14 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:15:14 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:15:14 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000061' at position 744460 2020-02-01 0:15:14 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:15:14 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:15:14 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:15:14 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:15:14 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:15:14 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:15:14 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:15:14 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:15:14 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:15:14 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:15:15 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:15:15 2020-02-01 0:15:16 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:15:16 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000062' at position 1464 2020-02-01 0:15:16 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:15:22 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000106, 400214) 2020-02-01 0:15:43 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:15:44 14 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:15:44 14 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:15:45 15 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:15:45 15 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:15:45 14 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 0:15:45 14 [Warning] Checking table: './wpms/wp_usermeta' 2020-02-01 0:15:47 18 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 0:15:47 18 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-02-01 0:15:50 25 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 0:15:50 25 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 0:15:53 28 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:15:53 28 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:15:53 29 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:15:53 29 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:15:53 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f8074000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f80d8e7dcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5626353b1f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x562634e483af] sigaction.c:0(__restore_rt)[0x7f80f4010630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x562634d76151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x562634eb4545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x562634c6accf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x562634c60d4b] /usr/sbin/mysqld(+0x6ca792)[0x562634c6b792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x562634c936da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x562634c9565d] /usr/sbin/mysqld(+0x65e3f3)[0x562634bff3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x562634bfecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x562634c92b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x562634c9565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x562634c95761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x562634c96377] /usr/sbin/mysqld(+0x58fd84)[0x562634b30d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x562634c3db6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x562634c55b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x562634c55cd2] /usr/sbin/mysqld(+0x6b55e7)[0x562634c565e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x562634c56677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x562634c4556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x562634c4707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x562634d22afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x562634d22bdd] pthread_create.c:0(start_thread)[0x7f80f4008ea5] /lib64/libc.so.6(clone+0x6d)[0x7f80f23a98cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f807400feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 30 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:15:59 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:15:59 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:15:59 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:15:59 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:15:59 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:15:59 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:15:59 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:15:59 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:15:59 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:15:59 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137091526 2020-02-01 0:15:59 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:16:00 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:16:00 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:16:00 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:16:00 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:16:00 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:16:00 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:16:00 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137091535; transaction id 5903905 2020-02-01 0:16:00 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:16:00 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:16:00 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:16:00 0 [Note] Starting crash recovery... 2020-02-01 0:16:00 0 [Note] Crash recovery finished. 2020-02-01 0:16:00 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:16:00 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:16:00 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:16:00 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000062' at position 1464 2020-02-01 0:16:00 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:16:00 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:16:00 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000062' at position 1464 2020-02-01 0:16:00 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:16:00 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:16:00 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:16:00 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:16:00 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:16:00 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:16:00 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:16:00 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:16:00 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:16:00 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:16:01 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:16:01 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000063' at position 23527 2020-02-01 0:16:01 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:16:01 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:16:01 2020-02-01 0:16:06 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000107, 58663) 2020-02-01 0:16:28 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:16:29 14 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 0:16:29 14 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-02-01 0:16:32 16 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 0:16:32 16 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 0:16:36 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:16:36 20 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:16:36 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:16:36 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:16:37 21 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:16:37 21 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:16:37 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe9e8000ac8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fea04ffccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e71d622f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e71d0b93af] sigaction.c:0(__restore_rt)[0x7fea67dad630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55e71cfe7151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55e71d125545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55e71cedbccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55e71ced1d4b] /usr/sbin/mysqld(+0x6ca792)[0x55e71cedc792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55e71cf046da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e71cf0665d] /usr/sbin/mysqld(+0x65e3f3)[0x55e71ce703f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55e71ce6fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55e71cf03b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e71cf0665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55e71cf06761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55e71cf07377] /usr/sbin/mysqld(+0x58fd84)[0x55e71cda1d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55e71ceaeb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55e71cec6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55e71cec6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55e71cec75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55e71cec7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55e71ceb656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55e71ceb807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55e71cf93afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e71cf93bdd] pthread_create.c:0(start_thread)[0x7fea67da5ea5] /lib64/libc.so.6(clone+0x6d)[0x7fea661468cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe9e800ff40): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:16:43 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:16:43 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:16:43 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:16:43 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:16:43 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:16:43 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:16:43 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:16:43 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:16:43 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:16:43 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137094389 2020-02-01 0:16:43 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:16:44 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:16:44 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:16:44 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:16:44 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:16:44 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:16:44 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:16:44 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137094398; transaction id 5903916 2020-02-01 0:16:44 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:16:44 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:16:44 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:16:44 0 [Note] Starting crash recovery... 2020-02-01 0:16:44 0 [Note] Crash recovery finished. 2020-02-01 0:16:44 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:16:44 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:16:44 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:16:44 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000063' at position 23527 2020-02-01 0:16:44 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:16:44 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:16:44 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:16:44 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000063' at position 23527 2020-02-01 0:16:44 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:16:44 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:16:44 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:16:44 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:16:44 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:16:44 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:16:44 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:16:44 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:16:44 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:16:45 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:16:45 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000064' at position 1454 2020-02-01 0:16:45 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:16:45 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:16:45 2020-02-01 0:16:50 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000108, 31491) 2020-02-01 0:17:09 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:17:13 17 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-02-01 0:17:13 17 [Warning] Checking table: './itstraining/wp_options' 2020-02-01 0:17:14 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:17:14 20 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:17:14 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:17:14 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:17:14 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f3878000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f38d934dcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56200ac4cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56200a6e33af] sigaction.c:0(__restore_rt)[0x7f38f4482630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56200a611151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56200a74f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56200a505ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56200a4fbd4b] /usr/sbin/mysqld(+0x6ca792)[0x56200a506792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56200a52e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56200a53065d] /usr/sbin/mysqld(+0x65e3f3)[0x56200a49a3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56200a499cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56200a52db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56200a53065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56200a530761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56200a531377] /usr/sbin/mysqld(+0x58fd84)[0x56200a3cbd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56200a4d8b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56200a4f0b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56200a4f0cd2] /usr/sbin/mysqld(+0x6b55e7)[0x56200a4f15e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56200a4f1677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56200a4e056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56200a4e207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56200a5bdafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56200a5bdbdd] pthread_create.c:0(start_thread)[0x7f38f447aea5] /lib64/libc.so.6(clone+0x6d)[0x7f38f281b8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f387800feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:17:20 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:17:20 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:17:20 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:17:20 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:17:20 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:17:20 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:17:20 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:17:20 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:17:20 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:17:20 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137097369 2020-02-01 0:17:20 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:17:21 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:17:21 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:17:21 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:17:21 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:17:21 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:17:21 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:17:21 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137097378; transaction id 5903927 2020-02-01 0:17:21 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:17:21 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:17:21 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:17:21 0 [Note] Starting crash recovery... 2020-02-01 0:17:21 0 [Note] Crash recovery finished. 2020-02-01 0:17:21 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:17:21 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:17:21 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:17:21 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000064' at position 1454 2020-02-01 0:17:21 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:17:21 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:17:21 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000064' at position 1454 2020-02-01 0:17:21 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:17:21 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:17:21 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:17:21 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:17:21 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:17:22 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:17:22 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000065' at position 2513 2020-02-01 0:17:22 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:17:22 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:17:22 2020-02-01 0:17:27 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000109, 7819) 2020-02-01 0:17:49 19 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:17:50 20 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 0:17:50 20 [Warning] Checking table: './wpcomm306/wp_options' 2020-02-01 0:17:52 23 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-02-01 0:17:52 23 [Warning] Checking table: './itstraining/wp_options' 2020-02-01 0:17:53 25 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-02-01 0:17:53 25 [Warning] Checking table: './wpcomm305/comm305_options' 2020-02-01 0:17:53 26 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-02-01 0:17:53 26 [Warning] Checking table: './wpcomm303/wp_options' 2020-02-01 0:17:53 27 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:17:53 27 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:17:58 32 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:17:58 32 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:17:58 33 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:17:58 33 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:17:58 33 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:17:58 33 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:17:58 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f824c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f82b00accf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d1c5bfef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55d1c56953af] sigaction.c:0(__restore_rt)[0x7f82ca96f630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55d1c55c3151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55d1c5701545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55d1c54b7ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55d1c54add4b] /usr/sbin/mysqld(+0x6ca792)[0x55d1c54b8792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55d1c54e06da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d1c54e265d] /usr/sbin/mysqld(+0x65e3f3)[0x55d1c544c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55d1c544bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55d1c54dfb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d1c54e265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55d1c54e2761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55d1c54e3377] /usr/sbin/mysqld(+0x58fd84)[0x55d1c537dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55d1c548ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55d1c54a2b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55d1c54a2cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55d1c54a35e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55d1c54a3677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55d1c549256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55d1c549407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55d1c556fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d1c556fbdd] pthread_create.c:0(start_thread)[0x7f82ca967ea5] /lib64/libc.so.6(clone+0x6d)[0x7f82c8d088cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f824c00feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 34 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:18:05 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:18:05 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:18:05 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:18:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:18:05 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:18:05 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:18:05 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:18:05 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:18:05 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:18:05 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137104724 2020-02-01 0:18:05 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:18:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:18:05 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:18:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:18:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:18:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:18:05 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137104733; transaction id 5903961 2020-02-01 0:18:05 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:18:05 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:18:05 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:18:05 0 [Note] Starting crash recovery... 2020-02-01 0:18:05 0 [Note] Crash recovery finished. 2020-02-01 0:18:05 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:18:05 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:18:05 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:18:05 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000065' at position 2513 2020-02-01 0:18:05 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:18:05 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:18:05 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000065' at position 2513 2020-02-01 0:18:05 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:18:05 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:18:05 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:18:05 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:18:05 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:18:05 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:18:05 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:18:05 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:18:05 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:18:05 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:18:06 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:18:06 2020-02-01 0:18:15 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:18:15 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000066' at position 3110 2020-02-01 0:18:15 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:18:21 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000110, 106319) 2020-02-01 0:18:34 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:18:35 17 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 0:18:35 17 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 0:18:39 24 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:18:39 24 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:18:39 25 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:18:39 25 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:18:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc8ec000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc88c14fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55cf3a70ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55cf3a1a53af] sigaction.c:0(__restore_rt)[0x7fc8f83fe630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55cf3a0d3151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55cf3a211545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55cf39fc7ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55cf39fbdd4b] /usr/sbin/mysqld(+0x6ca792)[0x55cf39fc8792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55cf39ff06da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55cf39ff265d] /usr/sbin/mysqld(+0x65e3f3)[0x55cf39f5c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55cf39f5bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55cf39fefb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55cf39ff265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55cf39ff2761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55cf39ff3377] /usr/sbin/mysqld(+0x58fd84)[0x55cf39e8dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55cf39f9ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55cf39fb2b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55cf39fb2cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55cf39fb35e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55cf39fb3677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55cf39fa256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55cf39fa407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55cf3a07fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55cf3a07fbdd] pthread_create.c:0(start_thread)[0x7fc8f83f6ea5] /lib64/libc.so.6(clone+0x6d)[0x7fc8f67978cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc8ec00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 27 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:18:46 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:18:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:18:46 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:18:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:18:46 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:18:46 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:18:46 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:18:46 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:18:46 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:18:46 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137108378 2020-02-01 0:18:46 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:18:46 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:18:46 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:18:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:18:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:18:46 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:18:46 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:18:46 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137108387; transaction id 5903972 2020-02-01 0:18:46 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:18:46 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:18:46 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:18:46 0 [Note] Starting crash recovery... 2020-02-01 0:18:46 0 [Note] Crash recovery finished. 2020-02-01 0:18:46 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:18:46 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:18:46 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:18:46 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000066' at position 3110 2020-02-01 0:18:46 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:18:46 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:18:46 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:18:46 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:18:46 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:18:46 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:18:46 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:18:46 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000066' at position 3110 2020-02-01 0:18:47 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:18:47 2020-02-01 0:18:52 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:18:52 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000067' at position 1990 2020-02-01 0:18:52 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:18:58 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000111, 34521) 2020-02-01 0:19:14 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:19:17 15 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 0:19:17 15 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 0:19:18 16 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:19:18 16 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:19:27 25 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-02-01 0:19:27 25 [Warning] Checking table: './openrecords/wp_options' 2020-02-01 0:19:27 26 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-02-01 0:19:27 26 [Warning] Checking table: './ulrich/wp_options' 2020-02-01 0:19:30 31 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:19:30 31 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:19:30 32 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:19:30 32 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:19:30 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f4bac000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4bb05d6cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56203c907f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56203c39e3af] sigaction.c:0(__restore_rt)[0x7f4c1d02b630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56203c2cc151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56203c40a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56203c1c0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56203c1b6d4b] /usr/sbin/mysqld(+0x6ca792)[0x56203c1c1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56203c1e96da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56203c1eb65d] /usr/sbin/mysqld(+0x65e3f3)[0x56203c1553f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56203c154cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56203c1e8b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56203c1eb65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56203c1eb761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56203c1ec377] /usr/sbin/mysqld(+0x58fd84)[0x56203c086d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56203c193b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56203c1abb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56203c1abcd2] /usr/sbin/mysqld(+0x6b55e7)[0x56203c1ac5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56203c1ac677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56203c19b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56203c19d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56203c278afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56203c278bdd] pthread_create.c:0(start_thread)[0x7f4c1d023ea5] /lib64/libc.so.6(clone+0x6d)[0x7f4c1b3c48cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f4bac00fe70): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 33 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:19:37 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:19:37 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:19:37 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:19:37 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:19:37 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:19:37 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:19:37 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:19:37 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:19:37 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:19:37 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137111240 2020-02-01 0:19:37 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:19:37 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:19:37 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:19:37 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:19:37 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:19:37 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:19:37 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:19:37 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137111249; transaction id 5903983 2020-02-01 0:19:37 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:19:37 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:19:37 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:19:37 0 [Note] Starting crash recovery... 2020-02-01 0:19:37 0 [Note] Crash recovery finished. 2020-02-01 0:19:37 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:19:37 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:19:37 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:19:37 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000067' at position 1990 2020-02-01 0:19:37 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:19:37 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:19:37 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000067' at position 1990 2020-02-01 0:19:37 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:19:37 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:19:37 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:19:37 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:19:37 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:19:38 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:19:38 2020-02-01 0:19:48 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:19:48 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000068' at position 48899 2020-02-01 0:19:48 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:19:54 14 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000112, 83144) 2020-02-01 0:20:05 15 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:20:05 16 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:20:05 16 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:20:10 21 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:20:10 21 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:20:10 23 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:20:10 23 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:20:10 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fa890000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fa8fd345cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x557c51b07f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x557c5159e3af] sigaction.c:0(__restore_rt)[0x7fa918468630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x557c514cc151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x557c5160a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x557c513c0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x557c513b6d4b] /usr/sbin/mysqld(+0x6ca792)[0x557c513c1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x557c513e96da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557c513eb65d] /usr/sbin/mysqld(+0x65e3f3)[0x557c513553f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x557c51354cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x557c513e8b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557c513eb65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x557c513eb761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x557c513ec377] /usr/sbin/mysqld(+0x58fd84)[0x557c51286d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x557c51393b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x557c513abb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x557c513abcd2] /usr/sbin/mysqld(+0x6b55e7)[0x557c513ac5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x557c513ac677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x557c5139b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x557c5139d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x557c51478afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x557c51478bdd] pthread_create.c:0(start_thread)[0x7fa918460ea5] /lib64/libc.so.6(clone+0x6d)[0x7fa9168018cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fa89000feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 24 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:20:16 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:20:16 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:20:16 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:20:16 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:20:16 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:20:16 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:20:16 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:20:16 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:20:16 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:20:16 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137114129 2020-02-01 0:20:16 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:20:16 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:20:16 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:20:16 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:20:16 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:20:16 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:20:16 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:20:16 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137114138; transaction id 5903995 2020-02-01 0:20:16 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:20:16 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:20:16 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:20:16 0 [Note] Starting crash recovery... 2020-02-01 0:20:16 0 [Note] Crash recovery finished. 2020-02-01 0:20:16 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:20:16 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:20:16 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:20:16 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000068' at position 48899 2020-02-01 0:20:16 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:20:16 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:20:16 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000068' at position 48899 2020-02-01 0:20:16 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:20:16 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:20:16 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:20:16 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:20:16 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:20:18 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:20:18 2020-02-01 0:20:29 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:20:29 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000069' at position 215625 2020-02-01 0:20:29 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:20:35 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000113, 77466) 2020-02-01 0:20:45 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:20:45 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:20:45 14 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:20:48 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:20:48 17 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:20:48 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:20:48 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:20:49 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f6e74000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f6ec80c3cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55cd50912f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55cd503a93af] sigaction.c:0(__restore_rt)[0x7f6ee2c25630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55cd502d7151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55cd50415545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55cd501cbccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55cd501c1d4b] /usr/sbin/mysqld(+0x6ca792)[0x55cd501cc792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55cd501f46da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55cd501f665d] /usr/sbin/mysqld(+0x65e3f3)[0x55cd501603f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55cd5015fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55cd501f3b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55cd501f665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55cd501f6761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55cd501f7377] /usr/sbin/mysqld(+0x58fd84)[0x55cd50091d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55cd5019eb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55cd501b6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55cd501b6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55cd501b75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55cd501b7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55cd501a656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55cd501a807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55cd50283afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55cd50283bdd] pthread_create.c:0(start_thread)[0x7f6ee2c1dea5] /lib64/libc.so.6(clone+0x6d)[0x7f6ee0fbe8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f6e7400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:20:55 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:20:55 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:20:55 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:20:55 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:20:55 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:20:55 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:20:55 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:20:55 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:20:55 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:20:55 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137117750 2020-02-01 0:20:55 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:20:55 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:20:55 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:20:55 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:20:55 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:20:55 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:20:55 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:20:55 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137117759; transaction id 5904006 2020-02-01 0:20:55 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:20:55 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:20:55 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:20:55 0 [Note] Starting crash recovery... 2020-02-01 0:20:55 0 [Note] Crash recovery finished. 2020-02-01 0:20:55 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:20:55 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:20:55 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:20:55 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000069' at position 215625 2020-02-01 0:20:55 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:20:55 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:20:55 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000069' at position 215625 2020-02-01 0:20:55 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:20:55 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:20:55 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:20:55 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:20:55 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:20:56 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:20:56 2020-02-01 0:21:02 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:21:02 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000070' at position 629903 2020-02-01 0:21:02 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:21:07 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000114, 30643) 2020-02-01 0:21:20 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:21:22 18 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:21:22 18 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:21:22 19 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:21:22 19 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:21:22 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=10 max_threads=502 thread_count=18 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc9f4000c38 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fca244aacf0 thread_stack 0x49000 2020-02-01 0:21:22 19 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:21:22 19 [Warning] Checking table: './ojsdata/controlled_vocabs' /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5622b7ec9f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5622b79603af] sigaction.c:0(__restore_rt)[0x7fca91049630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5622b788e151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5622b79cc545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5622b7782ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5622b7778d4b] /usr/sbin/mysqld(+0x6ca792)[0x5622b7783792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5622b77ab6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5622b77ad65d] /usr/sbin/mysqld(+0x65e3f3)[0x5622b77173f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5622b7716cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5622b77aab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5622b77ad65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5622b77ad761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5622b77ae377] /usr/sbin/mysqld(+0x58fd84)[0x5622b7648d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5622b7755b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5622b776db16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5622b776dcd2] /usr/sbin/mysqld(+0x6b55e7)[0x5622b776e5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5622b776e677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5622b775d56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5622b775f07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5622b783aafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5622b783abdd] pthread_create.c:0(start_thread)[0x7fca91041ea5] /lib64/libc.so.6(clone+0x6d)[0x7fca8f3e28cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc9f400ef00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:21:28 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:21:28 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:21:28 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:21:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:21:28 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:21:28 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:21:28 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:21:28 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:21:28 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:21:28 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137120596 2020-02-01 0:21:28 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:21:28 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:21:28 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:21:28 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:21:28 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:21:28 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:21:28 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:21:28 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137120605; transaction id 5904017 2020-02-01 0:21:28 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:21:28 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:21:28 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:21:28 0 [Note] Starting crash recovery... 2020-02-01 0:21:28 0 [Note] Crash recovery finished. 2020-02-01 0:21:28 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:21:28 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:21:28 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:21:28 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000070' at position 629903 2020-02-01 0:21:28 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:21:28 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:21:28 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:21:28 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:21:28 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:21:28 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:21:28 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:21:28 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000070' at position 629903 2020-02-01 0:21:30 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:21:30 2020-02-01 0:21:44 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:21:44 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000071' at position 587373 2020-02-01 0:21:44 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:21:49 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000115, 48226) 2020-02-01 0:21:55 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:21:56 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:21:56 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:21:56 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:21:56 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:21:56 15 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:21:56 15 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:21:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe3c00009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe42534dcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ea11238f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ea10ccf3af] sigaction.c:0(__restore_rt)[0x7fe4404e6630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55ea10bfd151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55ea10d3b545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55ea10af1ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55ea10ae7d4b] /usr/sbin/mysqld(+0x6ca792)[0x55ea10af2792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55ea10b1a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ea10b1c65d] /usr/sbin/mysqld(+0x65e3f3)[0x55ea10a863f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55ea10a85cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55ea10b19b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ea10b1c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55ea10b1c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55ea10b1d377] /usr/sbin/mysqld(+0x58fd84)[0x55ea109b7d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55ea10ac4b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55ea10adcb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55ea10adccd2] /usr/sbin/mysqld(+0x6b55e7)[0x55ea10add5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55ea10add677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55ea10acc56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55ea10ace07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55ea10ba9afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55ea10ba9bdd] pthread_create.c:0(start_thread)[0x7fe4404deea5] /lib64/libc.so.6(clone+0x6d)[0x7fe43e87f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe3c000fe00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:22:03 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:22:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:22:03 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:22:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:22:03 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:22:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:22:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:22:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:22:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:22:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137123459 2020-02-01 0:22:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:22:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:22:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:22:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:22:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:22:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:22:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137123468; transaction id 5904028 2020-02-01 0:22:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:22:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:22:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:22:03 0 [Note] Starting crash recovery... 2020-02-01 0:22:03 0 [Note] Crash recovery finished. 2020-02-01 0:22:03 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:22:03 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:22:03 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:22:03 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000071' at position 587373 2020-02-01 0:22:03 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:22:03 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:22:03 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000071' at position 587373 2020-02-01 0:22:03 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:22:03 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:22:03 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:22:03 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:22:03 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:22:04 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:22:04 2020-02-01 0:22:20 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:22:20 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000072' at position 576551 2020-02-01 0:22:20 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:22:26 18 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000116, 2513) 2020-02-01 0:22:31 19 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:22:32 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:22:32 20 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:22:32 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:22:32 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:22:33 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f3da8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f3df85dfcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c0b0326f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c0afdbd3af] sigaction.c:0(__restore_rt)[0x7f3e15759630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c0afceb151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c0afe29545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c0afbdfccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c0afbd5d4b] /usr/sbin/mysqld(+0x6ca792)[0x55c0afbe0792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c0afc086da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c0afc0a65d] /usr/sbin/mysqld(+0x65e3f3)[0x55c0afb743f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c0afb73cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c0afc07b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c0afc0a65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c0afc0a761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c0afc0b377] /usr/sbin/mysqld(+0x58fd84)[0x55c0afaa5d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c0afbb2b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c0afbcab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c0afbcacd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c0afbcb5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c0afbcb677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c0afbba56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c0afbbc07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c0afc97afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c0afc97bdd] pthread_create.c:0(start_thread)[0x7f3e15751ea5] /lib64/libc.so.6(clone+0x6d)[0x7f3e13af28cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f3da800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:22:39 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:22:39 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:22:39 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:22:39 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:22:39 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:22:39 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:22:39 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:22:39 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:22:39 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:22:39 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137126306 2020-02-01 0:22:39 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:22:39 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:22:39 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:22:39 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:22:39 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:22:39 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:22:39 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:22:39 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137126315; transaction id 5904039 2020-02-01 0:22:39 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:22:39 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:22:39 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:22:39 0 [Note] Starting crash recovery... 2020-02-01 0:22:39 0 [Note] Crash recovery finished. 2020-02-01 0:22:39 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:22:39 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:22:39 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:22:39 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000072' at position 576551 2020-02-01 0:22:39 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:22:39 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:22:39 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:22:39 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:22:39 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:22:39 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:22:39 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:22:39 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000072' at position 576551 2020-02-01 0:22:40 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:22:40 2020-02-01 0:22:53 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:22:53 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000073' at position 4472 2020-02-01 0:22:53 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:22:59 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000117, 1464) 2020-02-01 0:23:06 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:23:07 17 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-02-01 0:23:07 17 [Warning] Checking table: './wpcomm305/comm305_options' 2020-02-01 0:23:11 19 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:23:11 19 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:23:13 24 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:23:13 24 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:23:13 25 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:23:13 25 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:23:13 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2580000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2600073cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5578d5695f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5578d512c3af] sigaction.c:0(__restore_rt)[0x7f260619f630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5578d505a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5578d5198545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5578d4f4eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5578d4f44d4b] /usr/sbin/mysqld(+0x6ca792)[0x5578d4f4f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5578d4f776da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5578d4f7965d] /usr/sbin/mysqld(+0x65e3f3)[0x5578d4ee33f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5578d4ee2cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5578d4f76b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5578d4f7965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5578d4f79761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5578d4f7a377] /usr/sbin/mysqld(+0x58fd84)[0x5578d4e14d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5578d4f21b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5578d4f39b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5578d4f39cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5578d4f3a5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5578d4f3a677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5578d4f2956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5578d4f2b07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5578d5006afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5578d5006bdd] pthread_create.c:0(start_thread)[0x7f2606197ea5] /lib64/libc.so.6(clone+0x6d)[0x7f26045388cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f2580012150): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 26 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:23:20 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:23:20 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:23:20 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:23:20 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:23:20 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:23:20 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:23:20 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:23:20 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:23:20 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:23:20 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137129169 2020-02-01 0:23:20 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:23:20 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:23:20 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:23:20 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:23:20 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:23:20 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:23:20 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:23:20 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137129178; transaction id 5904050 2020-02-01 0:23:20 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:23:20 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:23:20 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:23:20 0 [Note] Starting crash recovery... 2020-02-01 0:23:20 0 [Note] Crash recovery finished. 2020-02-01 0:23:20 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:23:20 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:23:20 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:23:20 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000073' at position 4472 2020-02-01 0:23:20 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:23:20 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:23:20 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000073' at position 4472 2020-02-01 0:23:20 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:23:20 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:23:20 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:23:20 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:23:20 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:23:20 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:23:20 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:23:20 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:23:20 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:23:20 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:23:21 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:23:21 2020-02-01 0:23:29 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:23:29 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000074' at position 23638 2020-02-01 0:23:29 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:23:35 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000118, 175352) 2020-02-01 0:23:46 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:23:48 14 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:23:48 14 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:23:50 17 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-02-01 0:23:50 17 [Warning] Checking table: './wpstart/wp_options' 2020-02-01 0:23:50 18 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:23:50 18 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:23:50 19 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:23:50 19 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:23:50 19 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:23:50 19 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:23:51 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0ae8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0b4413fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e3d856ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e3d80053af] sigaction.c:0(__restore_rt)[0x7f0b5f0dd630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55e3d7f33151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55e3d8071545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55e3d7e27ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55e3d7e1dd4b] /usr/sbin/mysqld(+0x6ca792)[0x55e3d7e28792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55e3d7e506da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e3d7e5265d] /usr/sbin/mysqld(+0x65e3f3)[0x55e3d7dbc3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55e3d7dbbcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55e3d7e4fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e3d7e5265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55e3d7e52761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55e3d7e53377] /usr/sbin/mysqld(+0x58fd84)[0x55e3d7cedd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55e3d7dfab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55e3d7e12b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55e3d7e12cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55e3d7e135e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55e3d7e13677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55e3d7e0256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55e3d7e0407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55e3d7edfafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e3d7edfbdd] pthread_create.c:0(start_thread)[0x7f0b5f0d5ea5] /lib64/libc.so.6(clone+0x6d)[0x7f0b5d4768cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f0ae800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 20 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:23:57 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:23:57 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:23:57 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:23:57 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:23:57 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:23:57 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:23:57 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:23:57 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:23:57 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:23:57 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137132016 2020-02-01 0:23:57 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:23:57 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:23:57 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:23:57 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:23:57 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:23:57 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:23:57 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:23:57 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137132025; transaction id 5904061 2020-02-01 0:23:57 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:23:57 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:23:57 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:23:57 0 [Note] Starting crash recovery... 2020-02-01 0:23:57 0 [Note] Crash recovery finished. 2020-02-01 0:23:57 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:23:57 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:23:57 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:23:57 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000074' at position 23638 2020-02-01 0:23:57 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:23:57 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:23:57 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000074' at position 23638 2020-02-01 0:23:57 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:23:57 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:23:57 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:23:57 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:23:57 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:23:58 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:23:58 2020-02-01 0:24:06 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:24:06 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000075' at position 54293 2020-02-01 0:24:06 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:24:12 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000119, 25323) 2020-02-01 0:24:26 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:24:26 15 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:24:26 15 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:24:27 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:24:27 17 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:24:27 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:24:27 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:24:27 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2ee4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2e98213cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5590ea0bbf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5590e9b523af] sigaction.c:0(__restore_rt)[0x7f2f054cc630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5590e9a80151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5590e9bbe545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5590e9974ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5590e996ad4b] /usr/sbin/mysqld(+0x6ca792)[0x5590e9975792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5590e999d6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5590e999f65d] /usr/sbin/mysqld(+0x65e3f3)[0x5590e99093f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5590e9908cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5590e999cb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5590e999f65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5590e999f761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5590e99a0377] /usr/sbin/mysqld(+0x58fd84)[0x5590e983ad84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5590e9947b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5590e995fb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5590e995fcd2] /usr/sbin/mysqld(+0x6b55e7)[0x5590e99605e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5590e9960677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5590e994f56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5590e995107c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5590e9a2cafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5590e9a2cbdd] pthread_create.c:0(start_thread)[0x7f2f054c4ea5] /lib64/libc.so.6(clone+0x6d)[0x7f2f038658cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f2ee400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 20 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:24:33 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:24:33 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:24:33 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:24:33 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:24:33 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:24:33 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:24:33 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:24:33 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:24:33 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:24:33 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137134931 2020-02-01 0:24:33 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:24:33 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:24:33 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:24:33 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:24:33 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:24:33 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:24:34 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:24:34 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137134940; transaction id 5904072 2020-02-01 0:24:34 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:24:34 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:24:34 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:24:34 0 [Note] Starting crash recovery... 2020-02-01 0:24:34 0 [Note] Crash recovery finished. 2020-02-01 0:24:34 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:24:34 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:24:34 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:24:34 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000075' at position 54293 2020-02-01 0:24:34 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:24:34 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:24:34 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000075' at position 54293 2020-02-01 0:24:34 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:24:34 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:24:34 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:24:34 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:24:34 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:24:35 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:24:35 2020-02-01 0:24:42 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:24:42 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000076' at position 123613 2020-02-01 0:24:42 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:24:48 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000120, 1464) 2020-02-01 0:25:01 15 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:25:03 18 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:25:03 18 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:25:03 20 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-02-01 0:25:03 19 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:25:03 20 [Warning] Checking table: './wpcomm303/wp_options' 2020-02-01 0:25:03 19 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:25:03 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f3760000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f37640a9cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5560438cdf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5560433643af] sigaction.c:0(__restore_rt)[0x7f3780ec5630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x556043292151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5560433d0545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x556043186ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55604317cd4b] /usr/sbin/mysqld(+0x6ca792)[0x556043187792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5560431af6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5560431b165d] /usr/sbin/mysqld(+0x65e3f3)[0x55604311b3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55604311acb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5560431aeb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5560431b165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5560431b1761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5560431b2377] /usr/sbin/mysqld(+0x58fd84)[0x55604304cd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x556043159b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x556043171b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x556043171cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5560431725e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x556043172677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55604316156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55604316307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55604323eafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55604323ebdd] pthread_create.c:0(start_thread)[0x7f3780ebdea5] /lib64/libc.so.6(clone+0x6d)[0x7f377f25e8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f376000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 21 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:25:10 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:25:10 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:25:10 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:25:10 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:25:10 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:25:10 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:25:10 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:25:10 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:25:10 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:25:10 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137139392 2020-02-01 0:25:10 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:25:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:25:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:25:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:25:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:25:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:25:10 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:25:10 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137139401; transaction id 5904083 2020-02-01 0:25:10 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:25:10 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:25:10 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:25:10 0 [Note] Starting crash recovery... 2020-02-01 0:25:10 0 [Note] Crash recovery finished. 2020-02-01 0:25:10 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:25:10 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:25:10 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:25:10 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000076' at position 123613 2020-02-01 0:25:10 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:25:10 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:25:10 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000076' at position 123613 2020-02-01 0:25:10 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:25:10 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:25:10 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:25:10 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:25:10 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:25:11 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:25:11 2020-02-01 0:25:24 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:25:24 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000077' at position 3058 2020-02-01 0:25:24 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:25:30 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000121, 5562) 2020-02-01 0:25:37 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:25:42 15 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:25:42 15 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:25:43 16 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:25:43 16 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:25:44 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:25:44 20 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:25:44 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:25:44 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:25:44 21 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:25:44 21 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:25:45 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fb104000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fb1600fccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c3585cef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c3580653af] sigaction.c:0(__restore_rt)[0x7fb17a1e9630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c357f93151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c3580d1545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c357e87ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c357e7dd4b] /usr/sbin/mysqld(+0x6ca792)[0x55c357e88792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c357eb06da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c357eb265d] /usr/sbin/mysqld(+0x65e3f3)[0x55c357e1c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c357e1bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c357eafb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c357eb265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c357eb2761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c357eb3377] /usr/sbin/mysqld(+0x58fd84)[0x55c357d4dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c357e5ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c357e72b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c357e72cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c357e735e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c357e73677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c357e6256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c357e6407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c357f3fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c357f3fbdd] pthread_create.c:0(start_thread)[0x7fb17a1e1ea5] /lib64/libc.so.6(clone+0x6d)[0x7fb1785828cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fb10400feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:25:51 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:25:51 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:25:51 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:25:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:25:51 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:25:51 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:25:51 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:25:51 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:25:51 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:25:51 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137142237 2020-02-01 0:25:51 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:25:51 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:25:51 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:25:51 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:25:51 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:25:51 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:25:51 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:25:51 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137142246; transaction id 5904094 2020-02-01 0:25:51 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:25:51 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:25:51 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:25:51 0 [Note] Starting crash recovery... 2020-02-01 0:25:51 0 [Note] Crash recovery finished. 2020-02-01 0:25:51 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:25:51 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:25:51 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:25:51 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000077' at position 3058 2020-02-01 0:25:51 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:25:51 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:25:51 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000077' at position 3058 2020-02-01 0:25:51 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:25:51 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:25:51 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:25:51 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:25:51 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:25:52 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:25:52 2020-02-01 0:26:01 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:26:01 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000078' at position 99193 2020-02-01 0:26:01 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:26:07 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000122, 95148) 2020-02-01 0:26:17 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:26:18 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:26:18 14 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:26:19 19 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:26:19 19 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:26:19 21 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-02-01 0:26:19 20 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:26:19 21 [Warning] Checking table: './itstraining/wp_options' 2020-02-01 0:26:19 20 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:26:19 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=9 max_threads=502 thread_count=17 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x564a91bbb098 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4f9c103cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x564a8d9b3f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x564a8d44a3af] sigaction.c:0(__restore_rt)[0x7f4ffe47d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564a8d378151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564a8d4b6545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564a8d26cccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x564a8d262d4b] /usr/sbin/mysqld(+0x6ca792)[0x564a8d26d792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x564a8d2956da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564a8d29765d] /usr/sbin/mysqld(+0x65e3f3)[0x564a8d2013f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x564a8d200cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x564a8d294b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564a8d29765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x564a8d297761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x564a8d298377] /usr/sbin/mysqld(+0x58fd84)[0x564a8d132d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x564a8d23fb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564a8d257b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564a8d257cd2] /usr/sbin/mysqld(+0x6b55e7)[0x564a8d2585e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564a8d258677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x564a8d24756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x564a8d24907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x564a8d324afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x564a8d324bdd] pthread_create.c:0(start_thread)[0x7f4ffe475ea5] /lib64/libc.so.6(clone+0x6d)[0x7f4ffc8168cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x564a91bc9770): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:26:25 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:26:25 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:26:25 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:26:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:26:25 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:26:25 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:26:25 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:26:25 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:26:25 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:26:25 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137145100 2020-02-01 0:26:25 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:26:25 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:26:25 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:26:25 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:26:25 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:26:25 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:26:25 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:26:25 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137145109; transaction id 5904105 2020-02-01 0:26:25 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:26:25 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:26:25 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:26:25 0 [Note] Starting crash recovery... 2020-02-01 0:26:25 0 [Note] Crash recovery finished. 2020-02-01 0:26:25 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:26:25 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:26:25 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:26:25 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000078' at position 99193 2020-02-01 0:26:25 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:26:25 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:26:25 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:26:25 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:26:25 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:26:25 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000078' at position 99193 2020-02-01 0:26:25 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:26:25 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:26:25 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:26:25 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:26:25 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:26:25 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:26:25 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:26:26 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:26:26 2020-02-01 0:26:38 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:26:38 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000079' at position 32491 2020-02-01 0:26:38 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:26:44 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000123, 2606) 2020-02-01 0:26:52 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:26:53 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:26:53 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:26:53 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:26:53 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:26:53 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f52f4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f530014ccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x559c5df6cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x559c5da033af] sigaction.c:0(__restore_rt)[0x7f536d31b630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x559c5d931151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x559c5da6f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x559c5d825ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x559c5d81bd4b] /usr/sbin/mysqld(+0x6ca792)[0x559c5d826792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x559c5d84e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559c5d85065d] /usr/sbin/mysqld(+0x65e3f3)[0x559c5d7ba3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x559c5d7b9cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x559c5d84db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559c5d85065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x559c5d850761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x559c5d851377] /usr/sbin/mysqld(+0x58fd84)[0x559c5d6ebd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x559c5d7f8b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x559c5d810b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x559c5d810cd2] /usr/sbin/mysqld(+0x6b55e7)[0x559c5d8115e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x559c5d811677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x559c5d80056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x559c5d80207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x559c5d8ddafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x559c5d8ddbdd] pthread_create.c:0(start_thread)[0x7f536d313ea5] /lib64/libc.so.6(clone+0x6d)[0x7f536b6b48cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f52f400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:27:00 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:27:00 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:27:00 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:27:00 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:27:00 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:27:00 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:27:00 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:27:00 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:27:00 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:27:00 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137148080 2020-02-01 0:27:00 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:27:00 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:27:00 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:27:00 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:27:00 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:27:00 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:27:00 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:27:00 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137148089; transaction id 5904116 2020-02-01 0:27:00 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:27:00 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:27:00 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:27:00 0 [Note] Starting crash recovery... 2020-02-01 0:27:00 0 [Note] Crash recovery finished. 2020-02-01 0:27:00 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:27:00 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:27:00 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:27:00 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000079' at position 32491 2020-02-01 0:27:00 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:27:00 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:27:00 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000079' at position 32491 2020-02-01 0:27:00 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:27:00 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:27:00 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:27:00 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:27:00 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:27:01 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:27:01 2020-02-01 0:27:17 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:27:17 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000080' at position 74929 2020-02-01 0:27:17 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:27:22 17 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000124, 1464) 2020-02-01 0:27:27 19 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:27:29 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:27:29 20 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:27:29 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:27:29 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:27:29 21 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:27:29 21 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:27:29 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fb3b8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fb4100c3cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5608dabe7f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5608da67e3af] sigaction.c:0(__restore_rt)[0x7fb42a79f630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5608da5ac151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5608da6ea545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5608da4a0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5608da496d4b] /usr/sbin/mysqld(+0x6ca792)[0x5608da4a1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5608da4c96da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5608da4cb65d] /usr/sbin/mysqld(+0x65e3f3)[0x5608da4353f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5608da434cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5608da4c8b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5608da4cb65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5608da4cb761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5608da4cc377] /usr/sbin/mysqld(+0x58fd84)[0x5608da366d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5608da473b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5608da48bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5608da48bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x5608da48c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5608da48c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5608da47b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5608da47d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5608da558afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5608da558bdd] pthread_create.c:0(start_thread)[0x7fb42a797ea5] /lib64/libc.so.6(clone+0x6d)[0x7fb428b388cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fb3b800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:27:35 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:27:35 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:27:35 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:27:35 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:27:35 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:27:35 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:27:35 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:27:35 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:27:35 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:27:35 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137151086 2020-02-01 0:27:35 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:27:35 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:27:35 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:27:35 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:27:35 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:27:35 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:27:35 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:27:35 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137151095; transaction id 5904128 2020-02-01 0:27:35 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:27:35 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:27:35 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:27:35 0 [Note] Starting crash recovery... 2020-02-01 0:27:35 0 [Note] Crash recovery finished. 2020-02-01 0:27:35 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:27:35 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:27:35 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:27:35 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000080' at position 74929 2020-02-01 0:27:35 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:27:35 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:27:35 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000080' at position 74929 2020-02-01 0:27:35 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:27:35 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:27:35 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:27:35 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:27:35 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:27:36 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:27:36 2020-02-01 0:27:50 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:27:50 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000081' at position 25651 2020-02-01 0:27:50 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:27:56 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000125, 2513) 2020-02-01 0:28:03 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:28:03 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:28:03 14 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:28:11 24 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:28:11 24 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:28:11 25 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:28:11 25 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:28:11 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe5cc000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe589d96cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d4ebc3ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55d4eb6d53af] sigaction.c:0(__restore_rt)[0x7fe5eafc5630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55d4eb603151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55d4eb741545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55d4eb4f7ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55d4eb4edd4b] /usr/sbin/mysqld(+0x6ca792)[0x55d4eb4f8792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55d4eb5206da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d4eb52265d] /usr/sbin/mysqld(+0x65e3f3)[0x55d4eb48c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55d4eb48bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55d4eb51fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d4eb52265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55d4eb522761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55d4eb523377] /usr/sbin/mysqld(+0x58fd84)[0x55d4eb3bdd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55d4eb4cab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55d4eb4e2b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55d4eb4e2cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55d4eb4e35e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55d4eb4e3677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55d4eb4d256d] 2020-02-01 0:28:11 26 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55d4eb4d407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55d4eb5afafa] 2020-02-01 0:28:11 26 [Warning] Checking table: './ojsdata/controlled_vocabs' /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d4eb5afbdd] pthread_create.c:0(start_thread)[0x7fe5eafbdea5] /lib64/libc.so.6(clone+0x6d)[0x7fe5e935e8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe5cc00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 28 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:28:17 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:28:17 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:28:17 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:28:17 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:28:17 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:28:17 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:28:17 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:28:17 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:28:17 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:28:17 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137154857 2020-02-01 0:28:17 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:28:17 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:28:17 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:28:17 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:28:17 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:28:17 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:28:17 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:28:17 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137154866; transaction id 5904139 2020-02-01 0:28:17 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:28:17 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:28:17 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:28:17 0 [Note] Starting crash recovery... 2020-02-01 0:28:17 0 [Note] Crash recovery finished. 2020-02-01 0:28:17 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:28:17 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:28:17 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:28:17 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000081' at position 25651 2020-02-01 0:28:17 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:28:17 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:28:17 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000081' at position 25651 2020-02-01 0:28:17 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:28:17 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:28:17 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:28:17 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:28:17 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:28:17 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:28:17 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:28:17 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:28:17 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:28:17 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:28:19 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:28:19 2020-02-01 0:28:24 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:28:24 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000082' at position 1464 2020-02-01 0:28:24 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:28:30 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000126, 125351) 2020-02-01 0:28:43 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:28:46 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:28:46 16 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:28:46 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:28:46 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:28:47 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fa44c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fa4a8cabcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55bab4947f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55bab43de3af] sigaction.c:0(__restore_rt)[0x7fa4c3e63630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55bab430c151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55bab444a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55bab4200ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55bab41f6d4b] /usr/sbin/mysqld(+0x6ca792)[0x55bab4201792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55bab42296da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55bab422b65d] /usr/sbin/mysqld(+0x65e3f3)[0x55bab41953f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55bab4194cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55bab4228b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55bab422b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55bab422b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55bab422c377] /usr/sbin/mysqld(+0x58fd84)[0x55bab40c6d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55bab41d3b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55bab41ebb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55bab41ebcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55bab41ec5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55bab41ec677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55bab41db56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55bab41dd07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55bab42b8afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55bab42b8bdd] pthread_create.c:0(start_thread)[0x7fa4c3e5bea5] /lib64/libc.so.6(clone+0x6d)[0x7fa4c21fc8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fa44c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:28:53 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:28:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:28:53 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:28:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:28:53 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:28:53 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:28:53 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:28:53 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:28:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:28:53 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137157836 2020-02-01 0:28:53 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:28:53 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:28:53 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:28:53 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:28:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:28:53 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:28:53 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137157845; transaction id 5904150 2020-02-01 0:28:53 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:28:53 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:28:53 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:28:53 0 [Note] Starting crash recovery... 2020-02-01 0:28:53 0 [Note] Crash recovery finished. 2020-02-01 0:28:53 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:28:53 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:28:53 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:28:53 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000082' at position 1464 2020-02-01 0:28:53 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:28:53 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:28:53 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000082' at position 1464 2020-02-01 0:28:53 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:28:53 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:28:53 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:28:53 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:28:53 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:28:54 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:28:54 2020-02-01 0:29:00 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:29:00 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000083' at position 2513 2020-02-01 0:29:00 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:29:05 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000127, 1464) 2020-02-01 0:29:18 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:29:19 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:29:19 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:29:19 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:29:19 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:29:19 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f3b9c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f3bf9b2bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55cc86eb5f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55cc8694c3af] sigaction.c:0(__restore_rt)[0x7f3c1ccaa630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55cc8687a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55cc869b8545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55cc8676eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55cc86764d4b] /usr/sbin/mysqld(+0x6ca792)[0x55cc8676f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55cc867976da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55cc8679965d] /usr/sbin/mysqld(+0x65e3f3)[0x55cc867033f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55cc86702cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55cc86796b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55cc8679965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55cc86799761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55cc8679a377] /usr/sbin/mysqld(+0x58fd84)[0x55cc86634d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55cc86741b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55cc86759b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55cc86759cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55cc8675a5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55cc8675a677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55cc8674956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55cc8674b07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55cc86826afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55cc86826bdd] pthread_create.c:0(start_thread)[0x7f3c1cca2ea5] /lib64/libc.so.6(clone+0x6d)[0x7f3c1b0438cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f3b9c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:29:25 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:29:25 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:29:25 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:29:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:29:25 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:29:25 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:29:25 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:29:25 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:29:25 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:29:25 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137160683 2020-02-01 0:29:25 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:29:26 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:29:26 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:29:26 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:29:26 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:29:26 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:29:26 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:29:26 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137160692; transaction id 5904161 2020-02-01 0:29:26 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:29:26 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:29:26 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:29:26 0 [Note] Starting crash recovery... 2020-02-01 0:29:26 0 [Note] Crash recovery finished. 2020-02-01 0:29:26 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:29:26 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:29:26 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:29:26 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000083' at position 2513 2020-02-01 0:29:26 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:29:26 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:29:26 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:29:26 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:29:26 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:29:26 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:29:26 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:29:26 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000083' at position 2513 2020-02-01 0:29:27 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:29:27 2020-02-01 0:29:35 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:29:35 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000084' at position 1454 2020-02-01 0:29:35 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:29:41 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000128, 1464) 2020-02-01 0:29:53 15 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:29:54 17 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-02-01 0:29:54 17 [Warning] Checking table: './shockeripe/wp_options' 2020-02-01 0:29:54 18 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:29:54 18 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:29:57 27 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:29:57 27 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:29:57 28 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:29:57 28 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:29:57 28 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:29:57 28 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:29:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fee640009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7feed804ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5569a3f52f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5569a39e93af] sigaction.c:0(__restore_rt)[0x7feee0ba0630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5569a3917151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5569a3a55545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5569a380bccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5569a3801d4b] /usr/sbin/mysqld(+0x6ca792)[0x5569a380c792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5569a38346da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5569a383665d] /usr/sbin/mysqld(+0x65e3f3)[0x5569a37a03f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5569a379fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5569a3833b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5569a383665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5569a3836761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5569a3837377] /usr/sbin/mysqld(+0x58fd84)[0x5569a36d1d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5569a37deb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5569a37f6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5569a37f6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5569a37f75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5569a37f7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5569a37e656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5569a37e807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5569a38c3afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5569a38c3bdd] pthread_create.c:0(start_thread)[0x7feee0b98ea5] /lib64/libc.so.6(clone+0x6d)[0x7feedef398cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fee64012070): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 29 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:30:04 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:30:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:30:04 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:30:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:30:04 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:30:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:30:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:30:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:30:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:30:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137163546 2020-02-01 0:30:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:30:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:30:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:30:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:30:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:30:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:30:04 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:30:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137163555; transaction id 5904172 2020-02-01 0:30:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:30:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:30:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:30:04 0 [Note] Starting crash recovery... 2020-02-01 0:30:04 0 [Note] Crash recovery finished. 2020-02-01 0:30:04 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:30:04 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:30:04 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:30:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000084' at position 1454 2020-02-01 0:30:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:30:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:30:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000084' at position 1454 2020-02-01 0:30:04 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:30:04 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:30:04 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:30:04 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:30:04 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:30:04 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:30:04 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:30:04 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:30:04 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:30:04 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:30:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:30:05 2020-02-01 0:30:12 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:30:12 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000085' at position 1464 2020-02-01 0:30:12 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:30:17 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000129, 96186) 2020-02-01 0:30:34 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:30:35 14 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:30:35 14 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:30:37 19 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:30:37 19 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:30:38 20 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:30:38 20 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:30:38 17 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 0:30:38 17 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-02-01 0:30:38 20 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:30:38 20 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:30:38 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f8c14000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f8c5014dcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5588bd643f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5588bd0da3af] sigaction.c:0(__restore_rt)[0x7f8c8362b630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5588bd008151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5588bd146545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5588bcefcccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5588bcef2d4b] /usr/sbin/mysqld(+0x6ca792)[0x5588bcefd792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5588bcf256da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5588bcf2765d] /usr/sbin/mysqld(+0x65e3f3)[0x5588bce913f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5588bce90cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5588bcf24b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5588bcf2765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5588bcf27761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5588bcf28377] /usr/sbin/mysqld(+0x58fd84)[0x5588bcdc2d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5588bcecfb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5588bcee7b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5588bcee7cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5588bcee85e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5588bcee8677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5588bced756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5588bced907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5588bcfb4afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5588bcfb4bdd] pthread_create.c:0(start_thread)[0x7f8c83623ea5] /lib64/libc.so.6(clone+0x6d)[0x7f8c819c48cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f8c1400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 21 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:30:44 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:30:44 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:30:44 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:30:44 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:30:44 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:30:44 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:30:44 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:30:44 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:30:44 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:30:44 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137166409 2020-02-01 0:30:44 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:30:44 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:30:44 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:30:44 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:30:44 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:30:44 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:30:44 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:30:44 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137166418; transaction id 5904183 2020-02-01 0:30:44 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:30:44 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:30:44 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:30:44 0 [Note] Starting crash recovery... 2020-02-01 0:30:44 0 [Note] Crash recovery finished. 2020-02-01 0:30:44 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:30:44 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:30:44 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:30:44 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000085' at position 1464 2020-02-01 0:30:44 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:30:44 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:30:44 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000085' at position 1464 2020-02-01 0:30:44 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:30:44 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:30:44 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:30:44 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:30:44 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:30:44 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:30:44 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:30:44 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:30:44 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:30:44 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:30:45 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:30:45 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000086' at position 2513 2020-02-01 0:30:45 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:30:45 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:30:45 2020-02-01 0:30:51 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000130, 68073) 2020-02-01 0:31:09 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:31:13 19 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 0:31:13 19 [Warning] Checking table: './wpcomm306/wp_options' 2020-02-01 0:31:13 18 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 0:31:13 18 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-02-01 0:31:15 22 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:31:15 22 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:31:15 23 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:31:15 23 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:31:15 23 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:31:15 23 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:31:15 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7feba40009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7febb47ebcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55be2df9df2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55be2da343af] sigaction.c:0(__restore_rt)[0x7fec20f31630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55be2d962151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55be2daa0545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55be2d856ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55be2d84cd4b] /usr/sbin/mysqld(+0x6ca792)[0x55be2d857792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55be2d87f6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55be2d88165d] /usr/sbin/mysqld(+0x65e3f3)[0x55be2d7eb3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55be2d7eacb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55be2d87eb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55be2d88165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55be2d881761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55be2d882377] /usr/sbin/mysqld(+0x58fd84)[0x55be2d71cd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55be2d829b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55be2d841b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55be2d841cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55be2d8425e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55be2d842677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55be2d83156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55be2d83307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55be2d90eafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55be2d90ebdd] pthread_create.c:0(start_thread)[0x7fec20f29ea5] /lib64/libc.so.6(clone+0x6d)[0x7fec1f2ca8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7feba4010950): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 24 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:31:22 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:31:22 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:31:22 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:31:22 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:31:22 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:31:22 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:31:22 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:31:22 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:31:22 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:31:22 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137169340 2020-02-01 0:31:22 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:31:22 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:31:22 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:31:22 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:31:22 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:31:22 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:31:22 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:31:22 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137169349; transaction id 5904194 2020-02-01 0:31:22 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:31:22 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:31:22 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:31:22 0 [Note] Starting crash recovery... 2020-02-01 0:31:22 0 [Note] Crash recovery finished. 2020-02-01 0:31:22 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:31:22 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:31:22 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:31:22 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000086' at position 2513 2020-02-01 0:31:22 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:31:22 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:31:22 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:31:22 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:31:22 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:31:22 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:31:22 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:31:22 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:31:22 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:31:22 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:31:22 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:31:22 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:31:22 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:31:22 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:31:22 2020-02-01 0:31:26 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000131, 54197) 2020-02-01 0:31:49 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:31:52 14 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:31:52 14 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:31:54 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:31:54 16 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:31:54 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:31:54 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:31:54 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fca88000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fca95a98cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x558166ab4f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55816654b3af] sigaction.c:0(__restore_rt)[0x7fcaf6ced630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x558166479151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5581665b7545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55816636dccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x558166363d4b] /usr/sbin/mysqld(+0x6ca792)[0x55816636e792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5581663966da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55816639865d] /usr/sbin/mysqld(+0x65e3f3)[0x5581663023f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x558166301cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x558166395b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55816639865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x558166398761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x558166399377] /usr/sbin/mysqld(+0x58fd84)[0x558166233d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x558166340b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x558166358b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x558166358cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5581663595e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x558166359677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55816634856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55816634a07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x558166425afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x558166425bdd] pthread_create.c:0(start_thread)[0x7fcaf6ce5ea5] /lib64/libc.so.6(clone+0x6d)[0x7fcaf50868cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fca8800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:32:00 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:32:00 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:32:00 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:32:00 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:32:00 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:32:00 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:32:00 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:32:00 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:32:00 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:32:00 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137169704 2020-02-01 0:32:00 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:32:00 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:32:00 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:32:00 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:32:00 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:32:00 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:32:00 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:32:00 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137169713; transaction id 5904199 2020-02-01 0:32:00 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:32:00 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:32:00 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:32:00 0 [Note] Starting crash recovery... 2020-02-01 0:32:00 0 [Note] Crash recovery finished. 2020-02-01 0:32:00 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:32:00 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:32:00 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:32:00 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000086' at position 2513 2020-02-01 0:32:00 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:32:00 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:32:00 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:32:00 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:32:00 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:32:00 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:32:00 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:32:00 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:32:00 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:32:00 2020-02-01 0:32:06 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000132, 2471) 2020-02-01 0:32:29 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:32:41 28 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:32:41 28 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:32:41 29 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:32:41 29 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:32:41 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9cc8000ac8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f9d346e7cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560bcea6bf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560bce5023af] sigaction.c:0(__restore_rt)[0x7f9d5184b630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x560bce430151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x560bce56e545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x560bce324ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x560bce31ad4b] /usr/sbin/mysqld(+0x6ca792)[0x560bce325792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x560bce34d6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560bce34f65d] /usr/sbin/mysqld(+0x65e3f3)[0x560bce2b93f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x560bce2b8cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x560bce34cb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560bce34f65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x560bce34f761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x560bce350377] /usr/sbin/mysqld(+0x58fd84)[0x560bce1ead84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x560bce2f7b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x560bce30fb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x560bce30fcd2] /usr/sbin/mysqld(+0x6b55e7)[0x560bce3105e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x560bce310677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x560bce2ff56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x560bce30107c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x560bce3dcafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560bce3dcbdd] pthread_create.c:0(start_thread)[0x7f9d51843ea5] /lib64/libc.so.6(clone+0x6d)[0x7f9d4fbe48cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f9cc800ff20): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 30 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:32:48 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:32:48 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:32:48 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:32:48 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:32:48 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:32:48 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:32:48 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:32:48 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:32:48 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:32:48 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137169722 2020-02-01 0:32:48 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:32:48 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:32:48 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:32:48 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:32:48 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:32:48 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:32:48 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:32:48 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137169731; transaction id 5904199 2020-02-01 0:32:48 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:32:48 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:32:48 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:32:48 0 [Note] Starting crash recovery... 2020-02-01 0:32:48 0 [Note] Crash recovery finished. 2020-02-01 0:32:48 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:32:48 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:32:48 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:32:48 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000086' at position 2513 2020-02-01 0:32:48 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:32:48 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:32:48 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:32:48 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:32:48 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:32:48 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:32:48 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000086' at position 2513 2020-02-01 0:32:48 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:32:48 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:32:48 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:32:48 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:32:48 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:32:48 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:32:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:32:48 2020-02-01 0:32:48 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:32:48 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000089' at position 46874 2020-02-01 0:32:48 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:32:54 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000133, 224874) 2020-02-01 0:33:14 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:33:16 15 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:33:16 15 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:33:19 17 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-02-01 0:33:19 17 [Warning] Checking table: './wpcomm305/comm305_options' 2020-02-01 0:33:24 23 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:33:24 23 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:33:24 24 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:33:24 24 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:33:25 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0820000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0878b7ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563a6c29ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x563a6bd363af] sigaction.c:0(__restore_rt)[0x7f0895d49630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x563a6bc64151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x563a6bda2545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x563a6bb58ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x563a6bb4ed4b] /usr/sbin/mysqld(+0x6ca792)[0x563a6bb59792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x563a6bb816da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563a6bb8365d] /usr/sbin/mysqld(+0x65e3f3)[0x563a6baed3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x563a6baeccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x563a6bb80b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563a6bb8365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x563a6bb83761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x563a6bb84377] /usr/sbin/mysqld(+0x58fd84)[0x563a6ba1ed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x563a6bb2bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x563a6bb43b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x563a6bb43cd2] /usr/sbin/mysqld(+0x6b55e7)[0x563a6bb445e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x563a6bb44677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x563a6bb3356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x563a6bb3507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x563a6bc10afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x563a6bc10bdd] pthread_create.c:0(start_thread)[0x7f0895d41ea5] /lib64/libc.so.6(clone+0x6d)[0x7f08940e28cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f082000ff00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 26 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:33:31 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:33:31 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:33:31 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:33:31 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:33:31 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:33:31 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:33:31 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:33:31 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:33:31 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:33:31 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137170065 2020-02-01 0:33:31 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:33:31 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:33:31 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:33:31 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:33:31 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:33:31 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:33:31 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:33:31 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137170074; transaction id 5904204 2020-02-01 0:33:31 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:33:31 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:33:31 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:33:31 0 [Note] Starting crash recovery... 2020-02-01 0:33:31 0 [Note] Crash recovery finished. 2020-02-01 0:33:31 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:33:31 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:33:31 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:33:31 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000089' at position 46874 2020-02-01 0:33:31 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:33:31 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:33:31 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000089' at position 46874 2020-02-01 0:33:31 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:33:31 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:33:31 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:33:31 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:33:31 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:33:32 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:33:32 2020-02-01 0:33:34 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:33:34 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000090' at position 1464 2020-02-01 0:33:34 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:33:39 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000134, 79111) 2020-02-01 0:34:00 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:34:01 17 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:34:01 17 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:34:02 17 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 0:34:02 17 [Warning] Checking table: './wpms/wp_usermeta' 2020-02-01 0:34:05 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:34:05 20 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:34:05 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:34:05 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:34:05 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fd6e4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fd7508aacf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5635993daf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x563598e713af] sigaction.c:0(__restore_rt)[0x7fd76ba26630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x563598d9f151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x563598edd545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x563598c93ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x563598c89d4b] /usr/sbin/mysqld(+0x6ca792)[0x563598c94792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x563598cbc6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563598cbe65d] /usr/sbin/mysqld(+0x65e3f3)[0x563598c283f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x563598c27cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x563598cbbb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563598cbe65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x563598cbe761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x563598cbf377] /usr/sbin/mysqld(+0x58fd84)[0x563598b59d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x563598c66b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x563598c7eb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x563598c7ecd2] /usr/sbin/mysqld(+0x6b55e7)[0x563598c7f5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x563598c7f677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x563598c6e56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x563598c7007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x563598d4bafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x563598d4bbdd] pthread_create.c:0(start_thread)[0x7fd76ba1eea5] /lib64/libc.so.6(clone+0x6d)[0x7fd769dbf8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fd6e400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:34:12 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:34:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:34:12 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:34:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:34:12 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:34:12 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:34:12 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:34:12 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:34:12 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:34:12 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137172912 2020-02-01 0:34:12 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:34:12 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:34:12 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:34:12 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:34:12 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:34:12 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:34:12 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:34:12 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137172921; transaction id 5904215 2020-02-01 0:34:12 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:34:12 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:34:12 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:34:12 0 [Note] Starting crash recovery... 2020-02-01 0:34:12 0 [Note] Crash recovery finished. 2020-02-01 0:34:12 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:34:12 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:34:12 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:34:12 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000090' at position 1464 2020-02-01 0:34:12 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:34:12 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:34:12 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000090' at position 1464 2020-02-01 0:34:12 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:34:12 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:34:12 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:34:12 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:34:12 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:34:13 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:34:13 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000091' at position 2468 2020-02-01 0:34:13 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:34:13 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:34:13 2020-02-01 0:34:19 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000135, 42987) 2020-02-01 0:34:40 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:34:46 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:34:46 17 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:34:46 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:34:46 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:34:46 19 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:34:46 19 [Warning] Checking table: './ksarn/wp_options' 200201 0:34:46 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc208000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc26445fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ed9c85af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ed9c2f13af] sigaction.c:0(__restore_rt)[0x7fc27f626630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55ed9c21f151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55ed9c35d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55ed9c113ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55ed9c109d4b] /usr/sbin/mysqld(+0x6ca792)[0x55ed9c114792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55ed9c13c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ed9c13e65d] /usr/sbin/mysqld(+0x65e3f3)[0x55ed9c0a83f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55ed9c0a7cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55ed9c13bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ed9c13e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55ed9c13e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55ed9c13f377] /usr/sbin/mysqld(+0x58fd84)[0x55ed9bfd9d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55ed9c0e6b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55ed9c0feb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55ed9c0fecd2] /usr/sbin/mysqld(+0x6b55e7)[0x55ed9c0ff5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55ed9c0ff677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55ed9c0ee56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55ed9c0f007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55ed9c1cbafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55ed9c1cbbdd] pthread_create.c:0(start_thread)[0x7fc27f61eea5] /lib64/libc.so.6(clone+0x6d)[0x7fc27d9bf8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc20800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 20 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:34:52 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:34:52 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:34:52 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:34:52 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:34:52 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:34:52 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:34:52 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:34:52 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:34:52 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:34:52 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137175775 2020-02-01 0:34:52 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:34:52 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:34:52 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:34:52 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:34:52 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:34:52 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:34:52 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:34:52 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137175784; transaction id 5904226 2020-02-01 0:34:52 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:34:52 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:34:52 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:34:52 0 [Note] Starting crash recovery... 2020-02-01 0:34:52 0 [Note] Crash recovery finished. 2020-02-01 0:34:52 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:34:52 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:34:52 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:34:52 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000091' at position 2468 2020-02-01 0:34:52 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:34:52 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:34:52 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000091' at position 2468 2020-02-01 0:34:53 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:34:53 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:34:53 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:34:53 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:34:53 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:34:53 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:34:53 2020-02-01 0:34:55 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:34:55 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000092' at position 44964 2020-02-01 0:34:55 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:35:01 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000136, 1464) 2020-02-01 0:35:20 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:35:21 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:35:21 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:35:21 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:35:21 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:35:21 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2d94000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2db3fb3cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x556484fd1f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x556484a683af] sigaction.c:0(__restore_rt)[0x7f2e15164630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x556484996151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x556484ad4545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55648488accf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x556484880d4b] /usr/sbin/mysqld(+0x6ca792)[0x55648488b792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5564848b36da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5564848b565d] /usr/sbin/mysqld(+0x65e3f3)[0x55648481f3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55648481ecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5564848b2b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5564848b565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5564848b5761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5564848b6377] /usr/sbin/mysqld(+0x58fd84)[0x556484750d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55648485db6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x556484875b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x556484875cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5564848765e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x556484876677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55648486556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55648486707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x556484942afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x556484942bdd] pthread_create.c:0(start_thread)[0x7f2e1515cea5] /lib64/libc.so.6(clone+0x6d)[0x7f2e134fd8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f2d9400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:35:28 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:35:28 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:35:28 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:35:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:35:28 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:35:28 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:35:28 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:35:28 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:35:28 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:35:28 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137178638 2020-02-01 0:35:28 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:35:28 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:35:28 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:35:28 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:35:28 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:35:28 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:35:28 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:35:28 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137178647; transaction id 5904237 2020-02-01 0:35:28 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:35:28 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:35:28 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:35:28 0 [Note] Starting crash recovery... 2020-02-01 0:35:28 0 [Note] Crash recovery finished. 2020-02-01 0:35:28 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:35:28 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:35:28 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:35:28 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000092' at position 44964 2020-02-01 0:35:28 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:35:28 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:35:28 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000092' at position 44964 2020-02-01 0:35:28 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:35:28 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:35:28 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:35:28 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:35:28 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:35:29 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:35:29 2020-02-01 0:35:31 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:35:31 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000093' at position 133111 2020-02-01 0:35:31 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:35:37 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000137, 1464) 2020-02-01 0:35:55 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:35:57 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:35:57 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:35:57 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:35:57 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:35:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f18b4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f18b8150cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5595202aaf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55951fd413af] sigaction.c:0(__restore_rt)[0x7f1923399630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55951fc6f151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55951fdad545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55951fb63ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55951fb59d4b] /usr/sbin/mysqld(+0x6ca792)[0x55951fb64792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55951fb8c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55951fb8e65d] /usr/sbin/mysqld(+0x65e3f3)[0x55951faf83f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55951faf7cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55951fb8bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55951fb8e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55951fb8e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55951fb8f377] /usr/sbin/mysqld(+0x58fd84)[0x55951fa29d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55951fb36b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55951fb4eb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55951fb4ecd2] /usr/sbin/mysqld(+0x6b55e7)[0x55951fb4f5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55951fb4f677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55951fb3e56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55951fb4007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55951fc1bafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55951fc1bbdd] pthread_create.c:0(start_thread)[0x7f1923391ea5] /lib64/libc.so.6(clone+0x6d)[0x7f19217328cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f18b400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:36:04 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:36:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:36:04 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:36:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:36:04 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:36:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:36:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:36:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:36:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:36:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137181452 2020-02-01 0:36:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:36:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:36:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:36:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:36:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:36:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:36:04 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:36:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137181461; transaction id 5904248 2020-02-01 0:36:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:36:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:36:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:36:04 0 [Note] Starting crash recovery... 2020-02-01 0:36:04 0 [Note] Crash recovery finished. 2020-02-01 0:36:04 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:36:04 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:36:04 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:36:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000093' at position 133111 2020-02-01 0:36:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:36:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:36:04 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:36:04 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:36:04 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:36:04 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:36:04 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:36:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000093' at position 133111 2020-02-01 0:36:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:36:05 2020-02-01 0:36:09 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:36:09 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000094' at position 62356 2020-02-01 0:36:09 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:36:15 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000138, 2602) 2020-02-01 0:36:31 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:36:35 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:36:35 17 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:36:35 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:36:35 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:36:35 18 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:36:35 18 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:36:35 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f4f00000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4f5dc57cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55711b20ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55711aca53af] sigaction.c:0(__restore_rt)[0x7f4f78dca630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55711abd3151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55711ad11545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55711aac7ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55711aabdd4b] /usr/sbin/mysqld(+0x6ca792)[0x55711aac8792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55711aaf06da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55711aaf265d] /usr/sbin/mysqld(+0x65e3f3)[0x55711aa5c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55711aa5bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55711aaefb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55711aaf265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55711aaf2761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55711aaf3377] /usr/sbin/mysqld(+0x58fd84)[0x55711a98dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55711aa9ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55711aab2b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55711aab2cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55711aab35e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55711aab3677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55711aaa256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55711aaa407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55711ab7fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55711ab7fbdd] pthread_create.c:0(start_thread)[0x7f4f78dc2ea5] /lib64/libc.so.6(clone+0x6d)[0x7f4f771638cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f4f0000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:36:41 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:36:41 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:36:41 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:36:41 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:36:41 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:36:41 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:36:41 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:36:41 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:36:41 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:36:41 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137184315 2020-02-01 0:36:41 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:36:42 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:36:42 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:36:42 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:36:42 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:36:42 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:36:42 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:36:42 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137184324; transaction id 5904259 2020-02-01 0:36:42 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:36:42 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:36:42 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:36:42 0 [Note] Starting crash recovery... 2020-02-01 0:36:42 0 [Note] Crash recovery finished. 2020-02-01 0:36:42 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:36:42 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:36:42 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:36:42 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000094' at position 62356 2020-02-01 0:36:42 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:36:42 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:36:42 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000094' at position 62356 2020-02-01 0:36:42 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:36:42 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:36:42 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:36:42 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:36:42 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:36:42 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:36:42 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:36:42 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:36:42 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:36:42 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:36:43 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:36:43 2020-02-01 0:36:44 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:36:44 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000095' at position 7483 2020-02-01 0:36:44 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:36:50 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000139, 29659) 2020-02-01 0:37:11 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:37:11 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:37:11 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:37:11 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:37:11 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:37:11 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f7f54000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f7fb11c6cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55576648df2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x555765f243af] sigaction.c:0(__restore_rt)[0x7f7fcc341630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x555765e52151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x555765f90545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x555765d46ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x555765d3cd4b] /usr/sbin/mysqld(+0x6ca792)[0x555765d47792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x555765d6f6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x555765d7165d] /usr/sbin/mysqld(+0x65e3f3)[0x555765cdb3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x555765cdacb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x555765d6eb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x555765d7165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x555765d71761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x555765d72377] /usr/sbin/mysqld(+0x58fd84)[0x555765c0cd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x555765d19b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x555765d31b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x555765d31cd2] /usr/sbin/mysqld(+0x6b55e7)[0x555765d325e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x555765d32677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x555765d2156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x555765d2307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x555765dfeafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x555765dfebdd] pthread_create.c:0(start_thread)[0x7f7fcc339ea5] /lib64/libc.so.6(clone+0x6d)[0x7f7fca6da8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f7f5400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:37:18 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:37:18 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:37:18 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:37:18 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:37:18 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:37:18 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:37:18 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:37:18 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:37:18 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:37:18 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137187188 2020-02-01 0:37:18 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:37:18 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:37:18 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:37:18 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:37:18 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:37:18 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:37:18 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:37:18 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137187197; transaction id 5904271 2020-02-01 0:37:18 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:37:18 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:37:18 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:37:18 0 [Note] Starting crash recovery... 2020-02-01 0:37:18 0 [Note] Crash recovery finished. 2020-02-01 0:37:18 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:37:18 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:37:18 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:37:18 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000095' at position 7483 2020-02-01 0:37:18 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000095' at position 7483 2020-02-01 0:37:18 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:37:18 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:37:18 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:37:18 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:37:18 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:37:18 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:37:18 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:37:18 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:37:18 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000096' at position 11517 2020-02-01 0:37:18 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:37:19 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:37:19 2020-02-01 0:37:24 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000140, 1454) 2020-02-01 0:37:46 19 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:37:47 20 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:37:47 20 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:37:48 22 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-02-01 0:37:48 22 [Warning] Checking table: './openrecords/wp_options' 2020-02-01 0:37:54 26 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:37:54 26 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:37:56 30 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:37:56 30 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:37:56 31 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:37:56 31 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:37:56 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f30d8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f312c078cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x556d67d46f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x556d677dd3af] sigaction.c:0(__restore_rt)[0x7f314658e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x556d6770b151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x556d67849545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x556d675ffccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x556d675f5d4b] /usr/sbin/mysqld(+0x6ca792)[0x556d67600792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x556d676286da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556d6762a65d] /usr/sbin/mysqld(+0x65e3f3)[0x556d675943f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x556d67593cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x556d67627b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556d6762a65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x556d6762a761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x556d6762b377] /usr/sbin/mysqld(+0x58fd84)[0x556d674c5d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x556d675d2b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x556d675eab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x556d675eacd2] /usr/sbin/mysqld(+0x6b55e7)[0x556d675eb5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x556d675eb677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x556d675da56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x556d675dc07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x556d676b7afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x556d676b7bdd] pthread_create.c:0(start_thread)[0x7f3146586ea5] /lib64/libc.so.6(clone+0x6d)[0x7f31449278cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f30d800feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 33 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:38:03 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:38:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:38:03 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:38:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:38:03 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:38:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:38:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:38:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:38:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:38:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137190842 2020-02-01 0:38:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:38:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:38:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:38:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:38:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:38:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:38:03 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:38:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137190851; transaction id 5904282 2020-02-01 0:38:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:38:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:38:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:38:03 0 [Note] Starting crash recovery... 2020-02-01 0:38:03 0 [Note] Crash recovery finished. 2020-02-01 0:38:03 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:38:03 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:38:03 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:38:03 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000096' at position 11517 2020-02-01 0:38:03 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:38:03 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:38:03 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:38:03 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:38:03 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:38:03 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:38:03 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:38:03 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000096' at position 11517 2020-02-01 0:38:04 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:38:04 2020-02-01 0:38:06 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:38:06 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000097' at position 78532 2020-02-01 0:38:06 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:38:12 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000141, 140154) 2020-02-01 0:38:31 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:38:36 15 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:38:36 15 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:38:45 28 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:38:45 28 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:38:45 29 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:38:45 29 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:38:45 29 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:38:45 29 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:38:45 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f06bc0009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0729787cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55dc4cf39f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55dc4c9d03af] sigaction.c:0(__restore_rt)[0x7f0744862630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55dc4c8fe151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55dc4ca3c545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55dc4c7f2ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55dc4c7e8d4b] /usr/sbin/mysqld(+0x6ca792)[0x55dc4c7f3792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55dc4c81b6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55dc4c81d65d] /usr/sbin/mysqld(+0x65e3f3)[0x55dc4c7873f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55dc4c786cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55dc4c81ab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55dc4c81d65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55dc4c81d761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55dc4c81e377] /usr/sbin/mysqld(+0x58fd84)[0x55dc4c6b8d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55dc4c7c5b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55dc4c7ddb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55dc4c7ddcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55dc4c7de5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55dc4c7de677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55dc4c7cd56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55dc4c7cf07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55dc4c8aaafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55dc4c8aabdd] pthread_create.c:0(start_thread)[0x7f074485aea5] /lib64/libc.so.6(clone+0x6d)[0x7f0742bfb8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f06bc010950): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 31 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:38:51 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:38:51 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:38:51 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:38:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:38:51 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:38:51 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:38:51 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:38:51 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:38:51 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:38:51 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137193704 2020-02-01 0:38:51 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:38:52 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:38:52 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:38:52 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:38:52 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:38:52 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:38:52 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137193713; transaction id 5904293 2020-02-01 0:38:52 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:38:52 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:38:52 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:38:52 0 [Note] Starting crash recovery... 2020-02-01 0:38:52 0 [Note] Crash recovery finished. 2020-02-01 0:38:52 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:38:52 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:38:52 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:38:52 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000097' at position 78532 2020-02-01 0:38:52 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:38:52 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:38:52 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000097' at position 78532 2020-02-01 0:38:52 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:38:52 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:38:52 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:38:52 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:38:52 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:38:52 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:38:52 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:38:52 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:38:52 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:38:52 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:38:53 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:38:53 2020-02-01 0:38:57 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:38:57 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000098' at position 90245 2020-02-01 0:38:57 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:39:02 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000142, 114535) 2020-02-01 0:39:17 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:39:18 15 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-02-01 0:39:18 15 [Warning] Checking table: './wpcomm305/comm305_options' 2020-02-01 0:39:19 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:39:19 17 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:39:19 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:39:19 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:39:20 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f5460000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f54c06d8cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5622cbd6df2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5622cb8043af] sigaction.c:0(__restore_rt)[0x7f54db7e3630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5622cb732151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5622cb870545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5622cb626ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5622cb61cd4b] /usr/sbin/mysqld(+0x6ca792)[0x5622cb627792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5622cb64f6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5622cb65165d] /usr/sbin/mysqld(+0x65e3f3)[0x5622cb5bb3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5622cb5bacb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5622cb64eb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5622cb65165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5622cb651761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5622cb652377] /usr/sbin/mysqld(+0x58fd84)[0x5622cb4ecd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5622cb5f9b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5622cb611b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5622cb611cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5622cb6125e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5622cb612677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5622cb60156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5622cb60307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5622cb6deafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5622cb6debdd] pthread_create.c:0(start_thread)[0x7f54db7dbea5] /lib64/libc.so.6(clone+0x6d)[0x7f54d9b7c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f5460010a30): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:39:26 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:39:26 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:39:26 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:39:26 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:39:26 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:39:26 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:39:26 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:39:26 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:39:26 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:39:26 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137196567 2020-02-01 0:39:26 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:39:26 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:39:26 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:39:26 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:39:26 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:39:26 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:39:26 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:39:26 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137196576; transaction id 5904304 2020-02-01 0:39:26 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:39:26 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:39:26 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:39:26 0 [Note] Starting crash recovery... 2020-02-01 0:39:26 0 [Note] Crash recovery finished. 2020-02-01 0:39:26 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:39:26 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:39:26 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:39:26 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000098' at position 90245 2020-02-01 0:39:26 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:39:26 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:39:26 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000098' at position 90245 2020-02-01 0:39:26 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:39:26 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:39:26 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:39:26 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:39:26 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:39:27 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:39:27 2020-02-01 0:39:36 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:39:36 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000099' at position 99831 2020-02-01 0:39:36 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:39:42 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000143, 41987) 2020-02-01 0:39:52 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:39:54 16 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:39:54 16 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:39:54 18 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-02-01 0:39:54 17 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-02-01 0:39:54 18 [Warning] Checking table: './shockeripe/wp_options' 2020-02-01 0:39:54 17 [Warning] Checking table: './ulrich/wp_options' 2020-02-01 0:39:55 20 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:39:55 20 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:39:57 24 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:39:57 24 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:39:57 25 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:39:57 25 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:39:57 25 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:39:57 25 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:39:58 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=8 max_threads=502 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f40d0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4078075cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x564ece2bef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x564ecdd553af] sigaction.c:0(__restore_rt)[0x7f40e4972630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564ecdc83151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564ecddc1545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564ecdb77ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x564ecdb6dd4b] /usr/sbin/mysqld(+0x6ca792)[0x564ecdb78792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x564ecdba06da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564ecdba265d] /usr/sbin/mysqld(+0x65e3f3)[0x564ecdb0c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x564ecdb0bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x564ecdb9fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564ecdba265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x564ecdba2761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x564ecdba3377] /usr/sbin/mysqld(+0x58fd84)[0x564ecda3dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x564ecdb4ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564ecdb62b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564ecdb62cd2] /usr/sbin/mysqld(+0x6b55e7)[0x564ecdb635e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564ecdb63677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x564ecdb5256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x564ecdb5407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x564ecdc2fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x564ecdc2fbdd] pthread_create.c:0(start_thread)[0x7f40e496aea5] /lib64/libc.so.6(clone+0x6d)[0x7f40e2d0b8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f40d000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 27 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:40:04 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:40:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:40:04 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:40:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:40:04 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:40:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:40:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:40:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:40:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:40:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137199414 2020-02-01 0:40:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:40:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:40:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:40:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:40:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:40:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:40:04 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:40:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137199423; transaction id 5904315 2020-02-01 0:40:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:40:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:40:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:40:04 0 [Note] Starting crash recovery... 2020-02-01 0:40:04 0 [Note] Crash recovery finished. 2020-02-01 0:40:04 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:40:04 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:40:04 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:40:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000099' at position 99831 2020-02-01 0:40:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:40:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:40:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000099' at position 99831 2020-02-01 0:40:04 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:40:04 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:40:04 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:40:04 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:40:04 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:40:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:40:05 2020-02-01 0:40:11 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:40:11 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000100' at position 2513 2020-02-01 0:40:11 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:40:17 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000144, 226244) 2020-02-01 0:40:32 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:40:46 21 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:40:47 21 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:40:47 22 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:40:47 22 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:40:47 22 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:40:47 22 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:40:47 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f89cc000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f8a29211cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e004cdbf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e0047723af] sigaction.c:0(__restore_rt)[0x7f8a443a2630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55e0046a0151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55e0047de545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55e004594ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55e00458ad4b] /usr/sbin/mysqld(+0x6ca792)[0x55e004595792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55e0045bd6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e0045bf65d] /usr/sbin/mysqld(+0x65e3f3)[0x55e0045293f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55e004528cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55e0045bcb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e0045bf65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55e0045bf761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55e0045c0377] /usr/sbin/mysqld(+0x58fd84)[0x55e00445ad84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55e004567b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55e00457fb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55e00457fcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55e0045805e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55e004580677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55e00456f56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55e00457107c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55e00464cafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e00464cbdd] pthread_create.c:0(start_thread)[0x7f8a4439aea5] /lib64/libc.so.6(clone+0x6d)[0x7f8a4273b8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f89cc00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 23 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:40:53 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:40:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:40:53 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:40:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:40:53 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:40:53 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:40:53 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:40:53 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:40:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:40:53 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137202394 2020-02-01 0:40:53 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:40:53 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:40:53 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:40:53 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:40:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:40:53 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:40:53 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:40:53 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137202403; transaction id 5904326 2020-02-01 0:40:53 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:40:53 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:40:53 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:40:53 0 [Note] Starting crash recovery... 2020-02-01 0:40:53 0 [Note] Crash recovery finished. 2020-02-01 0:40:53 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:40:53 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:40:53 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:40:53 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000100' at position 2513 2020-02-01 0:40:53 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:40:53 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:40:53 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000100' at position 2513 2020-02-01 0:40:53 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:40:53 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:40:53 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:40:53 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:40:53 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:40:54 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:40:54 2020-02-01 0:40:55 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:40:55 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000101' at position 1464 2020-02-01 0:40:55 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:41:01 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000145, 53170) 2020-02-01 0:41:22 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:41:23 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:41:23 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:41:23 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:41:23 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:41:23 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f7514000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f756c066cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x558e19631f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x558e190c83af] sigaction.c:0(__restore_rt)[0x7f758630f630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x558e18ff6151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x558e19134545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x558e18eeaccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x558e18ee0d4b] /usr/sbin/mysqld(+0x6ca792)[0x558e18eeb792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x558e18f136da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x558e18f1565d] /usr/sbin/mysqld(+0x65e3f3)[0x558e18e7f3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x558e18e7ecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x558e18f12b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x558e18f1565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x558e18f15761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x558e18f16377] /usr/sbin/mysqld(+0x58fd84)[0x558e18db0d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x558e18ebdb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x558e18ed5b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x558e18ed5cd2] /usr/sbin/mysqld(+0x6b55e7)[0x558e18ed65e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x558e18ed6677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x558e18ec556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x558e18ec707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x558e18fa2afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x558e18fa2bdd] pthread_create.c:0(start_thread)[0x7f7586307ea5] /lib64/libc.so.6(clone+0x6d)[0x7f75846a88cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f751400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:41:29 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:41:29 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:41:29 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:41:29 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:41:29 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:41:29 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:41:29 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:41:29 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:41:29 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:41:29 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137205374 2020-02-01 0:41:29 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:41:30 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:41:30 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:41:30 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:41:30 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:41:30 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:41:30 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:41:30 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137205383; transaction id 5904337 2020-02-01 0:41:30 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:41:30 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:41:30 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:41:30 0 [Note] Starting crash recovery... 2020-02-01 0:41:30 0 [Note] Crash recovery finished. 2020-02-01 0:41:30 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:41:30 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:41:30 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:41:30 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000101' at position 1464 2020-02-01 0:41:30 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:41:30 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:41:30 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000101' at position 1464 2020-02-01 0:41:30 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:41:30 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:41:30 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:41:30 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:41:30 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:41:31 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:41:31 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000102' at position 26688 2020-02-01 0:41:31 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:41:31 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:41:31 2020-02-01 0:41:36 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000146, 1464) 2020-02-01 0:41:57 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:42:04 18 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:42:04 18 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:42:06 21 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:42:06 21 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:42:06 22 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:42:06 22 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:42:07 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f6298000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f628c7f7cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55f789fa7f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55f789a3e3af] sigaction.c:0(__restore_rt)[0x7f6306289630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55f78996c151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55f789aaa545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55f789860ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55f789856d4b] /usr/sbin/mysqld(+0x6ca792)[0x55f789861792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55f7898896da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55f78988b65d] /usr/sbin/mysqld(+0x65e3f3)[0x55f7897f53f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55f7897f4cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55f789888b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55f78988b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55f78988b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55f78988c377] /usr/sbin/mysqld(+0x58fd84)[0x55f789726d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55f789833b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55f78984bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55f78984bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55f78984c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55f78984c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55f78983b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55f78983d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55f789918afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55f789918bdd] pthread_create.c:0(start_thread)[0x7f6306281ea5] /lib64/libc.so.6(clone+0x6d)[0x7f63046228cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f629800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 23 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:42:12 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:42:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:42:12 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:42:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:42:12 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:42:12 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:42:12 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:42:12 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:42:12 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:42:12 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137208221 2020-02-01 0:42:12 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:42:13 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:42:13 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:42:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:42:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:42:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:42:13 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:42:13 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137208230; transaction id 5904348 2020-02-01 0:42:13 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:42:13 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:42:13 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:42:13 0 [Note] Starting crash recovery... 2020-02-01 0:42:13 0 [Note] Crash recovery finished. 2020-02-01 0:42:13 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:42:13 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:42:13 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:42:13 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000102' at position 26688 2020-02-01 0:42:13 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:42:13 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:42:13 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:42:13 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:42:13 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:42:13 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:42:13 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:42:13 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:42:13 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:42:13 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:42:13 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:42:13 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:42:13 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:42:13 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:42:13 2020-02-01 0:42:17 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000147, 49525) 2020-02-01 0:42:38 15 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:42:39 16 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:42:39 16 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:42:45 19 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-02-01 0:42:45 19 [Warning] Checking table: './wpcomm303/wp_options' 2020-02-01 0:42:46 21 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:42:46 21 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:42:46 22 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:42:46 22 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:42:46 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f65e0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f663cf5ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55fa4caa0f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55fa4c5373af] sigaction.c:0(__restore_rt)[0x7f6658190630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55fa4c465151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55fa4c5a3545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55fa4c359ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55fa4c34fd4b] /usr/sbin/mysqld(+0x6ca792)[0x55fa4c35a792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55fa4c3826da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55fa4c38465d] /usr/sbin/mysqld(+0x65e3f3)[0x55fa4c2ee3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55fa4c2edcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55fa4c381b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55fa4c38465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55fa4c384761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55fa4c385377] /usr/sbin/mysqld(+0x58fd84)[0x55fa4c21fd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55fa4c32cb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55fa4c344b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55fa4c344cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55fa4c3455e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55fa4c345677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55fa4c33456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55fa4c33607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55fa4c411afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55fa4c411bdd] pthread_create.c:0(start_thread)[0x7f6658188ea5] /lib64/libc.so.6(clone+0x6d)[0x7f66565298cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f65e000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 23 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:42:53 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:42:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:42:53 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:42:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:42:53 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:42:53 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:42:53 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:42:53 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:42:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:42:53 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137208585 2020-02-01 0:42:53 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:42:53 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:42:53 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:42:53 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:42:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:42:53 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:42:53 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:42:53 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137208594; transaction id 5904353 2020-02-01 0:42:53 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:42:53 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:42:53 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:42:53 0 [Note] Starting crash recovery... 2020-02-01 0:42:53 0 [Note] Crash recovery finished. 2020-02-01 0:42:53 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:42:53 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:42:53 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:42:53 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000102' at position 26688 2020-02-01 0:42:53 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:42:53 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:42:53 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000102' at position 26688 2020-02-01 0:42:53 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:42:53 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:42:53 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:42:53 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:42:53 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:42:53 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:42:53 2020-02-01 0:43:06 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:43:06 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000104' at position 78759 2020-02-01 0:43:06 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:43:09 13 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-02-01 0:43:12 14 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000148, 45586) 2020-02-01 0:43:18 16 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:43:20 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:43:20 17 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:43:20 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:43:20 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:43:20 18 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:43:20 18 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:43:20 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=502 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f949c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f94f9a3acf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55eb35c3af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55eb356d13af] sigaction.c:0(__restore_rt)[0x7f9514c15630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55eb355ff151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55eb3573d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55eb354f3ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55eb354e9d4b] /usr/sbin/mysqld(+0x6ca792)[0x55eb354f4792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55eb3551c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55eb3551e65d] /usr/sbin/mysqld(+0x65e3f3)[0x55eb354883f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55eb35487cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55eb3551bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55eb3551e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55eb3551e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55eb3551f377] /usr/sbin/mysqld(+0x58fd84)[0x55eb353b9d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55eb354c6b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55eb354deb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55eb354decd2] /usr/sbin/mysqld(+0x6b55e7)[0x55eb354df5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55eb354df677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55eb354ce56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55eb354d007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55eb355abafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55eb355abbdd] pthread_create.c:0(start_thread)[0x7f9514c0dea5] /lib64/libc.so.6(clone+0x6d)[0x7f9512fae8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f949c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:43:27 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:43:27 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:43:27 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:43:27 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:43:27 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:43:27 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:43:27 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:43:27 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:43:27 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:43:27 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137208928 2020-02-01 0:43:27 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:43:27 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:43:27 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:43:27 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:43:27 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:43:27 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:43:27 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:43:27 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137208937; transaction id 5904358 2020-02-01 0:43:27 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:43:27 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:43:27 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:43:27 0 [Note] Starting crash recovery... 2020-02-01 0:43:27 0 [Note] Crash recovery finished. 2020-02-01 0:43:27 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:43:27 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:43:27 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:43:27 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000104' at position 78759 2020-02-01 0:43:27 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:43:27 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:43:27 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000104' at position 78759 2020-02-01 0:43:27 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:43:27 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:43:27 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:43:27 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:43:27 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:43:28 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:43:28 2020-02-01 0:43:34 12 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:43:34 12 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:43:35 12 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:43:35 12 [Warning] Checking table: './ojsdata/controlled_vocabs' 2020-02-01 0:43:35 12 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:43:35 12 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:43:35 12 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:43:35 12 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:43:35 12 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-02-01 0:43:35 12 [Warning] Checking table: './openrecords/wp_options' 2020-02-01 0:43:35 12 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-02-01 0:43:35 12 [Warning] Checking table: './pizzahut/wp_options' 2020-02-01 0:43:35 12 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-02-01 0:43:35 12 [Warning] Checking table: './shockeripe/wp_options' 2020-02-01 0:43:35 12 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-02-01 0:43:35 12 [Warning] Checking table: './ulrich/wp_options' 2020-02-01 0:43:35 12 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-02-01 0:43:35 12 [Warning] Checking table: './wpcomm305/comm305_options' 2020-02-01 0:43:35 12 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:43:35 12 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:43:35 12 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-02-01 0:43:35 12 [Warning] Checking table: './wpsem/wp_options' 2020-02-01 0:43:36 12 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-02-01 0:43:36 12 [Warning] Checking table: './wpstart/wp_options' 2020-02-01 0:43:43 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:43:43 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000105' at position 47081 2020-02-01 0:43:43 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:43:48 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000149, 2513) 2020-02-01 0:43:53 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 200201 0:43:56 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=502 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f1a9c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f1a50075cf0 thread_stack 0x49000 *** buffer overflow detected ***: /usr/sbin/mysqld terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7f1abac51567] /lib64/libc.so.6(+0x1166e2)[0x7f1abac4f6e2] /lib64/libc.so.6(+0x1184c7)[0x7f1abac514c7] /usr/sbin/mysqld(my_addr_resolve+0xda)[0x55e1d48b594a] /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x55e1d489e0c2] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e1d43343af] /lib64/libpthread.so.0(+0xf630)[0x7f1abc89e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55e1d4262151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55e1d43a0545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55e1d4156ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55e1d414cd4b] /usr/sbin/mysqld(+0x6ca792)[0x55e1d4157792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55e1d417f6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e1d418165d] /usr/sbin/mysqld(+0x65e3f3)[0x55e1d40eb3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55e1d40eacb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55e1d417eb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e1d418165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55e1d4181761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55e1d4182377] /usr/sbin/mysqld(+0x58fd84)[0x55e1d401cd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55e1d4129b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55e1d4141b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55e1d4141cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55e1d41425e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55e1d4142677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55e1d413156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55e1d413307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55e1d420eafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e1d420ebdd] /lib64/libpthread.so.0(+0x7ea5)[0x7f1abc896ea5] /lib64/libc.so.6(clone+0x6d)[0x7f1abac378cd] ======= Memory map: ======== 55e1d3a8d000-55e1d4edb000 r-xp 00000000 fd:00 2101454 /usr/sbin/mysqld 55e1d50da000-55e1d5206000 r--p 0144d000 fd:00 2101454 /usr/sbin/mysqld 55e1d5206000-55e1d52c0000 rw-p 01579000 fd:00 2101454 /usr/sbin/mysqld 55e1d52c0000-55e1d5b6f000 rw-p 00000000 00:00 0 55e1d73a4000-55e1db948000 rw-p 00000000 00:00 0 [heap] 7f1a18000000-7f1a1802a000 rw-p 00000000 00:00 0 7f1a1802a000-7f1a1c000000 ---p 00000000 00:00 0 7f1a1c000000-7f1a1c021000 rw-p 00000000 00:00 0 7f1a1c021000-7f1a20000000 ---p 00000000 00:00 0 7f1a20000000-7f1a2003e000 rw-p 00000000 00:00 0 7f1a2003e000-7f1a24000000 ---p 00000000 00:00 0 7f1a24000000-7f1a2402b000 rw-p 00000000 00:00 0 7f1a2402b000-7f1a28000000 ---p 00000000 00:00 0 7f1a28000000-7f1a2812f000 rw-p 00000000 00:00 0 7f1a2812f000-7f1a2c000000 ---p 00000000 00:00 0 7f1a2c000000-7f1a2c021000 rw-p 00000000 00:00 0 7f1a2c021000-7f1a30000000 ---p 00000000 00:00 0 7f1a30000000-7f1a3002f000 rw-p 00000000 00:00 0 7f1a3002f000-7f1a34000000 ---p 00000000 00:00 0 7f1a34000000-7f1a34021000 rw-p 00000000 00:00 0 7f1a34021000-7f1a38000000 ---p 00000000 00:00 0 7f1a38000000-7f1a3a1b7000 rw-p 00000000 00:00 0 7f1a3a1b7000-7f1a3c000000 ---p 00000000 00:00 0 7f1a3c000000-7f1a3c043000 rw-p 00000000 00:00 0 7f1a3c043000-7f1a40000000 ---p 00000000 00:00 0 7f1a405e4000-7f1a405f6000 r-xp 00000000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f1a405f6000-7f1a407f5000 ---p 00012000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f1a407f5000-7f1a407f8000 r--p 00011000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f1a407f8000-7f1a407f9000 rw-p 00014000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f1a407f9000-7f1a407fa000 ---p 00000000 00:00 0 7f1a407fa000-7f1a40ffa000 rw-p 00000000 00:00 0 7f1a40ffa000-7f1a40ffb000 ---p 00000000 00:00 0 7f1a40ffb000-7f1a417fb000 rw-p 00000000 00:00 0 7f1a417fb000-7f1a417fc000 ---p 00000000 00:00 0 7f1a417fc000-7f1a41ffc000 rw-p 00000000 00:00 0 7f1a41ffc000-7f1a41ffd000 ---p 00000000 00:00 0 7f1a41ffd000-7f1a427fd000 rw-p 00000000 00:00 0 7f1a427fd000-7f1a427fe000 ---p 00000000 00:00 0 7f1a427fe000-7f1a42ffe000 rw-p 00000000 00:00 0 7f1a42ffe000-7f1a42fff000 ---p 00000000 00:00 0 7f1a42fff000-7f1a437ff000 rw-p 00000000 00:00 0 7f1a437ff000-7f1a43800000 ---p 00000000 00:00 0 7f1a43800000-7f1a44000000 rw-p 00000000 00:00 0 7f1a44000000-7f1a4406b000 rw-p 00000000 00:00 0 7f1a4406b000-7f1a48000000 ---p 00000000 00:00 0 7f1a48000000-7f1a483fb000 rw-p 00000000 00:00 0 7f1a483fb000-7f1a4c000000 ---p 00000000 00:00 0 7f1a4c000000-7f1a4c3f7000 rw-p 00000000 00:00 0 7f1a4c3f7000-7f1a50000000 ---p 00000000 00:00 0 7f1a5002c000-7f1a5002d000 ---p 00000000 00:00 0 7f1a5002d000-7f1a50077000 rw-p 00000000 00:00 0 7f1a50077000-7f1a50078000 ---p 00000000 00:00 0 7f1a50078000-7f1a500c2000 rw-p 00000000 00:00 0 7f1a500c2000-7f1a500c3000 ---p 00000000 00:00 0 7f1a500c3000-7f1a5010d000 rw-p 00000000 00:00 0 7f1a5010d000-7f1a5010e000 ---p 00000000 00:00 0 7f1a5010e000-7f1a50158000 rw-p 00000000 00:00 0 7f1a50158000-7f1a5015e000 r-xp 00000000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f1a5015e000-7f1a5035d000 ---p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f1a5035d000-7f1a5035e000 r--p 00005000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f1a5035e000-7f1a5035f000 rw-p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f1a5035f000-7f1a50360000 ---p 00000000 00:00 0 7f1a50360000-7f1a50b60000 rw-p 00000000 00:00 0 7f1a50b60000-7f1a50b61000 ---p 00000000 00:00 0 7f1a50b61000-7f1a51361000 rw-p 00000000 00:00 0 7f1a51361000-7f1a51362000 ---p 00000000 00:00 0 7f1a51362000-7f1a51b62000 rw-p 00000000 00:00 0 7f1a51b62000-7f1a51b63000 ---p 00000000 00:00 0 7f1a51b63000-7f1a52568000 rw-p 00000000 00:00 0 7f1a52568000-7f1a52569000 ---p 00000000 00:00 0 7f1a52569000-7f1a52d69000 rw-p 00000000 00:00 0 7f1a52d69000-7f1a52d6a000 ---p 00000000 00:00 0 7f1a52d6a000-7f1a5356a000 rw-p 00000000 00:00 0 7f1a5356a000-7f1a5356b000 ---p 00000000 00:00 0 7f1a5356b000-7f1a53d6b000 rw-p 00000000 00:00 0 7f1a53d6b000-7f1a53d6c000 ---p 00000000 00:00 0 7f1a53d6c000-7f1a5456c000 rw-p 00000000 00:00 0 7f1a5456c000-7f1a5456d000 ---p 00000000 00:00 0 7f1a5456d000-7f1a54d6d000 rw-p 00000000 00:00 0 7f1a54d6d000-7f1a54d6e000 ---p 00000000 00:00 0 7f1a54d6e000-7f1a5556e000 rw-p 00000000 00:00 0 7f1a5556e000-7f1a5556f000 ---p 00000000 00:00 0 7f1a5556f000-7f1a55d6f000 rw-p 00000000 00:00 0 7f1a55d6f000-7f1a55d70000 ---p 00000000 00:00 0 7f1a55d70000-7f1a56570000 rw-p 00000000 00:00 0 7f1a56570000-7f1a56571000 ---p 00000000 00:00 0 7f1a56571000-7f1a56d71000 rw-p 00000000 00:00 0 7f1a56d71000-7f1a56d72000 ---p 00000000 00:00 0 7f1a56d72000-7f1a57572000 rw-p 00000000 00:00 0 7f1a57572000-7f1a57573000 ---p 00000000 00:00 0 7f1a57573000-7f1a57d73000 rw-p 00000000 00:00 0 7f1a57d73000-7f1a57d74000 ---p 00000000 00:00 0 7f1a57d74000-7f1a58574000 rw-p 00000000 00:00 0 7f1a58574000-7f1a58575000 ---p 00000000 00:00 0 7f1a58575000-7f1a58d75000 rw-p 00000000 00:00 0 7f1a58d75000-7f1a58d76000 ---p 00000000 00:00 0 7f1a58d76000-7f1a59576000 rw-p 00000000 00:00 0 7f1a59576000-7f1a59577000 ---p 00000000 00:00 0 7f1a59577000-7f1a5a000000 rw-p 00000000 00:00 0 7f1a5a000000-7f1a9c000000 rw-p 00000000 00:00 0 7f1a9c000000-7f1a9c088000 rw-p 00000000 00:00 0 7f1a9c088000-7f1aa0000000 ---p 00000000 00:00 0 7f1aa002b000-7f1aa002c000 ---p 00000000 00:00 0 7f1aa002c000-7f1aa0588000 rw-p 00000000 00:00 0 7f1aa05bf000-7f1aa05c0000 ---p 00000000 00:00 0 7f1aa05c0000-7f1aa060a000 rw-p 00000000 00:00 0 7f1aa060a000-7f1aa060b000 ---p 00000000 00:00 0 7f1aa060b000-7f1aa0655000 rw-p 00000000 00:00 0 7f1aa0655000-7f1aa0661000 r-xp 00000000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f1aa0661000-7f1aa0860000 ---p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f1aa0860000-7f1aa0861000 r--p 0000b000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f1aa0861000-7f1aa0862000 rw-p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f1aa0862000-7f1aa0868000 rw-p 00000000 00:00 0 7f1aa0868000-7f1aa0869000 ---p 00000000 00:00 0 7f1aa0869000-7f1aa1069000 rw-p 00000000 00:00 0 7f1aa1069000-7f1aa106a000 ---p 00000000 00:00 0 7f1aa106a000-7f1ab0000000 rw-p 00000000 00:00 0 7f1ab0000000-7f1ab0021000 rw-p 00000000 00:00 0 7f1ab0021000-7f1ab4000000 ---p 00000000 00:00 0 7f1ab4001000-7f1ab4002000 ---p 00000000 00:00 0 7f1ab4002000-7f1ab404c000 rw-p 00000000 00:00 0 7f1ab404c000-7f1ab404d000 ---p 00000000 00:00 0 7f1ab404d000-7f1ab5146000 rw-p 00000000 00:00 0 7f1ab5146000-7f1ab5147000 ---p 00000000 00:00 0 7f1ab5147000-7f1ab77d6000 rw-p 00000000 00:00 0 7f1ab77d6000-7f1ab77d7000 ---p 00000000 00:00 0 7f1ab77d7000-7f1ab7fd7000 rw-p 00000000 00:00 0 7f1ab7fd7000-7f1ab7fda000 r-xp 00000000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f1ab7fda000-7f1ab81d9000 ---p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f1ab81d9000-7f1ab81da000 r--p 00002000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f1ab81da000-7f1ab81db000 rw-p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f1ab81db000-7f1ab81e9000 r-xp 00000000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f1ab81e9000-7f1ab83e9000 ---p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f1ab83e9000-7f1ab83ea000 r--p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f1ab83ea000-7f1ab83eb000 rw-p 0000f000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f1ab83eb000-7f1ab83fa000 r-xp 00000000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f1ab83fa000-7f1ab85f9000 ---p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f1ab85f9000-7f1ab85fa000 r--p 0000e000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f1ab85fa000-7f1ab85fb000 rw-p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f1ab85fb000-7f1ab8612000 r-xp 00000000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f1ab8612000-7f1ab8811000 ---p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f1ab8811000-7f1ab8812000 r--p 00016000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f1ab8812000-7f1ab8813000 rw-p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f1ab8813000-7f1ab8873000 r-xp 00000000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f1ab8873000-7f1ab8a73000 ---p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f1ab8a73000-7f1ab8a74000 r--p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f1ab8a74000-7f1ab8a75000 rw-p 00061000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f1ab8a75000-7f1ab8a79000 r-xp 00000000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f1ab8a79000-7f1ab8c78000 ---p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f1ab8c78000-7f1ab8c79000 r--p 00003000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f1ab8c79000-7f1ab8c7a000 rw-p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f1ab8c7a000-7f1ab8cab000 r-xp 00000000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f1ab8cab000-7f1ab8eaa000 ---p 00031000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f1ab8eaa000-7f1ab8eac000 r--p 00030000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f1ab8eac000-7f1ab8ead000 rw-p 00032000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f1ab8ead000-7f1ab8eb0000 r-xp 00000000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f1ab8eb0000-7f1ab90af000 ---p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f1ab90af000-7f1ab90b0000 r--p 00002000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f1ab90b0000-7f1ab90b1000 rw-p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f1ab90b1000-7f1ab918a000 r-xp 00000000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f1ab918a000-7f1ab9389000 ---p 000d9000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f1ab9389000-7f1ab9397000 r--p 000d8000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f1ab9397000-7f1ab939a000 rw-p 000e6000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f1ab939a000-7f1ab93e4000 r-xp 00000000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f1ab93e4000-7f1ab95e4000 ---p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f1ab95e4000-7f1ab95e5000 r--p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f1ab95e5000-7f1ab95e7000 rw-p 0004b000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f1ab95e7000-7f1ab95fc000 r-xp 00000000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f1ab95fc000-7f1ab97fb000 ---p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f1ab97fb000-7f1ab97fc000 r--p 00014000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f1ab97fc000-7f1ab97fd000 rw-p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f1ab97fd000-7f1ab984b000 r-xp 00000000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f1ab984b000-7f1ab9a4b000 ---p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f1ab9a4b000-7f1ab9a4d000 r--p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f1ab9a4d000-7f1ab9a4e000 rw-p 00050000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f1ab9a4e000-7f1ab9a64000 r-xp 00000000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f1ab9a64000-7f1ab9c63000 ---p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f1ab9c63000-7f1ab9c64000 r--p 00015000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f1ab9c64000-7f1ab9c65000 rw-p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f1ab9c65000-7f1ab9c67000 rw-p 00000000 00:00 0 7f1ab9c67000-7f1ab9c6b000 r-xp 00000000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f1ab9c6b000-7f1ab9e6a000 ---p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f1ab9e6a000-7f1ab9e6b000 r--p 00003000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f1ab9e6b000-7f1ab9e6c000 rw-p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f1ab9e6c000-7f1ab9ee9000 r-xp 00000000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f1ab9ee9000-7f1aba0e8000 ---p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f1aba0e8000-7f1aba0e9000 r--p 0007c000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f1aba0e9000-7f1aba0ec000 rw-p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f1aba0ec000-7f1aba0ed000 rw-p 00000000 00:00 0 7f1aba0ed000-7f1aba101000 r-xp 00000000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f1aba101000-7f1aba300000 ---p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f1aba300000-7f1aba301000 r--p 00013000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f1aba301000-7f1aba302000 rw-p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f1aba302000-7f1aba326000 r-xp 00000000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f1aba326000-7f1aba525000 ---p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f1aba525000-7f1aba526000 r--p 00023000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f1aba526000-7f1aba527000 rw-p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f1aba527000-7f1aba529000 rw-p 00000000 00:00 0 7f1aba529000-7f1aba530000 r-xp 00000000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f1aba530000-7f1aba72f000 ---p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f1aba72f000-7f1aba730000 r--p 00006000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f1aba730000-7f1aba731000 rw-p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f1aba731000-7f1aba735000 r-xp 00000000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f1aba735000-7f1aba934000 ---p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f1aba934000-7f1aba935000 r--p 00003000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f1aba935000-7f1aba936000 rw-p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f1aba936000-7f1aba938000 r-xp 00000000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f1aba938000-7f1abab37000 ---p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f1abab37000-7f1abab38000 r--p 00001000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f1abab38000-7f1abab39000 rw-p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f1abab39000-7f1abacfc000 r-xp 00000000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f1abacfc000-7f1abaefc000 ---p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f1abaefc000-7f1abaf00000 r--p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f1abaf00000-7f1abaf02000 rw-p 001c7000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f1abaf02000-7f1abaf07000 rw-p 00000000 00:00 0 7f1abaf07000-7f1abb008000 r-xp 00000000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f1abb008000-7f1abb207000 ---p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f1abb207000-7f1abb208000 r--p 00100000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f1abb208000-7f1abb209000 rw-p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f1abb209000-7f1abb2f2000 r-xp 00000000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f1abb2f2000-7f1abb4f1000 ---p 000e9000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f1abb4f1000-7f1abb4f9000 r--p 000e8000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f1abb4f9000-7f1abb4fb000 rw-p 000f0000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f1abb4fb000-7f1abb510000 rw-p 00000000 00:00 0 7f1abb510000-7f1abb512000 r-xp 00000000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f1abb512000-7f1abb712000 ---p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f1abb712000-7f1abb713000 r--p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f1abb713000-7f1abb714000 rw-p 00003000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f1abb714000-7f1abb94a000 r-xp 00000000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f1abb94a000-7f1abbb4a000 ---p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f1abbb4a000-7f1abbb66000 r--p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f1abbb66000-7f1abbb73000 rw-p 00252000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f1abbb73000-7f1abbb77000 rw-p 00000000 00:00 0 7f1abbb77000-7f1abbbde000 r-xp 00000000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f1abbbde000-7f1abbdde000 ---p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f1abbdde000-7f1abbde2000 r--p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f1abbde2000-7f1abbde9000 rw-p 0006b000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f1abbde9000-7f1abbe18000 r-xp 00000000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f1abbe18000-7f1abc018000 ---p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f1abc018000-7f1abc019000 r--p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f1abc019000-7f1abc01a000 rw-p 00030000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f1abc01a000-7f1abc022000 r-xp 00000000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f1abc022000-7f1abc221000 ---p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f1abc221000-7f1abc222000 r--p 00007000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f1abc222000-7f1abc223000 rw-p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f1abc223000-7f1abc251000 rw-p 00000000 00:00 0 7f1abc251000-7f1abc266000 r-xp 00000000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f1abc266000-7f1abc465000 ---p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f1abc465000-7f1abc466000 r--p 00014000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f1abc466000-7f1abc467000 rw-p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f1abc467000-7f1abc468000 r-xp 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f1abc468000-7f1abc667000 ---p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f1abc667000-7f1abc668000 r--p 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f1abc668000-7f1abc669000 rw-p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f1abc669000-7f1abc68e000 r-xp 00000000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f1abc68e000-7f1abc88d000 ---p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f1abc88d000-7f1abc88e000 r--p 00024000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f1abc88e000-7f1abc88f000 rw-p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f1abc88f000-7f1abc8a6000 r-xp 00000000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f1abc8a6000-7f1abcaa5000 ---p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f1abcaa5000-7f1abcaa6000 r--p 00016000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f1abcaa6000-7f1abcaa7000 rw-p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f1abcaa7000-7f1abcaab000 rw-p 00000000 00:00 0 7f1abcaab000-7f1abcacd000 r-xp 00000000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f1abcad1000-7f1abcad2000 ---p 00000000 00:00 0 7f1abcad2000-7f1abcb1d000 rw-p 00000000 00:00 0 7f1abcb1d000-7f1abcb20000 rw-s 00000000 00:0b 50789371 /[aio] (deleted) 7f1abcb20000-7f1abcb23000 rw-s 00000000 00:0b 50789370 /[aio] (deleted) 7f1abcb23000-7f1abcb26000 rw-s 00000000 00:0b 50789369 /[aio] (deleted) 7f1abcb26000-7f1abcb29000 rw-s 00000000 00:0b 50789368 /[aio] (deleted) 7f1abcb29000-7f1abcb2c000 rw-s 00000000 00:0b 50789367 /[aio] (deleted) 7f1abcb2c000-7f1abcb2f000 rw-s 00000000 00:0b 50789366 /[aio] (deleted) 7f1abcb2f000-7f1abcb32000 rw-s 00000000 00:0b 50789365 /[aio] (deleted) 7f1abcb32000-7f1abcb35000 rw-s 00000000 00:0b 50789364 /[aio] (deleted) 7f1abcb35000-7f1abccc3000 rw-p 00000000 00:00 0 7f1abccc3000-7f1abccc4000 rw-s 00000000 00:0b 50789372 /[aio] (deleted) 7f1abccc4000-7f1abccc7000 rw-s 00000000 00:0b 50789363 /[aio] (deleted) 7f1abccc7000-7f1abccca000 rw-s 00000000 00:0b 50789362 /[aio] (deleted) 7f1abccca000-7f1abcccb000 rw-s 00000000 00:0b 50789361 /[aio] (deleted) 7f1abcccb000-7f1abcccc000 rw-p 00000000 00:00 0 7f1abcccc000-7f1abcccd000 r--p 00021000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f1abcccd000-7f1abccce000 rw-p 00022000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f1abccce000-7f1abcccf000 rw-p 00000000 00:00 0 7ffc442dd000-7ffc442fe000 rw-p 00000000 00:00 0 [stack] 7ffc44357000-7ffc44359000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Fatal signal 6 while backtracing 2020-02-01 0:44:04 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:44:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:44:04 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:44:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:44:04 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:44:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:44:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:44:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:44:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:44:04 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137219529 2020-02-01 0:44:04 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:44:04 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:44:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:44:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:44:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:44:04 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:44:04 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:44:04 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137219538; transaction id 5904369 2020-02-01 0:44:04 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:44:04 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:44:04 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:44:04 0 [Note] Starting crash recovery... 2020-02-01 0:44:04 0 [Note] Crash recovery finished. 2020-02-01 0:44:04 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:44:04 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:44:04 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:44:04 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000105' at position 47081 2020-02-01 0:44:04 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:44:04 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:44:04 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000105' at position 47081 2020-02-01 0:44:04 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:44:04 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:44:04 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:44:04 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:44:04 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:44:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:44:05 2020-02-01 0:44:16 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:44:16 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000106' at position 1464 2020-02-01 0:44:16 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:44:22 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000150, 25323) 2020-02-01 0:44:28 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:44:30 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:44:30 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:44:30 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:44:30 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:44:30 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7efc40000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7efc5d26fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d9606e7f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55d96017e3af] sigaction.c:0(__restore_rt)[0x7efcbe429630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55d9600ac151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55d9601ea545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55d95ffa0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55d95ff96d4b] /usr/sbin/mysqld(+0x6ca792)[0x55d95ffa1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55d95ffc96da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d95ffcb65d] /usr/sbin/mysqld(+0x65e3f3)[0x55d95ff353f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55d95ff34cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55d95ffc8b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55d95ffcb65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55d95ffcb761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55d95ffcc377] /usr/sbin/mysqld(+0x58fd84)[0x55d95fe66d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55d95ff73b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55d95ff8bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55d95ff8bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55d95ff8c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55d95ff8c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55d95ff7b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55d95ff7d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55d960058afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d960058bdd] pthread_create.c:0(start_thread)[0x7efcbe421ea5] /lib64/libc.so.6(clone+0x6d)[0x7efcbc7c28cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7efc4000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:44:37 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:44:37 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:44:37 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:44:37 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:44:37 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:44:37 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:44:37 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:44:37 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:44:37 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:44:37 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137222364 2020-02-01 0:44:37 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:44:37 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:44:37 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:44:37 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:44:37 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:44:37 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:44:37 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:44:37 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137222373; transaction id 5904380 2020-02-01 0:44:37 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:44:37 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:44:37 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:44:37 0 [Note] Starting crash recovery... 2020-02-01 0:44:37 0 [Note] Crash recovery finished. 2020-02-01 0:44:37 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:44:37 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:44:37 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:44:37 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000106' at position 1464 2020-02-01 0:44:37 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:44:37 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:44:37 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000106' at position 1464 2020-02-01 0:44:37 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:44:37 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:44:37 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:44:37 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:44:37 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:44:38 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:44:38 2020-02-01 0:44:56 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:44:56 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000107' at position 51844 2020-02-01 0:44:56 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:45:02 15 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000151, 2513) 2020-02-01 0:45:04 16 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:45:07 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:45:07 20 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:45:07 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:45:07 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:45:08 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fa4c8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fa4d8ffccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x564e83152f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x564e82be93af] sigaction.c:0(__restore_rt)[0x7fa53bdb4630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564e82b17151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564e82c55545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564e82a0bccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x564e82a01d4b] /usr/sbin/mysqld(+0x6ca792)[0x564e82a0c792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x564e82a346da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564e82a3665d] /usr/sbin/mysqld(+0x65e3f3)[0x564e829a03f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x564e8299fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x564e82a33b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564e82a3665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x564e82a36761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x564e82a37377] /usr/sbin/mysqld(+0x58fd84)[0x564e828d1d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x564e829deb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564e829f6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564e829f6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x564e829f75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564e829f7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x564e829e656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x564e829e807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x564e82ac3afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x564e82ac3bdd] pthread_create.c:0(start_thread)[0x7fa53bdacea5] /lib64/libc.so.6(clone+0x6d)[0x7fa53a14d8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fa4c800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:45:14 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:45:14 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:45:14 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:45:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:45:14 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:45:14 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:45:14 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:45:14 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:45:14 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:45:14 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137225344 2020-02-01 0:45:14 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:45:14 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:45:14 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:45:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:45:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:45:14 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:45:14 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:45:14 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137225353; transaction id 5904391 2020-02-01 0:45:14 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:45:14 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:45:14 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:45:14 0 [Note] Starting crash recovery... 2020-02-01 0:45:14 0 [Note] Crash recovery finished. 2020-02-01 0:45:14 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:45:15 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:45:15 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:45:15 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000107' at position 51844 2020-02-01 0:45:15 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:45:15 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:45:15 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:45:15 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:45:15 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:45:15 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:45:15 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:45:15 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:45:15 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:45:15 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:45:15 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:45:15 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:45:15 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000107' at position 51844 2020-02-01 0:45:15 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:45:15 2020-02-01 0:45:32 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:45:32 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000108' at position 1464 2020-02-01 0:45:32 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:45:38 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000152, 27664) 2020-02-01 0:45:39 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:45:40 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:45:40 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:45:40 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:45:40 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:45:40 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f46e8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f474468dcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e7ed4c5f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e7ecf5c3af] sigaction.c:0(__restore_rt)[0x7f475f803630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55e7ece8a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55e7ecfc8545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55e7ecd7eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55e7ecd74d4b] /usr/sbin/mysqld(+0x6ca792)[0x55e7ecd7f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55e7ecda76da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e7ecda965d] /usr/sbin/mysqld(+0x65e3f3)[0x55e7ecd133f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55e7ecd12cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55e7ecda6b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e7ecda965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55e7ecda9761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55e7ecdaa377] /usr/sbin/mysqld(+0x58fd84)[0x55e7ecc44d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55e7ecd51b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55e7ecd69b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55e7ecd69cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55e7ecd6a5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55e7ecd6a677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55e7ecd5956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55e7ecd5b07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55e7ece36afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e7ece36bdd] pthread_create.c:0(start_thread)[0x7f475f7fbea5] /lib64/libc.so.6(clone+0x6d)[0x7f475db9c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f46e800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:45:46 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:45:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:45:46 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:45:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:45:46 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:45:46 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:45:46 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:45:46 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:45:46 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:45:46 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137228217 2020-02-01 0:45:46 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:45:46 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:45:46 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:45:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:45:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:45:46 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:45:46 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:45:46 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137228226; transaction id 5904402 2020-02-01 0:45:46 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:45:46 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:45:46 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:45:46 0 [Note] Starting crash recovery... 2020-02-01 0:45:46 0 [Note] Crash recovery finished. 2020-02-01 0:45:46 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:45:46 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:45:46 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:45:46 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000108' at position 1464 2020-02-01 0:45:46 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:45:46 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:45:46 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000108' at position 1464 2020-02-01 0:45:46 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:45:46 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:45:46 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:45:46 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:45:46 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:45:47 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:45:47 2020-02-01 0:46:14 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:46:17 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:46:17 20 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:46:17 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:46:17 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:46:17 21 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:46:17 21 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:46:17 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=502 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f124c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f11e47f7cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c18efa1f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c18ea383af] sigaction.c:0(__restore_rt)[0x7f12552e9630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c18e966151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c18eaa4545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c18e85accf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c18e850d4b] /usr/sbin/mysqld(+0x6ca792)[0x55c18e85b792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c18e8836da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c18e88565d] /usr/sbin/mysqld(+0x65e3f3)[0x55c18e7ef3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c18e7eecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c18e882b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c18e88565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c18e885761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c18e886377] /usr/sbin/mysqld(+0x58fd84)[0x55c18e720d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c18e82db6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c18e845b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c18e845cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c18e8465e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c18e846677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c18e83556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c18e83707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c18e912afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c18e912bdd] pthread_create.c:0(start_thread)[0x7f12552e1ea5] /lib64/libc.so.6(clone+0x6d)[0x7f12536828cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f124c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:46:23 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:46:23 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:46:23 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:46:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:46:23 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:46:23 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:46:23 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:46:23 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:46:23 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:46:23 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137231913 2020-02-01 0:46:23 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:46:24 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:46:24 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:46:24 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:46:24 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:46:24 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:46:24 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:46:24 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137231922; transaction id 5904414 2020-02-01 0:46:24 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:46:24 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:46:24 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:46:24 0 [Note] Starting crash recovery... 2020-02-01 0:46:24 0 [Note] Crash recovery finished. 2020-02-01 0:46:24 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:46:24 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:46:24 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:46:24 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000109' at position 155635 2020-02-01 0:46:24 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:46:24 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:46:24 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000109' at position 155635 2020-02-01 0:46:24 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:46:24 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:46:24 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:46:24 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:46:24 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:46:25 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:46:25 2020-02-01 0:46:26 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:46:26 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000109' at position 272194 2020-02-01 0:46:26 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:46:32 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000153, 1464) 2020-02-01 0:46:49 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:46:52 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:46:52 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:46:53 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:46:53 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:46:53 15 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:46:53 15 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:46:53 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=502 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1235745 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f6f5c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f6fc1ca2cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ac4043cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ac3fed33af] sigaction.c:0(__restore_rt)[0x7f6fdce06630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55ac3fe01151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55ac3ff3f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55ac3fcf5ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55ac3fcebd4b] /usr/sbin/mysqld(+0x6ca792)[0x55ac3fcf6792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55ac3fd1e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ac3fd2065d] /usr/sbin/mysqld(+0x65e3f3)[0x55ac3fc8a3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55ac3fc89cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55ac3fd1db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ac3fd2065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55ac3fd20761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55ac3fd21377] /usr/sbin/mysqld(+0x58fd84)[0x55ac3fbbbd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55ac3fcc8b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55ac3fce0b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55ac3fce0cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55ac3fce15e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55ac3fce1677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55ac3fcd056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55ac3fcd207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55ac3fdadafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55ac3fdadbdd] pthread_create.c:0(start_thread)[0x7f6fdcdfeea5] /lib64/libc.so.6(clone+0x6d)[0x7f6fdb19f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f6f5c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:46:59 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:46:59 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:46:59 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:46:59 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:46:59 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:46:59 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:46:59 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:46:59 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:46:59 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:46:59 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137235533 2020-02-01 0:46:59 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:46:59 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:46:59 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:46:59 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:46:59 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:46:59 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:46:59 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:46:59 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137235542; transaction id 5904425 2020-02-01 0:46:59 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:46:59 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:46:59 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:46:59 0 [Note] Starting crash recovery... 2020-02-01 0:46:59 0 [Note] Crash recovery finished. 2020-02-01 0:46:59 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:46:59 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:46:59 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:46:59 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000109' at position 272194 2020-02-01 0:46:59 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:46:59 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:46:59 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000109' at position 272194 2020-02-01 0:46:59 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:46:59 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:46:59 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:46:59 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:46:59 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:47:00 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:47:00 2020-02-01 0:47:10 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:47:10 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000110' at position 85997 2020-02-01 0:47:10 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:47:16 17 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000155, 2513) 2020-02-01 0:47:25 19 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:47:28 25 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:47:28 25 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:47:28 27 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:47:28 27 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:47:28 27 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:47:28 27 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:47:28 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=202 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f8760000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f8764d05cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563b93349f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x563b92de03af] sigaction.c:0(__restore_rt)[0x7f8781f72630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x563b92d0e151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x563b92e4c545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x563b92c02ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x563b92bf8d4b] /usr/sbin/mysqld(+0x6ca792)[0x563b92c03792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x563b92c2b6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563b92c2d65d] /usr/sbin/mysqld(+0x65e3f3)[0x563b92b973f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x563b92b96cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x563b92c2ab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563b92c2d65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x563b92c2d761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x563b92c2e377] /usr/sbin/mysqld(+0x58fd84)[0x563b92ac8d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x563b92bd5b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x563b92bedb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x563b92bedcd2] /usr/sbin/mysqld(+0x6b55e7)[0x563b92bee5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x563b92bee677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x563b92bdd56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x563b92bdf07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x563b92cbaafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x563b92cbabdd] pthread_create.c:0(start_thread)[0x7f8781f6aea5] /lib64/libc.so.6(clone+0x6d)[0x7f878030b8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f876000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 28 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:47:35 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:47:35 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:47:35 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:47:35 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:47:35 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:47:35 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:47:35 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:47:35 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:47:35 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:47:35 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137238346 2020-02-01 0:47:35 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:47:35 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:47:35 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:47:35 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:47:35 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:47:35 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:47:35 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:47:35 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137238355; transaction id 5904436 2020-02-01 0:47:35 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:47:35 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:47:35 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:47:35 0 [Note] Starting crash recovery... 2020-02-01 0:47:35 0 [Note] Crash recovery finished. 2020-02-01 0:47:35 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:47:35 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:47:35 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:47:35 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000110' at position 85997 2020-02-01 0:47:35 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:47:35 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:47:35 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000110' at position 85997 2020-02-01 0:47:35 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:47:35 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:47:35 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:47:35 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:47:35 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:47:36 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:47:36 2020-02-01 0:47:47 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:47:47 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000111' at position 95183 2020-02-01 0:47:47 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:47:53 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000156, 26665) 2020-02-01 0:48:00 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:48:00 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:48:00 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:48:00 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:48:00 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:48:00 15 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:48:00 15 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:48:00 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fae74000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7faec8940cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a5af97cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a5af4133af] sigaction.c:0(__restore_rt)[0x7faee3aa2630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a5af341151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a5af47f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a5af235ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a5af22bd4b] /usr/sbin/mysqld(+0x6ca792)[0x55a5af236792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a5af25e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a5af26065d] /usr/sbin/mysqld(+0x65e3f3)[0x55a5af1ca3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a5af1c9cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a5af25db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a5af26065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a5af260761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a5af261377] /usr/sbin/mysqld(+0x58fd84)[0x55a5af0fbd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a5af208b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a5af220b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a5af220cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a5af2215e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a5af221677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a5af21056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a5af21207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a5af2edafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a5af2edbdd] pthread_create.c:0(start_thread)[0x7faee3a9aea5] /lib64/libc.so.6(clone+0x6d)[0x7faee1e3b8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fae7400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:48:06 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:48:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:48:06 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:48:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:48:06 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:48:06 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:48:06 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:48:06 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:48:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:48:06 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137241209 2020-02-01 0:48:06 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:48:06 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:48:06 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:48:06 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:48:06 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:48:06 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:48:06 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:48:07 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137241218; transaction id 5904447 2020-02-01 0:48:07 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:48:07 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:48:07 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:48:07 0 [Note] Starting crash recovery... 2020-02-01 0:48:07 0 [Note] Crash recovery finished. 2020-02-01 0:48:07 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:48:07 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:48:07 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:48:07 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000111' at position 95183 2020-02-01 0:48:07 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:48:07 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:48:07 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000111' at position 95183 2020-02-01 0:48:07 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:48:07 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:48:07 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:48:07 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:48:07 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:48:08 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:48:08 2020-02-01 0:48:24 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:48:24 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000112' at position 216559 2020-02-01 0:48:24 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:48:30 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000157, 1464) 2020-02-01 0:48:35 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:48:35 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:48:35 14 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:48:39 19 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 0:48:39 19 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 0:48:43 21 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:48:43 21 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:48:43 22 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:48:43 22 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:48:43 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=202 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f93bc000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f941c5d6cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563c5c260f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x563c5bcf73af] sigaction.c:0(__restore_rt)[0x7f94376f9630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x563c5bc25151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x563c5bd63545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x563c5bb19ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x563c5bb0fd4b] /usr/sbin/mysqld(+0x6ca792)[0x563c5bb1a792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x563c5bb426da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563c5bb4465d] /usr/sbin/mysqld(+0x65e3f3)[0x563c5baae3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x563c5baadcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x563c5bb41b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x563c5bb4465d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x563c5bb44761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x563c5bb45377] /usr/sbin/mysqld(+0x58fd84)[0x563c5b9dfd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x563c5baecb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x563c5bb04b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x563c5bb04cd2] /usr/sbin/mysqld(+0x6b55e7)[0x563c5bb055e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x563c5bb05677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x563c5baf456d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x563c5baf607c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x563c5bbd1afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x563c5bbd1bdd] pthread_create.c:0(start_thread)[0x7f94376f1ea5] /lib64/libc.so.6(clone+0x6d)[0x7f9435a928cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f93bc00feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 23 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:48:50 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:48:50 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:48:50 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:48:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:48:50 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:48:50 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:48:50 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:48:50 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:48:50 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:48:50 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137244056 2020-02-01 0:48:50 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:48:50 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:48:50 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:48:50 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:48:50 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:48:50 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:48:50 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:48:50 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137244065; transaction id 5904458 2020-02-01 0:48:50 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:48:50 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:48:50 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:48:50 0 [Note] Starting crash recovery... 2020-02-01 0:48:50 0 [Note] Crash recovery finished. 2020-02-01 0:48:50 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:48:50 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:48:50 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:48:50 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000112' at position 216559 2020-02-01 0:48:50 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:48:50 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:48:50 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000112' at position 216559 2020-02-01 0:48:50 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:48:50 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:48:50 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:48:50 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:48:50 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:48:50 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:48:50 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:48:50 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:48:50 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:48:50 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:48:51 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:48:51 2020-02-01 0:48:56 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:48:56 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000113' at position 1464 2020-02-01 0:48:56 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:49:02 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000158, 117329) 2020-02-01 0:49:15 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:49:16 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:49:16 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:49:16 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:49:16 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:49:16 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f7c58000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f7cbc3ebcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x559a793b6f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x559a78e4d3af] sigaction.c:0(__restore_rt)[0x7f7cd877c630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x559a78d7b151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x559a78eb9545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x559a78c6fccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x559a78c65d4b] /usr/sbin/mysqld(+0x6ca792)[0x559a78c70792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x559a78c986da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559a78c9a65d] /usr/sbin/mysqld(+0x65e3f3)[0x559a78c043f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x559a78c03cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x559a78c97b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559a78c9a65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x559a78c9a761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x559a78c9b377] /usr/sbin/mysqld(+0x58fd84)[0x559a78b35d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x559a78c42b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x559a78c5ab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x559a78c5acd2] /usr/sbin/mysqld(+0x6b55e7)[0x559a78c5b5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x559a78c5b677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x559a78c4a56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x559a78c4c07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x559a78d27afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x559a78d27bdd] pthread_create.c:0(start_thread)[0x7f7cd8774ea5] /lib64/libc.so.6(clone+0x6d)[0x7f7cd6b158cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f7c5800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:49:23 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:49:23 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:49:23 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:49:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:49:23 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:49:23 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:49:23 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:49:23 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:49:23 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:49:23 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137246919 2020-02-01 0:49:23 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:49:23 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:49:23 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:49:23 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:49:23 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:49:23 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:49:23 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:49:23 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137246928; transaction id 5904469 2020-02-01 0:49:23 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:49:23 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:49:23 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:49:23 0 [Note] Starting crash recovery... 2020-02-01 0:49:23 0 [Note] Crash recovery finished. 2020-02-01 0:49:23 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:49:23 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:49:23 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:49:23 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000113' at position 1464 2020-02-01 0:49:23 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:49:23 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:49:23 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000113' at position 1464 2020-02-01 0:49:23 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:49:23 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:49:23 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:49:23 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:49:23 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:49:24 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:49:24 2020-02-01 0:49:38 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:49:38 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000114' at position 51360 2020-02-01 0:49:38 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:49:44 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000159, 1464) 2020-02-01 0:49:51 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:49:51 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:49:51 16 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:49:51 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:49:51 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:49:51 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=202 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f7410000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f742d87bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a0612faf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a060d913af] sigaction.c:0(__restore_rt)[0x7f748ea93630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a060cbf151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a060dfd545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a060bb3ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a060ba9d4b] /usr/sbin/mysqld(+0x6ca792)[0x55a060bb4792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a060bdc6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a060bde65d] /usr/sbin/mysqld(+0x65e3f3)[0x55a060b483f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a060b47cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a060bdbb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a060bde65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a060bde761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a060bdf377] /usr/sbin/mysqld(+0x58fd84)[0x55a060a79d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a060b86b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a060b9eb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a060b9ecd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a060b9f5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a060b9f677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a060b8e56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a060b9007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a060c6bafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a060c6bbdd] pthread_create.c:0(start_thread)[0x7f748ea8bea5] /lib64/libc.so.6(clone+0x6d)[0x7f748ce2c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f741000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:49:58 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:49:58 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:49:58 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:49:58 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:49:58 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:49:58 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:49:58 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:49:58 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:49:58 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:49:58 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137249766 2020-02-01 0:49:58 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:49:58 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:49:58 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:49:58 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:49:58 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:49:58 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:49:58 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:49:58 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137249775; transaction id 5904480 2020-02-01 0:49:58 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:49:58 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:49:58 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:49:58 0 [Note] Starting crash recovery... 2020-02-01 0:49:58 0 [Note] Crash recovery finished. 2020-02-01 0:49:58 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:49:58 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:49:58 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:49:58 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000114' at position 51360 2020-02-01 0:49:58 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:49:58 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:49:58 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:49:58 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:49:58 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:49:58 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:49:58 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:49:58 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000114' at position 51360 2020-02-01 0:49:59 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:49:59 2020-02-01 0:50:18 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:50:18 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000115' at position 241871 2020-02-01 0:50:18 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:50:23 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000160, 4411) 2020-02-01 0:50:26 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:50:26 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:50:26 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:50:26 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:50:26 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:50:27 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc9100009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc9703b8cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5596a072ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5596a01c53af] sigaction.c:0(__restore_rt)[0x7fc98d53d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5596a00f3151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5596a0231545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55969ffe7ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55969ffddd4b] /usr/sbin/mysqld(+0x6ca792)[0x55969ffe8792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5596a00106da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5596a001265d] /usr/sbin/mysqld(+0x65e3f3)[0x55969ff7c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55969ff7bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5596a000fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5596a001265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5596a0012761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5596a0013377] /usr/sbin/mysqld(+0x58fd84)[0x55969feadd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55969ffbab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55969ffd2b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55969ffd2cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55969ffd35e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55969ffd3677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55969ffc256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55969ffc407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5596a009fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5596a009fbdd] pthread_create.c:0(start_thread)[0x7fc98d535ea5] /lib64/libc.so.6(clone+0x6d)[0x7fc98b8d68cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc91000fe00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:50:33 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:50:33 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:50:33 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:50:33 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:50:33 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:50:33 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:50:33 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:50:33 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:50:33 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:50:33 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137252629 2020-02-01 0:50:33 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:50:33 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:50:33 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:50:33 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:50:33 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:50:33 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:50:33 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:50:33 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137252638; transaction id 5904491 2020-02-01 0:50:33 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:50:33 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:50:33 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:50:33 0 [Note] Starting crash recovery... 2020-02-01 0:50:33 0 [Note] Crash recovery finished. 2020-02-01 0:50:33 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:50:33 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:50:33 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:50:33 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000115' at position 241871 2020-02-01 0:50:33 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:50:33 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:50:33 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:50:33 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:50:33 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:50:33 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:50:33 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:50:33 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000115' at position 241871 2020-02-01 0:50:34 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:50:34 2020-02-01 0:51:01 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:51:02 15 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:51:02 15 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:51:05 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:51:05 20 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:51:05 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:51:05 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:51:05 21 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:51:05 21 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:51:05 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f01e0006bf8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f024898bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x556069427f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x556068ebe3af] sigaction.c:0(__restore_rt)[0x7f0263a1d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x556068dec151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x556068f2a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x556068ce0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x556068cd6d4b] /usr/sbin/mysqld(+0x6ca792)[0x556068ce1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x556068d096da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556068d0b65d] /usr/sbin/mysqld(+0x65e3f3)[0x556068c753f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x556068c74cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x556068d08b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556068d0b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x556068d0b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x556068d0c377] /usr/sbin/mysqld(+0x58fd84)[0x556068ba6d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x556068cb3b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x556068ccbb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x556068ccbcd2] /usr/sbin/mysqld(+0x6b55e7)[0x556068ccc5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x556068ccc677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x556068cbb56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x556068cbd07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x556068d98afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x556068d98bdd] pthread_create.c:0(start_thread)[0x7f0263a15ea5] /lib64/libc.so.6(clone+0x6d)[0x7f0261db68cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f01e0011000): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:51:11 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:51:11 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:51:11 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:51:11 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:51:11 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:51:11 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:51:11 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:51:11 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:51:11 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:51:12 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137255492 2020-02-01 0:51:12 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:51:13 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:51:13 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:51:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:51:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:51:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:51:13 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:51:13 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137255501; transaction id 5904502 2020-02-01 0:51:13 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:51:13 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:51:13 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:51:13 0 [Note] Starting crash recovery... 2020-02-01 0:51:13 0 [Note] Crash recovery finished. 2020-02-01 0:51:13 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:51:13 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:51:13 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:51:13 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000116' at position 4034 2020-02-01 0:51:13 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:51:13 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:51:13 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000116' at position 4034 2020-02-01 0:51:13 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:51:13 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:51:13 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:51:13 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:51:13 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:51:13 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:51:13 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:51:13 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:51:13 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:51:13 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:51:14 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:51:14 2020-02-01 0:51:27 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000161, 1464) 2020-02-01 0:51:32 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:51:32 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000116' at position 36730 2020-02-01 0:51:32 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:51:36 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:51:38 14 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000163, 346) 2020-02-01 0:51:39 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:51:39 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:51:39 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:51:39 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:51:39 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f37f8006bf8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f3850092cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ef6df34f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ef6d9cb3af] sigaction.c:0(__restore_rt)[0x7f386ade9630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55ef6d8f9151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55ef6da37545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55ef6d7edccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55ef6d7e3d4b] /usr/sbin/mysqld(+0x6ca792)[0x55ef6d7ee792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55ef6d8166da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ef6d81865d] /usr/sbin/mysqld(+0x65e3f3)[0x55ef6d7823f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55ef6d781cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55ef6d815b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ef6d81865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55ef6d818761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55ef6d819377] /usr/sbin/mysqld(+0x58fd84)[0x55ef6d6b3d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55ef6d7c0b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55ef6d7d8b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55ef6d7d8cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55ef6d7d95e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55ef6d7d9677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55ef6d7c856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55ef6d7ca07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55ef6d8a5afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55ef6d8a5bdd] pthread_create.c:0(start_thread)[0x7f386ade1ea5] /lib64/libc.so.6(clone+0x6d)[0x7f38691828cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f37f8010ea0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:51:46 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:51:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:51:46 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:51:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:51:46 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:51:46 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:51:46 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:51:46 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:51:46 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:51:46 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137258339 2020-02-01 0:51:46 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:51:46 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:51:46 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:51:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:51:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:51:46 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:51:46 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:51:46 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137258348; transaction id 5904513 2020-02-01 0:51:46 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:51:46 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:51:46 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:51:46 0 [Note] Starting crash recovery... 2020-02-01 0:51:46 0 [Note] Crash recovery finished. 2020-02-01 0:51:46 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:51:46 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:51:46 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:51:46 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000116' at position 36730 2020-02-01 0:51:46 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:51:46 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:51:46 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000116' at position 36730 2020-02-01 0:51:46 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:51:46 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:51:46 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:51:46 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:51:46 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:51:47 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:51:47 2020-02-01 0:52:12 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:52:12 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:52:12 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000117' at position 25313 2020-02-01 0:52:12 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 200201 0:52:12 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=1 max_threads=202 thread_count=9 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f3fcc006bb8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f3fec575cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x557e5f488f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x557e5ef1f3af] sigaction.c:0(__restore_rt)[0x7f404e84b630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x557e5ee4d151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x557e5ef8b545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x557e5ed41ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x557e5ed37d4b] /usr/sbin/mysqld(+0x6ca792)[0x557e5ed42792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x557e5ed6a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557e5ed6c65d] /usr/sbin/mysqld(+0x65e3f3)[0x557e5ecd63f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x557e5ecd5cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x557e5ed69b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557e5ed6c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x557e5ed6c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x557e5ed6d377] /usr/sbin/mysqld(+0x58fd84)[0x557e5ec07d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x557e5ed14b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x557e5ed2cb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x557e5ed2ccd2] /usr/sbin/mysqld(+0x6b55e7)[0x557e5ed2d5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x557e5ed2d677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x557e5ed1c56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x557e5ed1e07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x557e5edf9afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x557e5edf9bdd] pthread_create.c:0(start_thread)[0x7f404e843ea5] /lib64/libc.so.6(clone+0x6d)[0x7f404cbe48cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f3fcc010fc0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 13 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:52:18 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:52:18 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:52:18 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:52:18 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:52:18 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:52:18 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:52:18 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:52:18 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:52:18 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:52:18 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137261169 2020-02-01 0:52:18 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:52:18 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:52:18 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:52:18 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:52:18 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:52:18 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:52:18 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:52:18 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137261178; transaction id 5904524 2020-02-01 0:52:18 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:52:18 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:52:18 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:52:18 0 [Note] Starting crash recovery... 2020-02-01 0:52:18 0 [Note] Crash recovery finished. 2020-02-01 0:52:18 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:52:18 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:52:18 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:52:18 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000117' at position 25313 2020-02-01 0:52:18 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:52:18 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:52:18 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000117' at position 25313 2020-02-01 0:52:18 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:52:18 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:52:18 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:52:18 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:52:18 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:52:19 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:52:19 2020-02-01 0:52:44 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown 2020-02-01 0:52:44 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-02-01 0:52:44 0 [Note] InnoDB: FTS optimize thread exiting. 2020-02-01 0:52:44 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000118', position 346 2020-02-01 0:52:44 0 [Note] InnoDB: Starting shutdown... 2020-02-01 0:52:44 0 [Note] InnoDB: Dumping buffer pool(s) to /data/mysql/ib_buffer_pool 2020-02-01 0:52:44 0 [Note] InnoDB: Instance 0, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:52:44 0 [Note] InnoDB: Instance 1, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:52:44 0 [Note] InnoDB: Instance 2, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:52:44 0 [Note] InnoDB: Instance 3, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:52:44 0 [Note] InnoDB: Instance 4, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:52:44 0 [Note] InnoDB: Instance 5, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:52:44 0 [Note] InnoDB: Instance 6, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:52:44 0 [Note] InnoDB: Instance 7, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:52:44 0 [Note] InnoDB: Buffer pool(s) dump completed at 200201 0:52:44 2020-02-01 0:52:46 0 [Note] InnoDB: Shutdown completed; log sequence number 5137263992; transaction id 5904533 2020-02-01 0:52:46 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:52:46 0 [Note] /usr/sbin/mysqld: Shutdown complete 2020-02-01 0:52:46 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:52:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:52:46 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:52:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:52:46 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:52:46 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:52:46 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:52:46 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:52:46 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:52:46 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:52:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:52:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:52:46 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:52:46 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137263992; transaction id 5904535 2020-02-01 0:52:46 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:52:46 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:52:46 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:52:46 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:52:46 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:52:46 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000118' at position 346 2020-02-01 0:52:46 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:52:46 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:52:46 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:52:46 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:52:46 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:52:46 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:52:46 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:52:46 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000118' at position 346 2020-02-01 0:52:47 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:52:47 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:52:47 2020-02-01 0:52:47 13 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 0:52:47 13 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 0:52:52 16 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:52:52 16 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:52:55 24 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:52:55 24 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:52:55 25 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:52:55 25 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:52:55 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=7 max_threads=202 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f6ba8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f6c08a8dcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ffcb27cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ffcad133af] sigaction.c:0(__restore_rt)[0x7f6c2cce5630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55ffcac41151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55ffcad7f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55ffcab35ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55ffcab2bd4b] /usr/sbin/mysqld(+0x6ca792)[0x55ffcab36792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55ffcab5e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ffcab6065d] /usr/sbin/mysqld(+0x65e3f3)[0x55ffcaaca3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55ffcaac9cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55ffcab5db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ffcab6065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55ffcab60761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55ffcab61377] /usr/sbin/mysqld(+0x58fd84)[0x55ffca9fbd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55ffcab08b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55ffcab20b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55ffcab20cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55ffcab215e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55ffcab21677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55ffcab1056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55ffcab1207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55ffcabedafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55ffcabedbdd] pthread_create.c:0(start_thread)[0x7f6c2ccddea5] /lib64/libc.so.6(clone+0x6d)[0x7f6c2b07e8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f6ba800feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 26 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:53:02 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:53:02 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:53:02 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:53:02 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:53:02 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:53:02 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:53:02 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:53:02 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:53:02 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:53:02 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137266872 2020-02-01 0:53:02 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:53:02 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:53:02 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:53:02 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:53:02 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:53:02 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:53:02 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:53:02 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137266881; transaction id 5904547 2020-02-01 0:53:02 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:53:02 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:53:02 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:53:02 0 [Note] Starting crash recovery... 2020-02-01 0:53:02 0 [Note] Crash recovery finished. 2020-02-01 0:53:02 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:53:02 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:53:02 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:53:02 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000118' at position 346 2020-02-01 0:53:02 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:53:02 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:53:02 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:53:02 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:53:02 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:53:02 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000118' at position 346 2020-02-01 0:53:02 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:53:02 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:53:02 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:53:02 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:53:02 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:53:02 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:53:02 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:53:03 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:53:03 2020-02-01 0:53:06 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:53:06 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000118' at position 70775 2020-02-01 0:53:06 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:53:11 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000163, 1464) 2020-02-01 0:53:25 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown 2020-02-01 0:53:25 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-02-01 0:53:25 0 [Note] InnoDB: FTS optimize thread exiting. 2020-02-01 0:53:25 10 [Note] Slave I/O thread killed during or after a reconnect done to recover from failed read 2020-02-01 0:53:25 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000118', position 70775 2020-02-01 0:53:25 0 [Note] InnoDB: Starting shutdown... 2020-02-01 0:53:25 0 [Note] InnoDB: Dumping buffer pool(s) to /data/mysql/ib_buffer_pool 2020-02-01 0:53:25 0 [Note] InnoDB: Instance 0, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:53:25 0 [Note] InnoDB: Instance 1, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:53:25 0 [Note] InnoDB: Instance 2, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:53:25 0 [Note] InnoDB: Instance 3, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:53:25 0 [Note] InnoDB: Instance 4, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:53:25 0 [Note] InnoDB: Instance 5, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:53:25 0 [Note] InnoDB: Instance 6, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:53:25 0 [Note] InnoDB: Instance 7, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 0:53:25 0 [Note] InnoDB: Buffer pool(s) dump completed at 200201 0:53:25 2020-02-01 0:53:26 0 [Note] InnoDB: Shutdown completed; log sequence number 5137270535; transaction id 5904556 2020-02-01 0:53:26 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:53:26 0 [Note] /usr/sbin/mysqld: Shutdown complete 2020-02-01 0:53:26 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:53:26 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:53:26 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:53:26 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:53:26 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:53:26 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:53:26 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:53:26 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:53:26 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:53:26 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:53:26 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:53:26 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:53:26 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:53:26 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137270535; transaction id 5904558 2020-02-01 0:53:26 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:53:27 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:53:27 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:53:27 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:53:27 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:53:27 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000118' at position 70775 2020-02-01 0:53:27 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:53:27 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:53:27 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000118' at position 70775 2020-02-01 0:53:27 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:53:27 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:53:27 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:53:27 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:53:27 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:53:27 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:53:28 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:53:28 2020-02-01 0:53:28 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:53:28 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:53:28 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:53:28 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:53:28 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=3 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fa910000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fa990159cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x557de0a2bf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x557de04c23af] sigaction.c:0(__restore_rt)[0x7fa99e1eb630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x557de03f0151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x557de052e545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x557de02e4ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x557de02dad4b] /usr/sbin/mysqld(+0x6ca792)[0x557de02e5792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x557de030d6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557de030f65d] /usr/sbin/mysqld(+0x65e3f3)[0x557de02793f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x557de0278cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x557de030cb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557de030f65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x557de030f761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x557de0310377] /usr/sbin/mysqld(+0x58fd84)[0x557de01aad84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x557de02b7b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x557de02cfb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x557de02cfcd2] /usr/sbin/mysqld(+0x6b55e7)[0x557de02d05e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x557de02d0677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x557de02bf56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x557de02c107c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x557de039cafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x557de039cbdd] pthread_create.c:0(start_thread)[0x7fa99e1e3ea5] /lib64/libc.so.6(clone+0x6d)[0x7fa99c5848cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fa91000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:53:35 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:53:35 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:53:35 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:53:35 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:53:35 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:53:35 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:53:35 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:53:35 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:53:35 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:53:35 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137273138 2020-02-01 0:53:35 0 [Note] InnoDB: Starting final batch to recover 3 pages from redo log. 2020-02-01 0:53:35 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:53:35 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:53:35 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:53:35 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:53:35 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:53:35 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:53:36 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:53:36 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137273255; transaction id 5904569 2020-02-01 0:53:36 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:53:36 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:53:36 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:53:36 0 [Note] Starting crash recovery... 2020-02-01 0:53:36 0 [Note] Crash recovery finished. 2020-02-01 0:53:36 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:53:36 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:53:36 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:53:36 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000119' at position 346 2020-02-01 0:53:36 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000119' at position 346 2020-02-01 0:53:36 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:53:36 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:53:36 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:53:36 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:53:36 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:53:36 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:53:36 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:53:37 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:53:37 2020-02-01 0:53:38 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:53:38 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000119' at position 1464 2020-02-01 0:53:38 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:53:44 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000167, 346) 2020-02-01 0:54:02 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:54:05 14 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-02-01 0:54:05 14 [Warning] Checking table: './openrecords/wp_options' 2020-02-01 0:54:05 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:54:05 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:54:05 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:54:05 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:54:06 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=5 max_threads=202 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fae7c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7faee409ccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55658c08ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55658bb253af] sigaction.c:0(__restore_rt)[0x7faef1dc4630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55658ba53151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55658bb91545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55658b947ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55658b93dd4b] /usr/sbin/mysqld(+0x6ca792)[0x55658b948792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55658b9706da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55658b97265d] /usr/sbin/mysqld(+0x65e3f3)[0x55658b8dc3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55658b8dbcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55658b96fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55658b97265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55658b972761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55658b973377] /usr/sbin/mysqld(+0x58fd84)[0x55658b80dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55658b91ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55658b932b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55658b932cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55658b9335e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55658b933677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55658b92256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55658b92407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55658b9ffafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55658b9ffbdd] pthread_create.c:0(start_thread)[0x7faef1dbcea5] /lib64/libc.so.6(clone+0x6d)[0x7faef015d8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fae7c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:54:12 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:54:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:54:12 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:54:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:54:12 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:54:12 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:54:12 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:54:12 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:54:12 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:54:12 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137276224 2020-02-01 0:54:12 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:54:12 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:54:12 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:54:12 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:54:12 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:54:12 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:54:12 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:54:12 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137276233; transaction id 5904580 2020-02-01 0:54:12 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:54:12 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:54:12 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:54:12 0 [Note] Starting crash recovery... 2020-02-01 0:54:12 0 [Note] Crash recovery finished. 2020-02-01 0:54:12 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:54:12 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:54:12 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:54:12 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000119' at position 1464 2020-02-01 0:54:12 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:54:12 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:54:12 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000119' at position 1464 2020-02-01 0:54:12 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:54:12 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:54:12 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:54:12 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:54:12 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:54:13 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:54:13 2020-02-01 0:54:19 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:54:19 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000120' at position 2513 2020-02-01 0:54:19 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:54:24 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000169, 1464) 2020-02-01 0:54:32 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000170, 346) 2020-02-01 0:54:37 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:54:38 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:54:38 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:54:38 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:54:38 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:54:38 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f91ac000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f9214ad9cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5624df497f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5624def2e3af] sigaction.c:0(__restore_rt)[0x7f9237b5d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5624dee5c151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5624def9a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5624ded50ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5624ded46d4b] /usr/sbin/mysqld(+0x6ca792)[0x5624ded51792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5624ded796da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5624ded7b65d] /usr/sbin/mysqld(+0x65e3f3)[0x5624dece53f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5624dece4cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5624ded78b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5624ded7b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5624ded7b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5624ded7c377] /usr/sbin/mysqld(+0x58fd84)[0x5624dec16d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5624ded23b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5624ded3bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5624ded3bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x5624ded3c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5624ded3c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5624ded2b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5624ded2d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5624dee08afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5624dee08bdd] pthread_create.c:0(start_thread)[0x7f9237b55ea5] /lib64/libc.so.6(clone+0x6d)[0x7f9235ef68cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f91ac00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:54:44 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:54:44 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:54:44 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:54:44 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:54:44 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:54:44 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:54:44 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:54:44 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:54:44 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:54:44 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137279087 2020-02-01 0:54:44 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:54:44 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:54:44 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:54:44 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:54:44 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:54:44 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:54:44 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:54:44 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137279096; transaction id 5904591 2020-02-01 0:54:44 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:54:44 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:54:44 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:54:44 0 [Note] Starting crash recovery... 2020-02-01 0:54:44 0 [Note] Crash recovery finished. 2020-02-01 0:54:44 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:54:44 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:54:44 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:54:44 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000120' at position 2513 2020-02-01 0:54:44 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:54:44 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:54:44 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000120' at position 2513 2020-02-01 0:54:44 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:54:44 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:54:44 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:54:44 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:54:44 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:54:45 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:54:45 2020-02-01 0:54:58 14 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-02-01 0:54:59 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:54:59 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000122' at position 234859 2020-02-01 0:54:59 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:55:05 15 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-02-01 0:55:05 16 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000170, 1464) 2020-02-01 0:55:13 18 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:55:14 20 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 0:55:14 20 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 0:55:18 25 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 0:55:18 25 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 0:55:18 28 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:55:18 28 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:55:18 29 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:55:18 29 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:55:19 29 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:55:19 29 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:55:19 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=6 max_threads=202 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f6fd0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f70340f7cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561107fa8f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x561107a3f3af] sigaction.c:0(__restore_rt)[0x7f705691a630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56110796d151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x561107aab545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x561107861ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x561107857d4b] /usr/sbin/mysqld(+0x6ca792)[0x561107862792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56110788a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56110788c65d] /usr/sbin/mysqld(+0x65e3f3)[0x5611077f63f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5611077f5cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x561107889b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56110788c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56110788c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56110788d377] /usr/sbin/mysqld(+0x58fd84)[0x561107727d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x561107834b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56110784cb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56110784ccd2] /usr/sbin/mysqld(+0x6b55e7)[0x56110784d5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56110784d677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56110783c56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56110783e07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x561107919afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x561107919bdd] pthread_create.c:0(start_thread)[0x7f7056912ea5] /lib64/libc.so.6(clone+0x6d)[0x7f7054cb38cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f6fd000fe80): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 30 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:55:25 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:55:25 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:55:25 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:55:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:55:25 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:55:25 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:55:25 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:55:25 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:55:25 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:55:25 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137281934 2020-02-01 0:55:25 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:55:25 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:55:25 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:55:25 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:55:25 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:55:25 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:55:25 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:55:25 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137281943; transaction id 5904602 2020-02-01 0:55:25 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:55:25 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:55:25 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:55:25 0 [Note] Starting crash recovery... 2020-02-01 0:55:25 0 [Note] Crash recovery finished. 2020-02-01 0:55:25 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:55:25 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:55:25 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:55:25 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000122' at position 234859 2020-02-01 0:55:25 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:55:25 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:55:25 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000122' at position 234859 2020-02-01 0:55:25 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:55:25 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:55:25 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:55:25 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:55:25 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:55:27 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:55:27 2020-02-01 0:55:34 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:55:34 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000123' at position 1464 2020-02-01 0:55:34 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:55:40 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000171, 51585) 2020-02-01 0:55:53 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:55:54 14 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:55:54 14 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:55:56 19 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:55:57 19 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:55:57 20 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:55:57 20 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:55:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=7 max_threads=202 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f5964000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f59680accf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55fdef477f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55fdeef0e3af] sigaction.c:0(__restore_rt)[0x7f598a9d6630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55fdeee3c151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55fdeef7a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55fdeed30ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55fdeed26d4b] /usr/sbin/mysqld(+0x6ca792)[0x55fdeed31792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55fdeed596da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55fdeed5b65d] /usr/sbin/mysqld(+0x65e3f3)[0x55fdeecc53f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55fdeecc4cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55fdeed58b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55fdeed5b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55fdeed5b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55fdeed5c377] /usr/sbin/mysqld(+0x58fd84)[0x55fdeebf6d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55fdeed03b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55fdeed1bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55fdeed1bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55fdeed1c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55fdeed1c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55fdeed0b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55fdeed0d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55fdeede8afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55fdeede8bdd] pthread_create.c:0(start_thread)[0x7f598a9ceea5] /lib64/libc.so.6(clone+0x6d)[0x7f5988d6f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f596400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 21 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:56:03 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:56:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:56:03 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:56:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:56:03 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:56:03 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:56:03 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:56:03 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:56:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:56:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137284797 2020-02-01 0:56:03 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:56:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:56:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:56:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:56:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:56:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:56:03 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:56:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137284806; transaction id 5904613 2020-02-01 0:56:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:56:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:56:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:56:03 0 [Note] Starting crash recovery... 2020-02-01 0:56:03 0 [Note] Crash recovery finished. 2020-02-01 0:56:03 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:56:03 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:56:03 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:56:03 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000123' at position 1464 2020-02-01 0:56:03 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:56:03 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:56:03 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:56:03 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000123' at position 1464 2020-02-01 0:56:03 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:56:03 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:56:03 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:56:03 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:56:03 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:56:03 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:56:03 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:56:03 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:56:03 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:56:04 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:56:04 2020-02-01 0:56:12 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:56:12 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000124' at position 1464 2020-02-01 0:56:12 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:56:18 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000172, 100468) 2020-02-01 0:56:28 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:56:28 14 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 0:56:28 14 [Warning] Checking table: './wpms/wp_options' 2020-02-01 0:56:33 23 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:56:33 23 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:56:33 24 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:56:33 24 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:56:33 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=6 max_threads=202 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f4fec000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f504c4fecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x564eff64ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x564eff0e63af] sigaction.c:0(__restore_rt)[0x7f5071656630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564eff014151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564eff152545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564efef08ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x564efeefed4b] /usr/sbin/mysqld(+0x6ca792)[0x564efef09792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x564efef316da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564efef3365d] /usr/sbin/mysqld(+0x65e3f3)[0x564efee9d3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x564efee9ccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x564efef30b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564efef3365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x564efef33761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x564efef34377] /usr/sbin/mysqld(+0x58fd84)[0x564efedced84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x564efeedbb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564efeef3b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564efeef3cd2] /usr/sbin/mysqld(+0x6b55e7)[0x564efeef45e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564efeef4677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x564efeee356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x564efeee507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x564efefc0afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x564efefc0bdd] pthread_create.c:0(start_thread)[0x7f507164eea5] /lib64/libc.so.6(clone+0x6d)[0x7f506f9ef8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f4fec00fe80): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 25 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:56:39 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:56:39 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:56:39 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:56:39 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:56:39 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:56:39 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:56:39 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:56:39 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:56:39 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:56:39 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137287803 2020-02-01 0:56:39 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:56:40 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:56:40 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:56:40 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:56:40 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:56:40 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:56:40 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:56:40 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137287812; transaction id 5904624 2020-02-01 0:56:40 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:56:40 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:56:40 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:56:40 0 [Note] Starting crash recovery... 2020-02-01 0:56:40 0 [Note] Crash recovery finished. 2020-02-01 0:56:40 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:56:40 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:56:40 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:56:40 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000124' at position 1464 2020-02-01 0:56:40 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:56:40 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:56:40 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000124' at position 1464 2020-02-01 0:56:40 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:56:40 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:56:40 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:56:40 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:56:40 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:56:40 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:56:40 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:56:40 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:56:40 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:56:40 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:56:41 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:56:41 2020-02-01 0:56:48 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:56:48 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000125' at position 3579 2020-02-01 0:56:48 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:56:54 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000173, 54480) 2020-02-01 0:57:08 19 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:57:09 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:57:09 20 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:57:09 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:57:09 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:57:09 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f4c90000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4cec0c3cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5631fa837f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5631fa2ce3af] sigaction.c:0(__restore_rt)[0x7f4d0eb00630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5631fa1fc151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5631fa33a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5631fa0f0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5631fa0e6d4b] /usr/sbin/mysqld(+0x6ca792)[0x5631fa0f1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5631fa1196da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5631fa11b65d] /usr/sbin/mysqld(+0x65e3f3)[0x5631fa0853f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5631fa084cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5631fa118b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5631fa11b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5631fa11b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5631fa11c377] /usr/sbin/mysqld(+0x58fd84)[0x5631f9fb6d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5631fa0c3b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5631fa0dbb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5631fa0dbcd2] /usr/sbin/mysqld(+0x6b55e7)[0x5631fa0dc5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5631fa0dc677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5631fa0cb56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5631fa0cd07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5631fa1a8afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5631fa1a8bdd] pthread_create.c:0(start_thread)[0x7f4d0eaf8ea5] /lib64/libc.so.6(clone+0x6d)[0x7f4d0ce998cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f4c9000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:57:15 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:57:15 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:57:15 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:57:15 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:57:15 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:57:15 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:57:15 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:57:15 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:57:15 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:57:15 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137291457 2020-02-01 0:57:15 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:57:15 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:57:15 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:57:15 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:57:15 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:57:15 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:57:15 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137291466; transaction id 5904635 2020-02-01 0:57:15 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:57:15 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:57:15 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:57:15 0 [Note] Starting crash recovery... 2020-02-01 0:57:15 0 [Note] Crash recovery finished. 2020-02-01 0:57:15 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:57:15 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:57:15 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:57:15 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000125' at position 3579 2020-02-01 0:57:15 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:57:15 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:57:15 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000125' at position 3579 2020-02-01 0:57:15 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:57:15 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:57:15 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:57:15 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:57:15 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:57:16 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:57:16 2020-02-01 0:57:19 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 0:57:19 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000126' at position 1464 2020-02-01 0:57:19 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:57:25 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000174, 1464) 2020-02-01 0:57:44 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:57:44 15 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 0:57:44 15 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 0:57:47 24 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:57:47 24 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:57:47 25 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:57:47 25 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:57:47 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=7 max_threads=202 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f1e1c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f1e20fa9cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55831a8cff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55831a3663af] sigaction.c:0(__restore_rt)[0x7f1e4413b630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55831a294151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55831a3d2545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55831a188ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55831a17ed4b] /usr/sbin/mysqld(+0x6ca792)[0x55831a189792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55831a1b16da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55831a1b365d] /usr/sbin/mysqld(+0x65e3f3)[0x55831a11d3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55831a11ccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55831a1b0b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55831a1b365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55831a1b3761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55831a1b4377] /usr/sbin/mysqld(+0x58fd84)[0x55831a04ed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55831a15bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55831a173b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55831a173cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55831a1745e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55831a174677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55831a16356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55831a16507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55831a240afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55831a240bdd] pthread_create.c:0(start_thread)[0x7f1e44133ea5] /lib64/libc.so.6(clone+0x6d)[0x7f1e424d48cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f1e1c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 26 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:57:53 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:57:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:57:53 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:57:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:57:53 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:57:53 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:57:53 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:57:53 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:57:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:57:53 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137294303 2020-02-01 0:57:53 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:57:53 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:57:53 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:57:53 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:57:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:57:53 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:57:53 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:57:54 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137294312; transaction id 5904646 2020-02-01 0:57:54 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:57:54 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:57:54 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:57:54 0 [Note] Starting crash recovery... 2020-02-01 0:57:54 0 [Note] Crash recovery finished. 2020-02-01 0:57:54 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:57:54 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:57:54 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:57:54 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000126' at position 1464 2020-02-01 0:57:54 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:57:54 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:57:54 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000126' at position 1464 2020-02-01 0:57:54 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 0:57:54 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 0:57:54 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 0:57:54 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 0:57:54 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:57:54 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:57:54 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 0:57:54 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:57:54 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:57:54 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:57:55 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:57:55 2020-02-01 0:58:19 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:58:41 23 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:58:41 23 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:58:41 24 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:58:41 24 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 0:58:41 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=3 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc43c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc4b010dcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56554179ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5655412353af] sigaction.c:0(__restore_rt)[0x7fc4c0bc5630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x565541163151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5655412a1545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x565541057ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56554104dd4b] /usr/sbin/mysqld(+0x6ca792)[0x565541058792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5655410806da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56554108265d] /usr/sbin/mysqld(+0x65e3f3)[0x565540fec3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x565540febcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56554107fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56554108265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x565541082761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x565541083377] /usr/sbin/mysqld(+0x58fd84)[0x565540f1dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56554102ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x565541042b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x565541042cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5655410435e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x565541043677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56554103256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56554103407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56554110fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56554110fbdd] pthread_create.c:0(start_thread)[0x7fc4c0bbdea5] /lib64/libc.so.6(clone+0x6d)[0x7fc4bef5e8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc43c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 25 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:58:47 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:58:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:58:47 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:58:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:58:47 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:58:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:58:47 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:58:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:58:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:58:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137297166 2020-02-01 0:58:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:58:48 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:58:48 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:58:48 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:58:48 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:58:48 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:58:48 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:58:48 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137297175; transaction id 5904657 2020-02-01 0:58:48 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:58:48 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:58:48 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:58:48 0 [Note] Starting crash recovery... 2020-02-01 0:58:48 0 [Note] Crash recovery finished. 2020-02-01 0:58:48 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:58:48 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:58:48 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:58:48 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000127' at position 337348 2020-02-01 0:58:48 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:58:48 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:58:48 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 0:58:48 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:58:48 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:58:48 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:58:48 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:58:48 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:58:48 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:58:48 2020-02-01 0:58:52 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000175, 158227) 2020-02-01 0:59:14 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 0:59:20 15 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 0:59:20 15 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 0:59:28 25 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 0:59:28 25 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 0:59:28 26 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 0:59:28 26 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 0:59:28 26 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 0:59:28 26 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 0:59:28 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f69c00009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f6a340fccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x562c0763cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x562c070d33af] sigaction.c:0(__restore_rt)[0x7f6a440d8630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x562c07001151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x562c0713f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x562c06ef5ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x562c06eebd4b] /usr/sbin/mysqld(+0x6ca792)[0x562c06ef6792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x562c06f1e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x562c06f2065d] /usr/sbin/mysqld(+0x65e3f3)[0x562c06e8a3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x562c06e89cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x562c06f1db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x562c06f2065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x562c06f20761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x562c06f21377] /usr/sbin/mysqld(+0x58fd84)[0x562c06dbbd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x562c06ec8b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x562c06ee0b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x562c06ee0cd2] /usr/sbin/mysqld(+0x6b55e7)[0x562c06ee15e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x562c06ee1677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x562c06ed056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x562c06ed207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x562c06fadafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x562c06fadbdd] pthread_create.c:0(start_thread)[0x7f6a440d0ea5] /lib64/libc.so.6(clone+0x6d)[0x7f6a424718cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f69c0010950): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 27 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 0:59:35 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 0:59:35 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 0:59:35 0 [Note] InnoDB: Uses event mutexes 2020-02-01 0:59:35 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 0:59:35 0 [Note] InnoDB: Number of pools: 1 2020-02-01 0:59:35 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 0:59:35 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 0:59:35 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 0:59:35 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 0:59:35 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137297530 2020-02-01 0:59:35 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 0:59:35 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 0:59:35 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 0:59:35 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 0:59:35 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 0:59:35 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 0:59:35 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 0:59:35 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137297539; transaction id 5904662 2020-02-01 0:59:35 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 0:59:35 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 0:59:35 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 0:59:35 0 [Note] Starting crash recovery... 2020-02-01 0:59:35 0 [Note] Crash recovery finished. 2020-02-01 0:59:35 0 [Note] Server socket created on IP: '::'. 2020-02-01 0:59:35 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 0:59:35 0 [Note] Added new Master_info '' to hash table 2020-02-01 0:59:35 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000127' at position 337348 2020-02-01 0:59:35 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 0:59:35 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 0:59:35 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 0:59:35 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 0:59:35 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 0:59:35 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 0:59:35 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 0:59:35 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 0:59:35 2020-02-01 0:59:45 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (113 "No route to host"), Internal MariaDB error code: 2003 2020-02-01 0:59:59 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:00:01 20 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:00:01 20 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:00:02 19 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:00:02 19 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 1:00:02 20 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:00:02 20 [Warning] Checking table: './wpms/wp_usermeta' 2020-02-01 1:00:06 27 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:00:06 27 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:00:06 28 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:00:06 28 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:00:06 28 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 1:00:06 28 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 1:00:06 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=6 max_threads=202 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc648000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc6d011ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a1c4c67f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a1c46fe3af] sigaction.c:0(__restore_rt)[0x7fc6de3cd630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a1c462c151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a1c476a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a1c4520ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a1c4516d4b] /usr/sbin/mysqld(+0x6ca792)[0x55a1c4521792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a1c45496da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a1c454b65d] /usr/sbin/mysqld(+0x65e3f3)[0x55a1c44b53f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a1c44b4cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a1c4548b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a1c454b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a1c454b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a1c454c377] /usr/sbin/mysqld(+0x58fd84)[0x55a1c43e6d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a1c44f3b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a1c450bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a1c450bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a1c450c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a1c450c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a1c44fb56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a1c44fd07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a1c45d8afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a1c45d8bdd] pthread_create.c:0(start_thread)[0x7fc6de3c5ea5] /lib64/libc.so.6(clone+0x6d)[0x7fc6dc7668cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc64800fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 29 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:00:13 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:00:13 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:00:13 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:00:13 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:00:13 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:00:13 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:00:13 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:00:13 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:00:13 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:00:13 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137297548 2020-02-01 1:00:13 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000121', position 68925 2020-02-01 1:00:13 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:00:13 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:00:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:00:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:00:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:00:13 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137297557; transaction id 5904662 2020-02-01 1:00:13 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:00:13 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:00:13 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:00:13 0 [Note] Starting crash recovery... 2020-02-01 1:00:13 0 [Note] Crash recovery finished. 2020-02-01 1:00:13 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:00:13 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:00:13 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:00:13 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:00:13 2020-02-01 1:00:13 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000127' at position 337348 2020-02-01 1:00:13 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:00:13 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:00:13 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:00:13 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:00:13 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:00:13 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:00:13 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:00:23 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (113 "No route to host"), Internal MariaDB error code: 2003 2020-02-01 1:00:40 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:00:47 21 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:00:47 21 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:00:47 22 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 1:00:47 22 [Warning] Checking table: './wpcomm306/wp_options' 2020-02-01 1:00:50 31 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 1:00:50 31 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 1:00:53 36 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:00:53 36 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:00:54 39 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:00:54 39 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:00:54 40 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:00:54 40 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:00:55 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=7 max_threads=202 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0fd0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0f704f5cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x557e2755ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x557e26ff53af] sigaction.c:0(__restore_rt)[0x7f0fe4ed8630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x557e26f23151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x557e27061545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x557e26e17ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x557e26e0dd4b] /usr/sbin/mysqld(+0x6ca792)[0x557e26e18792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x557e26e406da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557e26e4265d] /usr/sbin/mysqld(+0x65e3f3)[0x557e26dac3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x557e26dabcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x557e26e3fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557e26e4265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x557e26e42761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x557e26e43377] /usr/sbin/mysqld(+0x58fd84)[0x557e26cddd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x557e26deab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x557e26e02b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x557e26e02cd2] /usr/sbin/mysqld(+0x6b55e7)[0x557e26e035e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x557e26e03677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x557e26df256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x557e26df407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x557e26ecfafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x557e26ecfbdd] pthread_create.c:0(start_thread)[0x7f0fe4ed0ea5] /lib64/libc.so.6(clone+0x6d)[0x7f0fe32718cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f0fd000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 41 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:01:00 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:01:00 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:01:00 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:01:00 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:01:00 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:01:00 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:01:00 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:01:00 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:01:00 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:01:00 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137298081 2020-02-01 1:01:00 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:01:01 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:01:01 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:01:01 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:01:01 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:01:01 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:01:01 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:01:01 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137298090; transaction id 5904670 2020-02-01 1:01:01 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:01:01 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:01:01 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:01:01 0 [Note] Starting crash recovery... 2020-02-01 1:01:01 0 [Note] Crash recovery finished. 2020-02-01 1:01:01 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:01:01 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:01:01 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:01:01 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000127' at position 337348 2020-02-01 1:01:01 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:01:01 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:01:01 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 1:01:01 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 1:01:01 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 1:01:01 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 1:01:01 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:01:01 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:01:01 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 1:01:01 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:01:01 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:01:01 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:01:01 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:01:01 2020-02-01 1:01:04 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (113 "No route to host"), Internal MariaDB error code: 2003 2020-02-01 1:01:29 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000177, 346) 2020-02-01 1:01:30 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:01:31 17 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:01:31 17 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:01:31 18 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-02-01 1:01:31 18 [Warning] Checking table: './wpcomm305/comm305_options' 2020-02-01 1:01:32 20 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:01:32 20 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-02-01 1:01:36 27 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:01:36 27 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:01:36 28 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:01:36 28 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:01:36 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=5 max_threads=202 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0c40000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0cc010fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x558143ef9f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5581439903af] sigaction.c:0(__restore_rt)[0x7f0cce3be630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5581438be151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5581439fc545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5581437b2ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5581437a8d4b] /usr/sbin/mysqld(+0x6ca792)[0x5581437b3792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5581437db6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5581437dd65d] /usr/sbin/mysqld(+0x65e3f3)[0x5581437473f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x558143746cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5581437dab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5581437dd65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5581437dd761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5581437de377] /usr/sbin/mysqld(+0x58fd84)[0x558143678d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x558143785b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55814379db16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55814379dcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55814379e5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55814379e677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55814378d56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55814378f07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55814386aafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55814386abdd] pthread_create.c:0(start_thread)[0x7f0cce3b6ea5] /lib64/libc.so.6(clone+0x6d)[0x7f0ccc7578cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f0c4000feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 29 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:01:43 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:01:43 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:01:43 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:01:43 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:01:43 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:01:43 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:01:43 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:01:43 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:01:43 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:01:44 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137298099 2020-02-01 1:01:44 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:01:44 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:01:44 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:01:44 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:01:44 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:01:44 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:01:44 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:01:44 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137298108; transaction id 5904670 2020-02-01 1:01:44 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:01:44 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:01:44 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:01:44 0 [Note] Starting crash recovery... 2020-02-01 1:01:44 0 [Note] Crash recovery finished. 2020-02-01 1:01:44 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:01:44 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:01:44 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:01:44 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000127' at position 337348 2020-02-01 1:01:44 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:01:44 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:01:44 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000127' at position 337348 2020-02-01 1:01:44 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 1:01:44 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 1:01:44 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 1:01:44 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 1:01:44 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:01:44 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:01:44 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 1:01:44 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:01:44 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:01:44 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:01:44 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:01:44 2020-02-01 1:02:10 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:02:11 13 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:02:11 13 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:02:11 14 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:02:11 14 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:02:11 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=3 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f76c4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f772c0a2cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55bf9c411f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55bf9bea83af] sigaction.c:0(__restore_rt)[0x7f773a351630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55bf9bdd6151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55bf9bf14545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55bf9bccaccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55bf9bcc0d4b] /usr/sbin/mysqld(+0x6ca792)[0x55bf9bccb792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55bf9bcf36da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55bf9bcf565d] /usr/sbin/mysqld(+0x65e3f3)[0x55bf9bc5f3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55bf9bc5ecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55bf9bcf2b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55bf9bcf565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55bf9bcf5761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55bf9bcf6377] /usr/sbin/mysqld(+0x58fd84)[0x55bf9bb90d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55bf9bc9db6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55bf9bcb5b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55bf9bcb5cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55bf9bcb65e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55bf9bcb6677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55bf9bca556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55bf9bca707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55bf9bd82afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55bf9bd82bdd] pthread_create.c:0(start_thread)[0x7f773a349ea5] /lib64/libc.so.6(clone+0x6d)[0x7f77386ea8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f76c400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 15 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:02:17 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:02:17 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:02:17 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:02:17 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:02:17 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:02:17 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:02:17 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:02:17 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:02:17 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:02:17 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137298451 2020-02-01 1:02:17 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:02:17 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:02:17 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:02:17 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:02:17 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:02:17 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:02:17 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:02:17 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137298460; transaction id 5904675 2020-02-01 1:02:17 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:02:17 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:02:17 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:02:17 0 [Note] Starting crash recovery... 2020-02-01 1:02:17 0 [Note] Crash recovery finished. 2020-02-01 1:02:17 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:02:17 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:02:17 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:02:17 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000129' at position 56315 2020-02-01 1:02:17 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:02:17 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:02:17 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000129' at position 56315 2020-02-01 1:02:17 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:02:17 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:02:17 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:02:17 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:02:17 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:02:17 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:02:17 2020-02-01 1:02:18 10 [Warning] Slave I/O: SET @master_heartbeat_period to master failed with error: Lost connection to MySQL server during query, Internal MariaDB error code: 2013 2020-02-01 1:02:18 10 [Warning] Slave I/O: Master command COM_REGISTER_SLAVE failed: failed registering on master, reconnecting to try again, log 'dbsvc-527-bin.000129' at position 56315, Internal MariaDB error code: 1597 2020-02-01 1:02:18 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:02:29 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000180, 57984) 2020-02-01 1:02:45 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:02:53 22 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:02:53 22 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:02:59 30 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:02:59 30 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:02:59 31 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:02:59 31 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:02:59 32 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:02:59 32 [Warning] Checking table: './ksarn/wp_options' 200201 1:02:59 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=7 max_threads=202 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc17c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc1e4063cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560dafd73f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560daf80a3af] sigaction.c:0(__restore_rt)[0x7fc1f4fe1630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x560daf738151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x560daf876545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x560daf62cccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x560daf622d4b] /usr/sbin/mysqld(+0x6ca792)[0x560daf62d792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x560daf6556da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560daf65765d] /usr/sbin/mysqld(+0x65e3f3)[0x560daf5c13f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x560daf5c0cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x560daf654b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560daf65765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x560daf657761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x560daf658377] /usr/sbin/mysqld(+0x58fd84)[0x560daf4f2d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x560daf5ffb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x560daf617b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x560daf617cd2] /usr/sbin/mysqld(+0x6b55e7)[0x560daf6185e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x560daf618677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x560daf60756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x560daf60907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x560daf6e4afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560daf6e4bdd] pthread_create.c:0(start_thread)[0x7fc1f4fd9ea5] /lib64/libc.so.6(clone+0x6d)[0x7fc1f337a8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc17c00fe80): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 33 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:03:05 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:03:05 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:03:05 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:03:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:03:05 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:03:05 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:03:05 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:03:05 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:03:05 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:03:05 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137298799 2020-02-01 1:03:05 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:03:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:03:05 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:03:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:03:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:03:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:03:05 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:03:05 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137298808; transaction id 5904680 2020-02-01 1:03:05 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:03:05 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:03:05 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:03:05 0 [Note] Starting crash recovery... 2020-02-01 1:03:05 0 [Note] Crash recovery finished. 2020-02-01 1:03:05 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:03:05 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:03:05 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:03:06 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000129' at position 56315 2020-02-01 1:03:06 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:03:06 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:03:06 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000129' at position 56315 2020-02-01 1:03:06 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:03:06 2020-02-01 1:03:06 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:03:06 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:03:06 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:03:06 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:03:06 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:03:31 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:03:33 13 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:03:33 13 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:03:33 14 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:03:33 14 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:03:34 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=3 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2b78000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2bf011bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5632ac282f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5632abd193af] sigaction.c:0(__restore_rt)[0x7f2bfd460630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5632abc47151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5632abd85545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5632abb3bccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5632abb31d4b] /usr/sbin/mysqld(+0x6ca792)[0x5632abb3c792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5632abb646da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5632abb6665d] /usr/sbin/mysqld(+0x65e3f3)[0x5632abad03f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5632abacfcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5632abb63b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5632abb6665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5632abb66761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5632abb67377] /usr/sbin/mysqld(+0x58fd84)[0x5632aba01d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5632abb0eb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5632abb26b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5632abb26cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5632abb275e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5632abb27677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5632abb1656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5632abb1807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5632abbf3afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5632abbf3bdd] pthread_create.c:0(start_thread)[0x7f2bfd458ea5] /lib64/libc.so.6(clone+0x6d)[0x7f2bfb7f98cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f2b7800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 15 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:03:40 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:03:40 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:03:40 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:03:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:03:40 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:03:40 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:03:40 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:03:40 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:03:40 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:03:40 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137299142 2020-02-01 1:03:40 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:03:40 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:03:40 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:03:40 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:03:40 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:03:40 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:03:40 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:03:40 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137299151; transaction id 5904685 2020-02-01 1:03:40 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:03:40 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:03:40 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:03:40 0 [Note] Starting crash recovery... 2020-02-01 1:03:40 0 [Note] Crash recovery finished. 2020-02-01 1:03:40 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:03:40 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:03:40 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:03:40 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000130' at position 372361 2020-02-01 1:03:40 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:03:40 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:03:40 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:03:40 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:03:40 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:03:40 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:03:40 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:03:40 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:03:41 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:03:41 2020-02-01 1:03:46 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000182, 229748) 2020-02-01 1:04:06 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:04:06 15 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 1:04:06 15 [Warning] Checking table: './wpcomm306/wp_options' 2020-02-01 1:04:09 17 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:04:09 17 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:04:17 27 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:04:17 27 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 1:04:18 29 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-02-01 1:04:18 29 [Warning] Checking table: './openrecords/wp_options' 2020-02-01 1:04:29 36 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:04:29 36 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:04:29 37 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:04:29 37 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:04:30 37 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 1:04:30 37 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 1:04:30 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=5 max_threads=202 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0a78000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0a85911cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5652ef4c6f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5652eef5d3af] sigaction.c:0(__restore_rt)[0x7f0aeea22630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5652eee8b151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5652eefc9545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5652eed7fccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5652eed75d4b] /usr/sbin/mysqld(+0x6ca792)[0x5652eed80792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5652eeda86da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5652eedaa65d] /usr/sbin/mysqld(+0x65e3f3)[0x5652eed143f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5652eed13cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5652eeda7b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5652eedaa65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5652eedaa761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5652eedab377] /usr/sbin/mysqld(+0x58fd84)[0x5652eec45d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5652eed52b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5652eed6ab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5652eed6acd2] /usr/sbin/mysqld(+0x6b55e7)[0x5652eed6b5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5652eed6b677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5652eed5a56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5652eed5c07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5652eee37afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5652eee37bdd] pthread_create.c:0(start_thread)[0x7f0aeea1aea5] /lib64/libc.so.6(clone+0x6d)[0x7f0aecdbb8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f0a7800feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 38 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:04:35 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:04:35 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:04:35 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:04:35 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:04:35 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:04:35 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:04:35 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:04:35 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:04:35 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:04:35 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137299506 2020-02-01 1:04:35 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:04:36 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:04:36 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:04:36 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:04:36 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:04:36 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:04:36 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:04:36 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137299515; transaction id 5904690 2020-02-01 1:04:36 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:04:36 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:04:36 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:04:36 0 [Note] Starting crash recovery... 2020-02-01 1:04:36 0 [Note] Crash recovery finished. 2020-02-01 1:04:36 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:04:36 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:04:36 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:04:36 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000130' at position 372361 2020-02-01 1:04:36 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:04:36 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:04:36 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000130' at position 372361 2020-02-01 1:04:36 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:04:36 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:04:36 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:04:36 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:04:36 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:04:36 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:04:36 2020-02-01 1:05:01 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:05:03 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:05:03 16 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:05:03 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:05:03 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:05:03 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=3 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0c24000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0c98123cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56451395ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5645133f53af] sigaction.c:0(__restore_rt)[0x7f0ca6f25630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564513323151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564513461545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564513217ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56451320dd4b] /usr/sbin/mysqld(+0x6ca792)[0x564513218792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5645132406da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56451324265d] /usr/sbin/mysqld(+0x65e3f3)[0x5645131ac3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5645131abcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56451323fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56451324265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x564513242761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x564513243377] /usr/sbin/mysqld(+0x58fd84)[0x5645130ddd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5645131eab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564513202b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564513202cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5645132035e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564513203677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5645131f256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5645131f407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5645132cfafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5645132cfbdd] pthread_create.c:0(start_thread)[0x7f0ca6f1dea5] /lib64/libc.so.6(clone+0x6d)[0x7f0ca52be8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f0c2400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:05:09 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:05:09 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:05:09 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:05:09 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:05:09 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:05:09 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:05:09 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:05:09 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:05:09 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:05:09 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137299833 2020-02-01 1:05:09 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:05:09 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:05:09 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:05:09 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:05:09 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:05:09 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:05:09 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:05:09 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137299842; transaction id 5904695 2020-02-01 1:05:09 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:05:09 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:05:09 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:05:09 0 [Note] Starting crash recovery... 2020-02-01 1:05:09 0 [Note] Crash recovery finished. 2020-02-01 1:05:09 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:05:09 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:05:09 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:05:09 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000131' at position 32988 2020-02-01 1:05:09 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:05:09 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:05:09 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000131' at position 32988 2020-02-01 1:05:09 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:05:09 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:05:09 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:05:09 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:05:09 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:05:09 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:05:09 2020-02-01 1:05:18 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:05:18 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000131' at position 34705 2020-02-01 1:05:18 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:05:23 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000184, 412216) 2020-02-01 1:05:36 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:05:37 15 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 1:05:37 15 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 1:05:38 15 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:05:38 15 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 1:05:41 19 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:05:41 19 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:05:42 20 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:05:42 20 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:05:42 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=6 max_threads=202 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff788000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff7dd130cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56070e05ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56070daf53af] sigaction.c:0(__restore_rt)[0x7ff80025c630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56070da23151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56070db61545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56070d917ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56070d90dd4b] /usr/sbin/mysqld(+0x6ca792)[0x56070d918792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56070d9406da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56070d94265d] /usr/sbin/mysqld(+0x65e3f3)[0x56070d8ac3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56070d8abcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56070d93fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56070d94265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56070d942761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56070d943377] /usr/sbin/mysqld(+0x58fd84)[0x56070d7ddd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56070d8eab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56070d902b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56070d902cd2] /usr/sbin/mysqld(+0x6b55e7)[0x56070d9035e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56070d903677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56070d8f256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56070d8f407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56070d9cfafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56070d9cfbdd] pthread_create.c:0(start_thread)[0x7ff800254ea5] /lib64/libc.so.6(clone+0x6d)[0x7ff7fe5f58cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff78800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 21 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:05:48 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:05:48 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:05:48 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:05:48 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:05:48 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:05:48 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:05:48 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:05:48 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:05:48 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:05:48 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137302813 2020-02-01 1:05:48 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:05:49 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:05:49 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:05:49 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:05:49 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:05:49 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:05:49 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:05:49 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137302822; transaction id 5904706 2020-02-01 1:05:49 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:05:49 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:05:49 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:05:49 0 [Note] Starting crash recovery... 2020-02-01 1:05:49 0 [Note] Crash recovery finished. 2020-02-01 1:05:49 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:05:49 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:05:49 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:05:49 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000131' at position 34705 2020-02-01 1:05:49 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:05:49 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:05:49 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000131' at position 34705 2020-02-01 1:05:49 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:05:49 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:05:49 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:05:49 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:05:49 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:05:50 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:05:50 2020-02-01 1:05:54 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:05:54 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000132' at position 1464 2020-02-01 1:05:54 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:06:00 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000186, 30899) 2020-02-01 1:06:13 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown 2020-02-01 1:06:13 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-02-01 1:06:13 0 [Note] InnoDB: FTS optimize thread exiting. 2020-02-01 1:06:13 10 [Note] Slave I/O thread killed during or after a reconnect done to recover from failed read 2020-02-01 1:06:13 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000132', position 1464 2020-02-01 1:06:14 0 [Note] InnoDB: Starting shutdown... 2020-02-01 1:06:14 0 [Note] InnoDB: Dumping buffer pool(s) to /data/mysql/ib_buffer_pool 2020-02-01 1:06:14 0 [Note] InnoDB: Instance 0, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:06:14 0 [Note] InnoDB: Instance 1, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:06:14 0 [Note] InnoDB: Instance 2, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:06:14 0 [Note] InnoDB: Instance 3, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:06:14 0 [Note] InnoDB: Instance 4, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:06:14 0 [Note] InnoDB: Instance 5, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:06:14 0 [Note] InnoDB: Instance 6, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:06:14 0 [Note] InnoDB: Instance 7, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:06:14 0 [Note] InnoDB: Buffer pool(s) dump completed at 200201 1:06:14 2020-02-01 1:06:16 0 [Note] InnoDB: Shutdown completed; log sequence number 5137305678; transaction id 5904715 2020-02-01 1:06:16 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:06:16 0 [Note] /usr/sbin/mysqld: Shutdown complete 2020-02-01 1:08:04 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:08:04 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:08:04 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:08:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:08:04 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:08:04 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:08:04 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:08:04 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:08:04 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:08:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:08:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:08:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:08:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:08:05 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:08:05 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137305678; transaction id 5904718 2020-02-01 1:08:05 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:08:05 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:08:05 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:08:06 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:08:06 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:08:06 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000132' at position 1464 2020-02-01 1:08:06 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:08:06 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:08:06 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:08:06 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:08:06 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:08:06 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:08:06 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:08:06 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:08:07 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:08:07 2020-02-01 1:08:09 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000187, 346) 2020-02-01 1:08:10 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:08:10 14 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:08:10 14 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:08:12 14 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-02-01 1:08:12 14 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-02-01 1:08:12 16 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 1:08:12 16 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 1:08:14 20 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-02-01 1:08:14 20 [Warning] Checking table: './wpstart/wp_options' 2020-02-01 1:08:23 26 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 1:08:23 26 [Warning] Checking table: './wpcomm306/wp_options' 2020-02-01 1:08:25 28 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:08:25 28 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:08:25 29 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:08:25 29 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:08:25 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=5 max_threads=202 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe790000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe79858bcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5570c8f35f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5570c89cc3af] sigaction.c:0(__restore_rt)[0x7fe80ce7c630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5570c88fa151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5570c8a38545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5570c87eeccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5570c87e4d4b] /usr/sbin/mysqld(+0x6ca792)[0x5570c87ef792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5570c88176da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5570c881965d] /usr/sbin/mysqld(+0x65e3f3)[0x5570c87833f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5570c8782cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5570c8816b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5570c881965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5570c8819761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5570c881a377] /usr/sbin/mysqld(+0x58fd84)[0x5570c86b4d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5570c87c1b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5570c87d9b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5570c87d9cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5570c87da5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5570c87da677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5570c87c956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5570c87cb07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5570c88a6afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5570c88a6bdd] pthread_create.c:0(start_thread)[0x7fe80ce74ea5] /lib64/libc.so.6(clone+0x6d)[0x7fe80b2158cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe79000fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 30 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:08:31 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:08:31 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:08:31 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:08:31 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:08:31 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:08:31 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:08:31 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:08:31 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:08:31 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:08:32 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137306017 2020-02-01 1:08:32 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:08:32 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:08:32 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:08:32 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:08:32 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:08:32 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:08:32 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:08:33 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137306026; transaction id 5904723 2020-02-01 1:08:33 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:08:33 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:08:33 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:08:33 0 [Note] Starting crash recovery... 2020-02-01 1:08:33 0 [Note] Crash recovery finished. 2020-02-01 1:08:33 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:08:33 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:08:33 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:08:33 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000132' at position 1464 2020-02-01 1:08:33 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:08:33 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:08:33 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 1:08:33 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 1:08:33 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 1:08:33 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000132' at position 1464 2020-02-01 1:08:33 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 1:08:33 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:08:33 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:08:33 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 1:08:33 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:08:33 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:08:33 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:08:33 10 [Note] Slave I/O thread killed while calling get_master_version_and_clock(...) 2020-02-01 1:08:33 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000132', position 1464 2020-02-01 1:08:33 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:08:33 2020-02-01 1:09:00 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:09:02 15 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 1:09:02 15 [Warning] Checking table: './wpcomm306/wp_options' 2020-02-01 1:09:03 17 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 1:09:03 17 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 1:09:03 18 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000188, 202763) 2020-02-01 1:09:04 19 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:09:04 19 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:09:04 20 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:09:04 20 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:09:04 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=5 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2a00000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2a800bdcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a4e3012f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a4e2aa93af] sigaction.c:0(__restore_rt)[0x7f2a8ece3630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a4e29d7151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a4e2b15545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a4e28cbccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a4e28c1d4b] /usr/sbin/mysqld(+0x6ca792)[0x55a4e28cc792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a4e28f46da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a4e28f665d] /usr/sbin/mysqld(+0x65e3f3)[0x55a4e28603f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a4e285fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a4e28f3b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a4e28f665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a4e28f6761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a4e28f7377] /usr/sbin/mysqld(+0x58fd84)[0x55a4e2791d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a4e289eb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a4e28b6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a4e28b6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a4e28b75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a4e28b7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a4e28a656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a4e28a807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a4e2983afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a4e2983bdd] pthread_create.c:0(start_thread)[0x7f2a8ecdbea5] /lib64/libc.so.6(clone+0x6d)[0x7f2a8d07c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f2a0000fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 21 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:09:10 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:09:10 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:09:10 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:09:10 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:09:10 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:09:10 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:09:10 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:09:10 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:09:10 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:09:10 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137306035 2020-02-01 1:09:10 0 [Note] InnoDB: Transaction 5904728 was in the XA prepared state. 2020-02-01 1:09:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo 2020-02-01 1:09:10 0 [Note] InnoDB: Trx id counter is 5904729 2020-02-01 1:09:10 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:09:10 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:09:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions 2020-02-01 1:09:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed 2020-02-01 1:09:10 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:09:10 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:09:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:09:10 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:09:10 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:09:10 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137306331; transaction id 5904731 2020-02-01 1:09:10 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:09:10 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:09:10 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:09:10 0 [Note] Starting crash recovery... 2020-02-01 1:09:10 0 [Note] InnoDB: Starting recovery for XA transactions... 2020-02-01 1:09:10 0 [Note] InnoDB: Transaction 5904728 in prepared state after recovery 2020-02-01 1:09:10 0 [Note] InnoDB: Transaction contains changes to 1 rows 2020-02-01 1:09:10 0 [Note] InnoDB: 1 transactions in prepared state after recovery 2020-02-01 1:09:10 0 [Note] Found 1 prepared transaction(s) in InnoDB 2020-02-01 1:09:10 0 [Note] Crash recovery finished. 2020-02-01 1:09:11 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:09:11 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:09:11 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:09:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000132' at position 1464 2020-02-01 1:09:11 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:09:11 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:09:11 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:09:11 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:09:11 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:09:11 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:09:11 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:09:11 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000132' at position 1464 2020-02-01 1:09:12 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:09:12 2020-02-01 1:09:35 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:09:40 15 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:09:40 15 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:09:57 26 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:09:57 26 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:09:57 27 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-02-01 1:09:57 27 [Warning] Checking table: './ulrich/wp_options' 2020-02-01 1:09:59 31 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-02-01 1:09:59 31 [Warning] Checking table: './shockeripe/wp_options' 2020-02-01 1:09:59 30 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:09:59 30 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 1:10:05 36 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000189, 22537) 2020-02-01 1:10:31 45 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-02-01 1:10:31 45 [Warning] Checking table: './wpstart/wp_options' 2020-02-01 1:10:53 53 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:10:53 53 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:10:53 54 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:10:53 54 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:10:53 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=6 max_threads=202 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fa0d0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fa1240b1cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5636ce49cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5636cdf333af] sigaction.c:0(__restore_rt)[0x7fa1462b2630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5636cde61151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5636cdf9f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5636cdd55ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5636cdd4bd4b] /usr/sbin/mysqld(+0x6ca792)[0x5636cdd56792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5636cdd7e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5636cdd8065d] /usr/sbin/mysqld(+0x65e3f3)[0x5636cdcea3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5636cdce9cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5636cdd7db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5636cdd8065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5636cdd80761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5636cdd81377] /usr/sbin/mysqld(+0x58fd84)[0x5636cdc1bd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5636cdd28b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5636cdd40b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5636cdd40cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5636cdd415e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5636cdd41677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5636cdd3056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5636cdd3207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5636cde0dafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5636cde0dbdd] pthread_create.c:0(start_thread)[0x7fa1462aaea5] /lib64/libc.so.6(clone+0x6d)[0x7fa14464b8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fa0d000fe80): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 55 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:11:00 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:11:00 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:11:00 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:11:00 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:11:00 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:11:00 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:11:00 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:11:00 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:11:00 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:11:00 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137306858 2020-02-01 1:11:00 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:11:00 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:11:00 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:11:00 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:11:00 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:11:00 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:11:00 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:11:00 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137306867; transaction id 5904737 2020-02-01 1:11:00 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:11:00 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:11:00 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:11:00 0 [Note] Starting crash recovery... 2020-02-01 1:11:00 0 [Note] Crash recovery finished. 2020-02-01 1:11:00 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:11:00 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:11:00 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:11:00 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000137' at position 285480 2020-02-01 1:11:00 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:11:00 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:11:00 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 1:11:00 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 1:11:00 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 1:11:00 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 1:11:00 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000137' at position 285480 2020-02-01 1:11:00 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:11:00 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:11:00 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 1:11:00 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:11:00 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:11:00 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:11:01 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:11:01 2020-02-01 1:11:10 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:11:10 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000137' at position 286598 2020-02-01 1:11:10 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:11:15 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000190, 392464) 2020-02-01 1:11:26 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:11:27 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:11:27 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:11:27 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:11:27 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:11:27 17 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 200201 1:11:27 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f5d44000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f5d9c7c8cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e3a3f12f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e3a39a93af] sigaction.c:0(__restore_rt)[0x7f5dc186d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55e3a38d7151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55e3a3a15545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55e3a37cbccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55e3a37c1d4b] /usr/sbin/mysqld(+0x6ca792)[0x55e3a37cc792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55e3a37f46da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e3a37f665d] /usr/sbin/mysqld(+0x65e3f3)[0x55e3a37603f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55e3a375fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55e3a37f3b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e3a37f665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55e3a37f6761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55e3a37f7377] /usr/sbin/mysqld(+0x58fd84)[0x55e3a3691d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55e3a379eb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55e3a37b6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55e3a37b6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55e3a37b75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55e3a37b7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55e3a37a656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55e3a37a807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55e3a3883afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e3a3883bdd] pthread_create.c:0(start_thread)[0x7f5dc1865ea5] /lib64/libc.so.6(clone+0x6d)[0x7f5dbfc068cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f5d4400fe50): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:11:33 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:11:33 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:11:33 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:11:33 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:11:33 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:11:33 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:11:33 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:11:33 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:11:33 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:11:33 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137310512 2020-02-01 1:11:33 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:11:33 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:11:33 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:11:33 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:11:33 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:11:33 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:11:33 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:11:33 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137310521; transaction id 5904748 2020-02-01 1:11:33 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:11:33 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:11:33 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:11:33 0 [Note] Starting crash recovery... 2020-02-01 1:11:33 0 [Note] Crash recovery finished. 2020-02-01 1:11:33 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:11:33 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:11:33 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:11:33 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000137' at position 286598 2020-02-01 1:11:33 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:11:33 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:11:33 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000137' at position 286598 2020-02-01 1:11:33 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:11:33 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:11:33 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:11:33 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:11:33 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:11:34 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:11:34 2020-02-01 1:12:01 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:12:10 19 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 1:12:10 19 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 1:12:10 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:12:10 20 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:12:10 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:12:10 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:12:10 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=5 max_threads=202 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ffac8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ffb2476ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5596737d7f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55967326e3af] sigaction.c:0(__restore_rt)[0x7ffb47856630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55967319c151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5596732da545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x559673090ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x559673086d4b] /usr/sbin/mysqld(+0x6ca792)[0x559673091792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5596730b96da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5596730bb65d] /usr/sbin/mysqld(+0x65e3f3)[0x5596730253f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x559673024cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5596730b8b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5596730bb65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5596730bb761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5596730bc377] /usr/sbin/mysqld(+0x58fd84)[0x559672f56d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x559673063b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55967307bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55967307bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55967307c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55967307c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55967306b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55967306d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x559673148afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x559673148bdd] pthread_create.c:0(start_thread)[0x7ffb4784eea5] /lib64/libc.so.6(clone+0x6d)[0x7ffb45bef8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ffac800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:12:17 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:12:17 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:12:17 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:12:17 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:12:17 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:12:17 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:12:17 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:12:17 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:12:17 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:12:17 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137313374 2020-02-01 1:12:17 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:12:17 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:12:17 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:12:17 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:12:17 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:12:17 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:12:17 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:12:17 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137313383; transaction id 5904759 2020-02-01 1:12:17 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:12:17 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:12:17 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:12:17 0 [Note] Starting crash recovery... 2020-02-01 1:12:17 0 [Note] Crash recovery finished. 2020-02-01 1:12:17 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:12:17 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:12:17 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:12:17 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000138' at position 162119 2020-02-01 1:12:17 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:12:17 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:12:17 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000138' at position 162119 2020-02-01 1:12:17 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 1:12:17 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 1:12:17 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 1:12:17 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 1:12:17 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:12:17 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:12:17 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 1:12:17 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:12:17 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:12:17 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:12:18 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:12:18 2020-02-01 1:12:27 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000191, 1464) 2020-02-01 1:12:41 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:12:41 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000138' at position 163237 2020-02-01 1:12:41 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:12:46 19 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:12:49 20 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 346) 2020-02-01 1:12:52 22 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-02-01 1:12:52 22 [Warning] Checking table: './openrecords/wp_options' 2020-02-01 1:12:56 25 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:12:56 25 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:12:56 26 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:12:56 26 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:12:57 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=6 max_threads=202 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fd1d0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fd2203b8cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x562668a86f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56266851d3af] sigaction.c:0(__restore_rt)[0x7fd2454cc630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56266844b151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x562668589545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56266833fccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x562668335d4b] /usr/sbin/mysqld(+0x6ca792)[0x562668340792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5626683686da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56266836a65d] /usr/sbin/mysqld(+0x65e3f3)[0x5626682d43f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5626682d3cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x562668367b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56266836a65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56266836a761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56266836b377] /usr/sbin/mysqld(+0x58fd84)[0x562668205d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x562668312b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56266832ab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56266832acd2] /usr/sbin/mysqld(+0x6b55e7)[0x56266832b5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56266832b677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56266831a56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56266831c07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5626683f7afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5626683f7bdd] pthread_create.c:0(start_thread)[0x7fd2454c4ea5] /lib64/libc.so.6(clone+0x6d)[0x7fd2438658cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fd1d000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 27 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:13:02 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:13:02 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:13:02 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:13:02 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:13:02 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:13:02 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:13:02 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:13:02 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:13:02 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:13:02 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137316221 2020-02-01 1:13:02 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:13:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:13:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:13:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:13:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:13:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:13:03 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137316230; transaction id 5904770 2020-02-01 1:13:03 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:13:03 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:13:03 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:13:03 0 [Note] Starting crash recovery... 2020-02-01 1:13:03 0 [Note] Crash recovery finished. 2020-02-01 1:13:03 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:13:03 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:13:03 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:13:03 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000138' at position 163237 2020-02-01 1:13:03 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:13:03 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:13:03 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000138' at position 163237 2020-02-01 1:13:03 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:13:03 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:13:03 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:13:03 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:13:03 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:13:04 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:13:04 2020-02-01 1:13:31 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:13:39 14 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:13:39 14 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:13:57 21 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:14:00 22 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 1:14:00 22 [Warning] Checking table: './wpcomm306/wp_options' 2020-02-01 1:14:11 30 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:14:11 30 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:14:11 31 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:14:11 31 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:14:11 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=6 max_threads=202 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0468000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f047573fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x555c5c9b9f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x555c5c4503af] sigaction.c:0(__restore_rt)[0x7f04de840630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x555c5c37e151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x555c5c4bc545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x555c5c272ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x555c5c268d4b] /usr/sbin/mysqld(+0x6ca792)[0x555c5c273792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x555c5c29b6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x555c5c29d65d] /usr/sbin/mysqld(+0x65e3f3)[0x555c5c2073f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x555c5c206cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x555c5c29ab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x555c5c29d65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x555c5c29d761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x555c5c29e377] /usr/sbin/mysqld(+0x58fd84)[0x555c5c138d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x555c5c245b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x555c5c25db16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x555c5c25dcd2] /usr/sbin/mysqld(+0x6b55e7)[0x555c5c25e5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x555c5c25e677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x555c5c24d56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x555c5c24f07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x555c5c32aafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x555c5c32abdd] pthread_create.c:0(start_thread)[0x7f04de838ea5] /lib64/libc.so.6(clone+0x6d)[0x7f04dcbd98cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f046800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 32 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:14:17 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:14:17 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:14:17 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:14:17 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:14:17 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:14:17 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:14:17 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:14:17 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:14:17 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:14:17 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137319084 2020-02-01 1:14:17 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:14:17 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:14:17 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:14:17 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:14:17 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:14:17 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:14:17 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:14:17 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137319093; transaction id 5904781 2020-02-01 1:14:17 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:14:17 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:14:17 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:14:17 0 [Note] Starting crash recovery... 2020-02-01 1:14:17 0 [Note] Crash recovery finished. 2020-02-01 1:14:18 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:14:18 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:14:18 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:14:18 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000139' at position 390387 2020-02-01 1:14:18 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:14:18 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:14:18 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000139' at position 390387 2020-02-01 1:14:18 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:14:18 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:14:18 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:14:18 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:14:18 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:14:19 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:14:19 2020-02-01 1:14:32 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:14:32 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000139' at position 417780 2020-02-01 1:14:32 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:14:37 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:14:47 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:14:48 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:14:48 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:14:48 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:14:48 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:14:48 16 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 1:14:48 16 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 1:14:48 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=3 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7effac000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f00200efcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55aac24e9f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55aac1f803af] sigaction.c:0(__restore_rt)[0x7f0030fd7630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55aac1eae151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55aac1fec545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55aac1da2ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55aac1d98d4b] /usr/sbin/mysqld(+0x6ca792)[0x55aac1da3792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55aac1dcb6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55aac1dcd65d] /usr/sbin/mysqld(+0x65e3f3)[0x55aac1d373f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55aac1d36cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55aac1dcab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55aac1dcd65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55aac1dcd761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55aac1dce377] /usr/sbin/mysqld(+0x58fd84)[0x55aac1c68d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55aac1d75b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55aac1d8db16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55aac1d8dcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55aac1d8e5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55aac1d8e677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55aac1d7d56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55aac1d7f07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55aac1e5aafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55aac1e5abdd] pthread_create.c:0(start_thread)[0x7f0030fcfea5] /lib64/libc.so.6(clone+0x6d)[0x7f002f3708cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7effac00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 17 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:14:54 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:14:54 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:14:54 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:14:54 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:14:54 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:14:54 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:14:54 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:14:54 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:14:54 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:14:54 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137321931 2020-02-01 1:14:54 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:14:54 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:14:54 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:14:54 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:14:54 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:14:54 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:14:54 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:14:54 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137321940; transaction id 5904792 2020-02-01 1:14:54 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:14:54 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:14:54 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:14:54 0 [Note] Starting crash recovery... 2020-02-01 1:14:54 0 [Note] Crash recovery finished. 2020-02-01 1:14:54 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:14:54 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:14:54 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:14:54 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000139' at position 417780 2020-02-01 1:14:54 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:14:54 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:14:54 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:14:54 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:14:54 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:14:54 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:14:54 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:14:54 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000139' at position 417780 2020-02-01 1:14:55 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:14:55 2020-02-01 1:15:18 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:15:18 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000140' at position 55482 2020-02-01 1:15:18 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:15:22 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:15:23 17 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:15:23 17 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:15:23 18 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:15:23 18 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:15:23 19 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:15:23 19 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:15:23 19 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 1:15:23 19 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 1:15:23 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=7 max_threads=202 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f5534000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f5538bc9cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x558adac61f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x558ada6f83af] sigaction.c:0(__restore_rt)[0x7f555dd3e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x558ada626151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x558ada764545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x558ada51accf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x558ada510d4b] /usr/sbin/mysqld(+0x6ca792)[0x558ada51b792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x558ada5436da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x558ada54565d] /usr/sbin/mysqld(+0x65e3f3)[0x558ada4af3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x558ada4aecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x558ada542b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x558ada54565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x558ada545761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x558ada546377] /usr/sbin/mysqld(+0x58fd84)[0x558ada3e0d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x558ada4edb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x558ada505b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x558ada505cd2] /usr/sbin/mysqld(+0x6b55e7)[0x558ada5065e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x558ada506677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x558ada4f556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x558ada4f707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x558ada5d2afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x558ada5d2bdd] pthread_create.c:0(start_thread)[0x7f555dd36ea5] /lib64/libc.so.6(clone+0x6d)[0x7f555c0d78cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f553400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 21 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:15:29 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:15:29 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:15:29 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:15:29 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:15:29 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:15:29 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:15:29 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:15:29 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:15:29 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:15:29 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137324820 2020-02-01 1:15:29 0 [Note] InnoDB: Starting final batch to recover 1 pages from redo log. 2020-02-01 1:15:30 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:15:30 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:15:30 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:15:30 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:15:30 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:15:30 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:15:30 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:15:30 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137333661; transaction id 5904810 2020-02-01 1:15:30 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:15:30 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:15:30 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:15:30 0 [Note] Starting crash recovery... 2020-02-01 1:15:30 0 [Note] Crash recovery finished. 2020-02-01 1:15:30 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:15:30 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:15:30 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:15:30 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000140' at position 55482 2020-02-01 1:15:30 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:15:30 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:15:30 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000140' at position 55482 2020-02-01 1:15:30 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:15:30 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:15:30 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:15:30 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:15:30 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:15:30 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:15:30 2020-02-01 1:15:57 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:15:59 15 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:15:59 15 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:15:59 17 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 1:15:59 17 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 1:16:04 24 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:16:04 24 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:16:04 25 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:16:04 25 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:16:05 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=7 max_threads=202 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2e0c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2e1d8c1cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x557020e5ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5570208f63af] sigaction.c:0(__restore_rt)[0x7f2e34ed3630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x557020824151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x557020962545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x557020718ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55702070ed4b] /usr/sbin/mysqld(+0x6ca792)[0x557020719792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5570207416da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55702074365d] /usr/sbin/mysqld(+0x65e3f3)[0x5570206ad3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5570206accb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x557020740b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55702074365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x557020743761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x557020744377] /usr/sbin/mysqld(+0x58fd84)[0x5570205ded84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5570206ebb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x557020703b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x557020703cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5570207045e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x557020704677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5570206f356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5570206f507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5570207d0afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5570207d0bdd] pthread_create.c:0(start_thread)[0x7f2e34ecbea5] /lib64/libc.so.6(clone+0x6d)[0x7f2e3326c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f2e0c00fe80): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 27 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:16:11 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:16:11 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:16:11 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:16:11 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:16:11 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:16:11 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:16:11 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:16:11 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:16:11 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:16:11 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137337378 2020-02-01 1:16:11 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:16:11 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:16:11 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:16:11 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:16:11 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:16:11 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:16:11 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:16:11 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137337387; transaction id 5904821 2020-02-01 1:16:11 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:16:11 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:16:11 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:16:11 0 [Note] Starting crash recovery... 2020-02-01 1:16:11 0 [Note] Crash recovery finished. 2020-02-01 1:16:11 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:16:11 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:16:11 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:16:11 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000141' at position 1410 2020-02-01 1:16:11 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:16:11 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:16:11 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000141' at position 1410 2020-02-01 1:16:11 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:16:11 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:16:11 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:16:11 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:16:11 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:16:12 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:16:12 2020-02-01 1:16:20 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:16:20 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000141' at position 26730 2020-02-01 1:16:20 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:16:26 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:16:37 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:16:40 16 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 1:16:40 16 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 1:16:40 18 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-02-01 1:16:40 18 [Warning] Checking table: './openrecords/wp_options' 2020-02-01 1:16:49 30 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:16:49 30 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:16:52 32 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:16:52 32 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:16:52 33 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:16:52 33 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:16:52 33 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 1:16:52 33 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 1:16:52 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=5 max_threads=202 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe9dc0009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fea5014ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55cbe4f49f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55cbe49e03af] sigaction.c:0(__restore_rt)[0x7fea60564630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55cbe490e151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55cbe4a4c545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55cbe4802ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55cbe47f8d4b] /usr/sbin/mysqld(+0x6ca792)[0x55cbe4803792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55cbe482b6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55cbe482d65d] /usr/sbin/mysqld(+0x65e3f3)[0x55cbe47973f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55cbe4796cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55cbe482ab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55cbe482d65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55cbe482d761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55cbe482e377] /usr/sbin/mysqld(+0x58fd84)[0x55cbe46c8d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55cbe47d5b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55cbe47edb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55cbe47edcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55cbe47ee5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55cbe47ee677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55cbe47dd56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55cbe47df07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55cbe48baafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55cbe48babdd] pthread_create.c:0(start_thread)[0x7fea6055cea5] /lib64/libc.so.6(clone+0x6d)[0x7fea5e8fd8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe9dc00fda0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 34 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:16:58 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:16:58 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:16:58 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:16:58 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:16:58 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:16:58 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:16:58 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:16:58 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:16:58 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:16:58 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137340225 2020-02-01 1:16:58 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:16:59 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:16:59 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:16:59 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:16:59 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:16:59 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:16:59 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137340234; transaction id 5904832 2020-02-01 1:16:59 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:16:59 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:16:59 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:16:59 0 [Note] Starting crash recovery... 2020-02-01 1:16:59 0 [Note] Crash recovery finished. 2020-02-01 1:16:59 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:16:59 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:16:59 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:16:59 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000141' at position 26730 2020-02-01 1:16:59 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:16:59 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:16:59 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000141' at position 26730 2020-02-01 1:16:59 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 1:16:59 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 1:16:59 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 1:16:59 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 1:16:59 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:16:59 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:16:59 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 1:16:59 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:16:59 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:16:59 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:17:00 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:17:00 2020-02-01 1:17:04 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:17:04 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000142' at position 2513 2020-02-01 1:17:04 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:17:10 17 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:17:28 18 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:17:28 19 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:17:28 19 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:17:28 20 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-02-01 1:17:28 20 [Warning] Checking table: './wpsem/wp_options' 2020-02-01 1:17:30 25 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-02-01 1:17:30 25 [Warning] Checking table: './wpcomm303/wp_options' 2020-02-01 1:17:33 30 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:17:33 30 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:17:33 31 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:17:33 31 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:17:33 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fa21c0009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fa28013fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e5aafa5f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e5aaa3c3af] sigaction.c:0(__restore_rt)[0x7fa2a3189630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55e5aa96a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55e5aaaa8545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55e5aa85eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55e5aa854d4b] /usr/sbin/mysqld(+0x6ca792)[0x55e5aa85f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55e5aa8876da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e5aa88965d] /usr/sbin/mysqld(+0x65e3f3)[0x55e5aa7f33f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55e5aa7f2cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55e5aa886b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55e5aa88965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55e5aa889761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55e5aa88a377] /usr/sbin/mysqld(+0x58fd84)[0x55e5aa724d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55e5aa831b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55e5aa849b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55e5aa849cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55e5aa84a5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55e5aa84a677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55e5aa83956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55e5aa83b07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55e5aa916afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e5aa916bdd] pthread_create.c:0(start_thread)[0x7fa2a3181ea5] /lib64/libc.so.6(clone+0x6d)[0x7fa2a15228cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fa21c00fde0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 32 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:17:39 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:17:39 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:17:39 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:17:39 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:17:39 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:17:39 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:17:39 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:17:39 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:17:39 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:17:39 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137343538 2020-02-01 1:17:39 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:17:39 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:17:39 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:17:39 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:17:39 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:17:39 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:17:39 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:17:39 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137343586; transaction id 5904847 2020-02-01 1:17:39 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:17:39 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:17:39 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:17:39 0 [Note] Starting crash recovery... 2020-02-01 1:17:39 0 [Note] Crash recovery finished. 2020-02-01 1:17:39 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:17:39 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:17:39 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:17:39 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000142' at position 2513 2020-02-01 1:17:39 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:17:39 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:17:39 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000142' at position 2513 2020-02-01 1:17:39 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:17:39 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:17:39 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:17:39 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:17:39 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:17:40 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:17:40 2020-02-01 1:18:08 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:18:08 13 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:18:08 13 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:18:10 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:18:10 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:18:10 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:18:10 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:18:11 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=3 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe4440009c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe4c8059cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5617df59bf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5617df0323af] sigaction.c:0(__restore_rt)[0x7fe4d51b5630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5617def60151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5617df09e545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5617dee54ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5617dee4ad4b] /usr/sbin/mysqld(+0x6ca792)[0x5617dee55792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5617dee7d6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5617dee7f65d] /usr/sbin/mysqld(+0x65e3f3)[0x5617dede93f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5617dede8cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5617dee7cb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5617dee7f65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5617dee7f761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5617dee80377] /usr/sbin/mysqld(+0x58fd84)[0x5617ded1ad84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5617dee27b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5617dee3fb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5617dee3fcd2] /usr/sbin/mysqld(+0x6b55e7)[0x5617dee405e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5617dee40677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5617dee2f56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5617dee3107c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5617def0cafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5617def0cbdd] pthread_create.c:0(start_thread)[0x7fe4d51adea5] /lib64/libc.so.6(clone+0x6d)[0x7fe4d354e8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe44400fe00): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:18:17 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:18:17 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:18:17 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:18:17 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:18:17 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:18:17 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:18:17 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M 2020-02-01 1:18:17 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:18:17 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:18:17 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137346502 2020-02-01 1:18:17 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:18:17 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:18:17 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:18:17 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:18:17 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:18:17 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:18:17 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:18:17 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137346511; transaction id 5904858 2020-02-01 1:18:17 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:18:17 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:18:17 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:18:17 0 [Note] Starting crash recovery... 2020-02-01 1:18:17 0 [Note] Crash recovery finished. 2020-02-01 1:18:17 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:18:17 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:18:17 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:18:17 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000143' at position 86426 2020-02-01 1:18:17 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:18:17 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:18:17 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000143' at position 86426 2020-02-01 1:18:17 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:18:17 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:18:17 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:18:17 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:18:17 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:18:18 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:18:18 2020-02-01 1:18:19 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:18:19 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000143' at position 87534 2020-02-01 1:18:19 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:18:25 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:18:43 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:18:54 17 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 1:18:54 17 [Warning] Checking table: './wpcomm306/wp_options' 2020-02-01 1:18:55 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown 2020-02-01 1:18:55 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-02-01 1:18:55 0 [Note] InnoDB: FTS optimize thread exiting. 2020-02-01 1:18:55 10 [Note] Slave I/O thread killed during or after a reconnect done to recover from failed read 2020-02-01 1:18:55 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000143', position 87534 2020-02-01 1:18:55 0 [Note] InnoDB: Starting shutdown... 2020-02-01 1:18:55 0 [Note] InnoDB: Dumping buffer pool(s) to /data/mysql/ib_buffer_pool 2020-02-01 1:18:55 0 [Note] InnoDB: Instance 0, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:18:55 0 [Note] InnoDB: Instance 1, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:18:55 0 [Note] InnoDB: Instance 2, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:18:55 0 [Note] InnoDB: Instance 3, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:18:55 0 [Note] InnoDB: Instance 4, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:18:55 0 [Note] InnoDB: Instance 5, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:18:55 0 [Note] InnoDB: Instance 6, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:18:55 0 [Note] InnoDB: Instance 7, restricted to 2003 pages due to innodb_buf_pool_dump_pct=25 2020-02-01 1:18:55 0 [Note] InnoDB: Buffer pool(s) dump completed at 200201 1:18:55 2020-02-01 1:18:56 0 [Note] InnoDB: Shutdown completed; log sequence number 5137349374; transaction id 5904868 2020-02-01 1:18:56 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:18:56 0 [Note] /usr/sbin/mysqld: Shutdown complete 2020-02-01 1:18:56 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:18:56 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:18:56 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:18:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:18:56 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:18:56 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:18:56 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:18:56 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:18:56 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:18:56 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:18:56 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:18:56 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:18:56 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:18:56 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:18:56 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137349374; transaction id 5904869 2020-02-01 1:18:56 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:18:56 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:18:56 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:18:56 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:18:56 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:18:56 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000143' at position 87534 2020-02-01 1:18:56 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:18:56 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:18:57 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000143' at position 87534 2020-02-01 1:18:57 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:18:57 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:18:57 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:18:57 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:18:57 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:18:57 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:18:57 13 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 1:18:57 13 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 1:18:58 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:18:58 2020-02-01 1:18:59 16 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:18:59 16 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:19:07 22 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-02-01 1:19:21 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:19:21 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000144', position 346 2020-02-01 1:19:22 29 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:19:22 29 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:19:22 30 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:19:22 30 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:19:22 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=4 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fccec000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fcdb58bfcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x556f6558ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x556f650263af] sigaction.c:0(__restore_rt)[0x7fcdb5627630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x556f64f54151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x556f65092545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x556f64e48ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x556f64e3ed4b] /usr/sbin/mysqld(+0x6ca792)[0x556f64e49792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x556f64e716da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556f64e7365d] /usr/sbin/mysqld(+0x65e3f3)[0x556f64ddd3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x556f64ddccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x556f64e70b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556f64e7365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x556f64e73761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x556f64e74377] /usr/sbin/mysqld(+0x58fd84)[0x556f64d0ed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x556f64e1bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x556f64e33b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x556f64e33cd2] /usr/sbin/mysqld(+0x6b55e7)[0x556f64e345e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x556f64e34677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x556f64e2356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x556f64e2507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x556f64f00afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x556f64f00bdd] pthread_create.c:0(start_thread)[0x7fcdb561fea5] /lib64/libc.so.6(clone+0x6d)[0x7fcdb39c08cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fccec00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 31 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:19:28 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:19:28 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:19:28 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:19:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:19:28 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:19:28 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:19:28 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:19:28 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:19:28 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:19:28 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137352254 2020-02-01 1:19:28 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:19:28 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:19:28 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:19:28 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:19:28 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:19:28 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:19:28 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:19:29 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137352263; transaction id 5904881 2020-02-01 1:19:29 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:19:29 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:19:29 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:19:29 0 [Note] Starting crash recovery... 2020-02-01 1:19:29 0 [Note] Crash recovery finished. 2020-02-01 1:19:29 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:19:29 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:19:29 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:19:29 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000144' at position 346 2020-02-01 1:19:29 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:19:29 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:19:29 10 [ERROR] Slave I/O: error connecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:19:29 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 1:19:29 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 1:19:29 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 1:19:29 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 1:19:29 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:19:29 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:19:29 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 1:19:29 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:19:29 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:19:29 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:19:29 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:19:29 2020-02-01 1:19:30 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:19:42 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:19:53 16 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:19:57 19 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-02-01 1:19:57 19 [Warning] Checking table: './ulrich/wp_options' 2020-02-01 1:19:58 24 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-02-01 1:19:58 24 [Warning] Checking table: './shockeripe/wp_options' 2020-02-01 1:19:59 22 [ERROR] mysqld: Table './wpms/wp_gf_form_view' is marked as crashed and should be repaired 2020-02-01 1:19:59 22 [Warning] Checking table: './wpms/wp_gf_form_view' 2020-02-01 1:19:59 27 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:19:59 27 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:19:59 20 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:19:59 20 [Warning] Checking table: './wpms/wp_usermeta' 2020-02-01 1:19:59 25 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:19:59 26 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:19:59 25 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 1:20:03 37 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-02-01 1:20:03 37 [Warning] Checking table: './wpstart/wp_options' 2020-02-01 1:20:07 49 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:20:07 49 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:20:07 50 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:20:07 50 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:20:07 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=6 max_threads=202 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9900000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f99b40d6cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5619303a4f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56192fe3b3af] sigaction.c:0(__restore_rt)[0x7f99c3558630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56192fd69151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56192fea7545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56192fc5dccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56192fc53d4b] /usr/sbin/mysqld(+0x6ca792)[0x56192fc5e792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56192fc866da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56192fc8865d] /usr/sbin/mysqld(+0x65e3f3)[0x56192fbf23f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56192fbf1cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56192fc85b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56192fc8865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56192fc88761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56192fc89377] /usr/sbin/mysqld(+0x58fd84)[0x56192fb23d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56192fc30b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56192fc48b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56192fc48cd2] /usr/sbin/mysqld(+0x6b55e7)[0x56192fc495e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56192fc49677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56192fc3856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56192fc3a07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56192fd15afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56192fd15bdd] pthread_create.c:0(start_thread)[0x7f99c3550ea5] /lib64/libc.so.6(clone+0x6d)[0x7f99c18f18cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f990000fe80): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 52 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:20:14 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:20:14 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:20:14 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:20:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:20:14 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:20:14 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:20:14 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:20:14 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:20:14 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:20:14 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137352602 2020-02-01 1:20:14 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:20:14 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:20:14 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:20:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:20:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:20:14 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:20:14 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:20:15 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137352611; transaction id 5904886 2020-02-01 1:20:15 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:20:15 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:20:15 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:20:15 0 [Note] Starting crash recovery... 2020-02-01 1:20:15 0 [Note] Crash recovery finished. 2020-02-01 1:20:15 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:20:15 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:20:15 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:20:15 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000144' at position 346 2020-02-01 1:20:15 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:20:15 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:20:15 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:20:15 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:20:15 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:20:15 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:20:15 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:20:15 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:20:15 2020-02-01 1:20:17 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000144' at position 346 2020-02-01 1:20:24 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:20:24 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000146' at position 113239 2020-02-01 1:20:25 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:20:31 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:20:43 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:20:46 15 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 1:20:46 15 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 1:20:51 19 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:20:51 19 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:20:51 20 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:20:51 20 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:20:51 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=5 max_threads=202 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f6d9c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f6e54076cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c2c920df2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c2c8ca43af] sigaction.c:0(__restore_rt)[0x7f6e638f8630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c2c8bd2151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c2c8d10545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c2c8ac6ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c2c8abcd4b] /usr/sbin/mysqld(+0x6ca792)[0x55c2c8ac7792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c2c8aef6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c2c8af165d] /usr/sbin/mysqld(+0x65e3f3)[0x55c2c8a5b3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c2c8a5acb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c2c8aeeb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c2c8af165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c2c8af1761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c2c8af2377] /usr/sbin/mysqld(+0x58fd84)[0x55c2c898cd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c2c8a99b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c2c8ab1b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c2c8ab1cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c2c8ab25e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c2c8ab2677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c2c8aa156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c2c8aa307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c2c8b7eafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c2c8b7ebdd] pthread_create.c:0(start_thread)[0x7f6e638f0ea5] /lib64/libc.so.6(clone+0x6d)[0x7f6e61c918cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f6d9c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 21 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:20:57 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:20:57 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:20:57 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:20:57 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:20:57 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:20:57 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:20:57 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:20:57 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:20:57 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:20:57 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137352954 2020-02-01 1:20:57 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:20:57 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:20:57 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:20:57 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:20:57 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:20:57 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:20:57 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:20:58 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137352963; transaction id 5904891 2020-02-01 1:20:58 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:20:58 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:20:58 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:20:58 0 [Note] Starting crash recovery... 2020-02-01 1:20:58 0 [Note] Crash recovery finished. 2020-02-01 1:20:58 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:20:58 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:20:58 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:20:58 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000146' at position 113239 2020-02-01 1:20:58 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:20:58 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:20:58 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:20:58 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:20:58 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:20:58 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:20:58 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:20:58 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000146' at position 113239 2020-02-01 1:20:59 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:20:59 2020-02-01 1:21:14 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:21:14 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000147' at position 150851 2020-02-01 1:21:14 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:21:20 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:21:24 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:21:34 23 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:21:34 23 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:21:34 24 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:21:34 24 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:21:34 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=3 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f78a4000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f7958153cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x555b8dc25f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x555b8d6bc3af] sigaction.c:0(__restore_rt)[0x7f79678f4630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x555b8d5ea151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x555b8d728545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x555b8d4deccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x555b8d4d4d4b] /usr/sbin/mysqld(+0x6ca792)[0x555b8d4df792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x555b8d5076da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x555b8d50965d] /usr/sbin/mysqld(+0x65e3f3)[0x555b8d4733f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x555b8d472cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x555b8d506b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x555b8d50965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x555b8d509761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x555b8d50a377] /usr/sbin/mysqld(+0x58fd84)[0x555b8d3a4d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x555b8d4b1b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x555b8d4c9b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x555b8d4c9cd2] /usr/sbin/mysqld(+0x6b55e7)[0x555b8d4ca5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x555b8d4ca677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x555b8d4b956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x555b8d4bb07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x555b8d596afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x555b8d596bdd] pthread_create.c:0(start_thread)[0x7f79678ecea5] /lib64/libc.so.6(clone+0x6d)[0x7f7965c8d8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f78a400feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 25 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:21:40 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:21:40 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:21:40 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:21:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:21:40 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:21:40 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:21:40 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:21:40 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:21:40 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:21:40 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137356634 2020-02-01 1:21:40 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:21:40 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:21:40 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:21:40 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:21:40 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:21:40 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:21:40 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:21:40 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137356643; transaction id 5904902 2020-02-01 1:21:40 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:21:40 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:21:40 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:21:40 0 [Note] Starting crash recovery... 2020-02-01 1:21:40 0 [Note] Crash recovery finished. 2020-02-01 1:21:40 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:21:40 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:21:40 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:21:40 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000147' at position 150851 2020-02-01 1:21:40 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:21:40 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:21:40 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 1:21:40 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 1:21:40 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 1:21:40 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 1:21:40 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:21:40 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:21:40 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 1:21:40 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:21:40 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:21:40 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:21:41 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:21:41 2020-02-01 1:21:41 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000147' at position 150851 2020-02-01 1:21:53 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:21:53 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000148' at position 1464 2020-02-01 1:21:53 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:21:58 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:22:09 18 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:22:09 19 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:22:09 19 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:22:17 22 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:22:17 22 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:22:17 23 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:22:17 23 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:22:17 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=3 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe20c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe21a6a4cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x564cab102f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x564caab993af] sigaction.c:0(__restore_rt)[0x7fe2c6a2d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564caaac7151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564caac05545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564caa9bbccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x564caa9b1d4b] /usr/sbin/mysqld(+0x6ca792)[0x564caa9bc792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x564caa9e46da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564caa9e665d] /usr/sbin/mysqld(+0x65e3f3)[0x564caa9503f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x564caa94fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x564caa9e3b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x564caa9e665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x564caa9e6761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x564caa9e7377] /usr/sbin/mysqld(+0x58fd84)[0x564caa881d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x564caa98eb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x564caa9a6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x564caa9a6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x564caa9a75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x564caa9a7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x564caa99656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x564caa99807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x564caaa73afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x564caaa73bdd] pthread_create.c:0(start_thread)[0x7fe2c6a25ea5] /lib64/libc.so.6(clone+0x6d)[0x7fe2c4dc68cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe20c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 24 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:22:23 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:22:23 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:22:23 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:22:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:22:23 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:22:23 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:22:23 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:22:23 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:22:23 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:22:23 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137360287 2020-02-01 1:22:23 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:22:23 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:22:23 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:22:23 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:22:23 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:22:23 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:22:23 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:22:23 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137360296; transaction id 5904913 2020-02-01 1:22:23 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:22:23 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:22:23 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:22:23 0 [Note] Starting crash recovery... 2020-02-01 1:22:23 0 [Note] Crash recovery finished. 2020-02-01 1:22:23 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:22:23 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:22:23 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:22:23 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000148' at position 1464 2020-02-01 1:22:23 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:22:23 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:22:23 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:22:23 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:22:23 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:22:23 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:22:23 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:22:23 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000148' at position 1464 2020-02-01 1:22:24 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:22:24 2020-02-01 1:22:27 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:22:27 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000149' at position 1464 2020-02-01 1:22:27 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:22:33 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:22:43 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown 2020-02-01 1:22:43 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-02-01 1:22:43 0 [Note] InnoDB: FTS optimize thread exiting. 2020-02-01 1:22:43 10 [Note] Slave I/O thread killed during or after a reconnect done to recover from failed read 2020-02-01 1:22:43 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000149', position 1464 2020-02-01 1:22:43 0 [Note] InnoDB: Starting shutdown... 2020-02-01 1:22:43 0 [Note] InnoDB: Dumping buffer pool(s) to /data/mysql/ib_buffer_pool 2020-02-01 1:22:43 0 [Note] InnoDB: Buffer pool(s) dump completed at 200201 1:22:43 2020-02-01 1:22:44 0 [Note] InnoDB: Shutdown completed; log sequence number 5137363158; transaction id 5904922 2020-02-01 1:22:44 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:22:44 0 [Note] /usr/sbin/mysqld: Shutdown complete 2020-02-01 1:22:45 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:22:45 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:22:45 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:22:45 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:22:45 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:22:45 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:22:45 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:22:45 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:22:45 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:22:45 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:22:45 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:22:45 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:22:45 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:22:45 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:22:45 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137363158; transaction id 5904924 2020-02-01 1:22:45 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:22:45 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:22:45 0 [ERROR] /usr/sbin/mysqld: unknown variable 'ax_allowed_packet=128M' 2020-02-01 1:22:45 0 [ERROR] Aborting 2020-02-01 1:23:02 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:23:02 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:23:02 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:23:02 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:23:02 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:23:02 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:23:02 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:23:02 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:23:02 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:23:02 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:23:02 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:23:02 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:23:02 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:23:02 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:23:02 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137363167; transaction id 5904924 2020-02-01 1:23:02 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:23:02 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:23:02 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:23:02 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:23:02 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:23:02 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000149' at position 1464 2020-02-01 1:23:02 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:23:02 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:23:02 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:23:02 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:23:02 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:23:02 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:23:02 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:23:02 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000149' at position 1464 2020-02-01 1:23:02 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:23:02 2020-02-01 1:23:03 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:23:03 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000150' at position 1464 2020-02-01 1:23:03 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:23:09 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:23:12 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:23:19 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:23:21 15 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-02-01 1:23:21 15 [Warning] Checking table: './wpcomm303/wp_options' 2020-02-01 1:23:23 17 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:23:23 17 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:23:24 16 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:23:24 16 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 1:23:24 20 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-02-01 1:23:24 20 [Warning] Checking table: './shockeripe/wp_options' 2020-02-01 1:23:24 22 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:23:24 22 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:23:25 23 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:23:25 23 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:23:25 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=6 max_threads=202 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0f04000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0f9c084cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5576a53bef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5576a4e553af] sigaction.c:0(__restore_rt)[0x7f0fbeb58630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5576a4d83151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5576a4ec1545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5576a4c77ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5576a4c6dd4b] /usr/sbin/mysqld(+0x6ca792)[0x5576a4c78792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5576a4ca06da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5576a4ca265d] /usr/sbin/mysqld(+0x65e3f3)[0x5576a4c0c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5576a4c0bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5576a4c9fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5576a4ca265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5576a4ca2761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5576a4ca3377] /usr/sbin/mysqld(+0x58fd84)[0x5576a4b3dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5576a4c4ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5576a4c62b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5576a4c62cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5576a4c635e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5576a4c63677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5576a4c5256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5576a4c5407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5576a4d2fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5576a4d2fbdd] pthread_create.c:0(start_thread)[0x7f0fbeb50ea5] /lib64/libc.so.6(clone+0x6d)[0x7f0fbcef18cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f0f0400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 25 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:23:31 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:23:31 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:23:31 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:23:31 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:23:31 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:23:31 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:23:31 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:23:31 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:23:31 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:23:31 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137366030 2020-02-01 1:23:31 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:23:31 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:23:31 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:23:31 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:23:31 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:23:31 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:23:31 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:23:31 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137366039; transaction id 5904935 2020-02-01 1:23:31 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:23:31 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:23:31 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:23:31 0 [Note] Starting crash recovery... 2020-02-01 1:23:31 0 [Note] Crash recovery finished. 2020-02-01 1:23:31 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:23:31 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:23:31 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:23:31 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000150' at position 1464 2020-02-01 1:23:31 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:23:31 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:23:31 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:23:31 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:23:31 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:23:31 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:23:31 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:23:31 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000150' at position 1464 2020-02-01 1:23:32 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:23:32 2020-02-01 1:23:38 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:23:38 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000152' at position 3879 2020-02-01 1:23:38 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:23:38 12 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-02-01 1:23:43 13 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-02-01 1:23:45 14 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:24:00 15 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:24:06 19 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:24:06 19 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:24:09 21 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:24:09 21 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:24:13 24 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:24:13 24 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 1:24:22 32 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:24:22 32 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:24:22 33 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:24:22 33 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:24:22 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f3dc8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f3e60063cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5592c0324f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5592bfdbb3af] sigaction.c:0(__restore_rt)[0x7f3e82876630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5592bfce9151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5592bfe27545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5592bfbddccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5592bfbd3d4b] /usr/sbin/mysqld(+0x6ca792)[0x5592bfbde792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5592bfc066da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5592bfc0865d] /usr/sbin/mysqld(+0x65e3f3)[0x5592bfb723f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5592bfb71cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5592bfc05b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5592bfc0865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5592bfc08761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5592bfc09377] /usr/sbin/mysqld(+0x58fd84)[0x5592bfaa3d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5592bfbb0b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5592bfbc8b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5592bfbc8cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5592bfbc95e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5592bfbc9677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5592bfbb856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5592bfbba07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5592bfc95afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5592bfc95bdd] pthread_create.c:0(start_thread)[0x7f3e8286eea5] /lib64/libc.so.6(clone+0x6d)[0x7f3e80c0f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f3dc800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 34 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:24:29 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:24:29 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:24:29 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:24:29 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:24:29 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:24:29 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:24:29 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:24:29 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:24:29 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:24:29 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137368877 2020-02-01 1:24:29 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:24:29 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:24:29 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:24:29 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:24:29 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:24:29 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:24:29 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:24:29 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137368886; transaction id 5904946 2020-02-01 1:24:29 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:24:29 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:24:29 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:24:29 0 [Note] Starting crash recovery... 2020-02-01 1:24:29 0 [Note] Crash recovery finished. 2020-02-01 1:24:29 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:24:29 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:24:29 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:24:29 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000152' at position 3879 2020-02-01 1:24:29 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:24:29 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:24:29 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:24:29 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:24:29 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:24:29 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:24:29 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:24:29 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000152' at position 3879 2020-02-01 1:24:30 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:24:30 2020-02-01 1:24:36 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:24:36 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000153' at position 1464 2020-02-01 1:24:36 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:24:47 14 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:24:55 16 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:24:56 19 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:24:56 19 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:24:59 25 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:24:59 25 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:24:59 26 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:24:59 26 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:24:59 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=5 max_threads=202 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549711 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f695c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f6a14078cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55f46882af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55f4682c13af] sigaction.c:0(__restore_rt)[0x7f6a224c0630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55f4681ef151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55f46832d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55f4680e3ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55f4680d9d4b] /usr/sbin/mysqld(+0x6ca792)[0x55f4680e4792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55f46810c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55f46810e65d] /usr/sbin/mysqld(+0x65e3f3)[0x55f4680783f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55f468077cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55f46810bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55f46810e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55f46810e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55f46810f377] /usr/sbin/mysqld(+0x58fd84)[0x55f467fa9d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55f4680b6b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55f4680ceb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55f4680cecd2] /usr/sbin/mysqld(+0x6b55e7)[0x55f4680cf5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55f4680cf677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55f4680be56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55f4680c007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55f46819bafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55f46819bbdd] pthread_create.c:0(start_thread)[0x7f6a224b8ea5] /lib64/libc.so.6(clone+0x6d)[0x7f6a208598cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f695c0121f0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 27 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:25:05 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:25:05 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:25:05 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:25:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:25:05 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:25:05 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:25:05 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:25:05 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:25:05 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:25:05 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137371740 2020-02-01 1:25:05 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:25:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:25:05 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:25:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:25:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:25:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:25:05 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:25:05 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137371749; transaction id 5904957 2020-02-01 1:25:05 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:25:05 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:25:05 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:25:05 0 [Note] Starting crash recovery... 2020-02-01 1:25:05 0 [Note] Crash recovery finished. 2020-02-01 1:25:05 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:25:05 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:25:05 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:25:05 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000153' at position 1464 2020-02-01 1:25:05 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:25:05 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:25:05 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:25:05 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:25:05 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:25:05 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:25:05 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:25:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:25:05 2020-02-01 1:25:05 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000153' at position 1464 2020-02-01 1:25:30 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:25:33 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown 2020-02-01 1:25:33 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-02-01 1:25:33 0 [Note] InnoDB: FTS optimize thread exiting. 2020-02-01 1:25:33 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000154', position 197778 2020-02-01 1:25:33 0 [Note] InnoDB: Starting shutdown... 2020-02-01 1:25:33 0 [Note] InnoDB: Dumping buffer pool(s) to /data/mysql/ib_buffer_pool 2020-02-01 1:25:33 0 [Note] InnoDB: Buffer pool(s) dump completed at 200201 1:25:33 2020-02-01 1:25:34 0 [Note] InnoDB: Shutdown completed; log sequence number 5137374596; transaction id 5904966 2020-02-01 1:25:34 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:25:34 0 [Note] /usr/sbin/mysqld: Shutdown complete 2020-02-01 1:25:34 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:25:34 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:25:34 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:25:34 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:25:34 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:25:34 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:25:34 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:25:34 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:25:34 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:25:34 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:25:34 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:25:34 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:25:34 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:25:34 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:25:34 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137374596; transaction id 5904968 2020-02-01 1:25:34 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:25:34 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:25:34 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:25:35 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:25:35 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:25:35 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000154' at position 197778 2020-02-01 1:25:35 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:25:35 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:25:35 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000154' at position 197778 2020-02-01 1:25:35 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:25:35 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:25:35 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:25:35 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:25:35 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:25:35 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:25:35 2020-02-01 1:25:35 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:25:36 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:25:36 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:25:36 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:25:36 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:25:37 15 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 1:25:37 15 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 1:25:37 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f6fc0000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f70641f0cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c12a575f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c12a00c3af] sigaction.c:0(__restore_rt)[0x7f707f30b630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c129f3a151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c12a078545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c129e2eccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c129e24d4b] /usr/sbin/mysqld(+0x6ca792)[0x55c129e2f792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c129e576da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c129e5965d] /usr/sbin/mysqld(+0x65e3f3)[0x55c129dc33f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c129dc2cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c129e56b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c129e5965d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c129e59761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c129e5a377] /usr/sbin/mysqld(+0x58fd84)[0x55c129cf4d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c129e01b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c129e19b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c129e19cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c129e1a5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c129e1a677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c129e0956d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c129e0b07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c129ee6afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c129ee6bdd] pthread_create.c:0(start_thread)[0x7f707f303ea5] /lib64/libc.so.6(clone+0x6d)[0x7f707d6a48cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f6fc000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:25:43 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:25:43 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:25:43 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:25:43 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:25:43 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:25:43 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:25:43 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:25:43 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:25:43 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:25:43 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137377200 2020-02-01 1:25:43 0 [Note] InnoDB: Starting final batch to recover 3 pages from redo log. 2020-02-01 1:25:44 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:25:44 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:25:44 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:25:44 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:25:44 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:25:44 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:25:44 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:25:44 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137377333; transaction id 5904979 2020-02-01 1:25:44 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:25:44 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:25:44 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:25:44 0 [Note] Starting crash recovery... 2020-02-01 1:25:44 0 [Note] Crash recovery finished. 2020-02-01 1:25:44 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:25:44 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:25:44 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:25:44 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000154' at position 197778 2020-02-01 1:25:44 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:25:44 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:25:44 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000154' at position 197778 2020-02-01 1:25:44 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:25:44 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:25:44 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:25:44 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:25:44 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:25:44 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:25:44 2020-02-01 1:25:56 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:25:56 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000154' at position 330429 2020-02-01 1:25:56 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:26:02 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:26:10 15 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:26:11 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:26:11 17 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:26:11 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:26:11 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:26:11 18 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 1:26:11 18 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 1:26:11 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=202 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe684000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe71941ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560d541d4f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560d53c6b3af] sigaction.c:0(__restore_rt)[0x7fe7356cd630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x560d53b99151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x560d53cd7545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x560d53a8dccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x560d53a83d4b] /usr/sbin/mysqld(+0x6ca792)[0x560d53a8e792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x560d53ab66da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560d53ab865d] /usr/sbin/mysqld(+0x65e3f3)[0x560d53a223f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x560d53a21cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x560d53ab5b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560d53ab865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x560d53ab8761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x560d53ab9377] /usr/sbin/mysqld(+0x58fd84)[0x560d53953d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x560d53a60b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x560d53a78b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x560d53a78cd2] /usr/sbin/mysqld(+0x6b55e7)[0x560d53a795e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x560d53a79677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x560d53a6856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x560d53a6a07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x560d53b45afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560d53b45bdd] pthread_create.c:0(start_thread)[0x7fe7356c5ea5] /lib64/libc.so.6(clone+0x6d)[0x7fe733a668cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe68400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:26:17 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:26:17 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:26:17 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:26:17 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:26:17 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:26:17 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:26:17 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:26:18 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:26:18 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:26:18 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137380169 2020-02-01 1:26:18 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:26:18 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:26:18 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:26:18 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:26:18 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:26:18 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:26:18 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:26:18 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137380178; transaction id 5904990 2020-02-01 1:26:18 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:26:18 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:26:18 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:26:18 0 [Note] Starting crash recovery... 2020-02-01 1:26:18 0 [Note] Crash recovery finished. 2020-02-01 1:26:18 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:26:18 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:26:18 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:26:18 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000154' at position 330429 2020-02-01 1:26:18 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:26:18 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:26:18 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:26:18 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:26:18 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:26:18 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:26:18 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:26:18 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000154' at position 330429 2020-02-01 1:26:19 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:26:19 2020-02-01 1:26:37 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:26:37 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000155' at position 350554 2020-02-01 1:26:37 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:26:44 14 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:26:46 15 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:26:46 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:26:46 16 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:26:46 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:26:46 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:26:47 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff23c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff253ffacf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b2cec79f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55b2ce7103af] sigaction.c:0(__restore_rt)[0x7ff2f9592630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55b2ce63e151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55b2ce77c545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55b2ce532ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55b2ce528d4b] /usr/sbin/mysqld(+0x6ca792)[0x55b2ce533792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55b2ce55b6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b2ce55d65d] /usr/sbin/mysqld(+0x65e3f3)[0x55b2ce4c73f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55b2ce4c6cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55b2ce55ab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b2ce55d65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55b2ce55d761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55b2ce55e377] /usr/sbin/mysqld(+0x58fd84)[0x55b2ce3f8d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55b2ce505b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55b2ce51db16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55b2ce51dcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55b2ce51e5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55b2ce51e677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55b2ce50d56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55b2ce50f07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55b2ce5eaafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55b2ce5eabdd] pthread_create.c:0(start_thread)[0x7ff2f958aea5] /lib64/libc.so.6(clone+0x6d)[0x7ff2f792b8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff23c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:26:53 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:26:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:26:53 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:26:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:26:53 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:26:53 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:26:53 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:26:53 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:26:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:26:53 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137383032 2020-02-01 1:26:53 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:26:53 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:26:53 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:26:53 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:26:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:26:53 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:26:53 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:26:53 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137383041; transaction id 5905001 2020-02-01 1:26:53 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:26:53 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:26:53 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:26:53 0 [Note] Starting crash recovery... 2020-02-01 1:26:53 0 [Note] Crash recovery finished. 2020-02-01 1:26:53 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:26:53 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:26:53 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:26:53 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000155' at position 350554 2020-02-01 1:26:53 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:26:53 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:26:53 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:26:53 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:26:53 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:26:53 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:26:53 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:26:53 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000155' at position 350554 2020-02-01 1:26:54 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:26:54 2020-02-01 1:27:17 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:27:17 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000156' at position 44093 2020-02-01 1:27:17 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:27:21 18 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:27:21 19 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:27:21 19 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:27:22 20 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:27:25 21 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:27:30 23 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-02-01 1:27:30 23 [Warning] Checking table: './wpsem/wp_options' 2020-02-01 1:27:32 27 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:27:32 27 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:27:32 28 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:27:32 28 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:27:32 28 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 1:27:32 28 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 1:27:32 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fbe00000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fbe94f68cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56048d5e1f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56048d0783af] sigaction.c:0(__restore_rt)[0x7fbeb11df630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56048cfa6151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56048d0e4545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56048ce9accf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56048ce90d4b] /usr/sbin/mysqld(+0x6ca792)[0x56048ce9b792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56048cec36da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56048cec565d] /usr/sbin/mysqld(+0x65e3f3)[0x56048ce2f3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56048ce2ecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56048cec2b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56048cec565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56048cec5761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56048cec6377] /usr/sbin/mysqld(+0x58fd84)[0x56048cd60d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56048ce6db6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56048ce85b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56048ce85cd2] /usr/sbin/mysqld(+0x6b55e7)[0x56048ce865e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56048ce86677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56048ce7556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56048ce7707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56048cf52afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56048cf52bdd] pthread_create.c:0(start_thread)[0x7fbeb11d7ea5] /lib64/libc.so.6(clone+0x6d)[0x7fbeaf5788cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fbe0000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 29 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:27:38 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:27:38 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:27:38 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:27:38 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:27:38 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:27:38 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:27:38 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:27:38 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:27:38 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:27:38 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137385846 2020-02-01 1:27:38 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:27:38 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:27:38 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:27:38 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:27:38 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:27:38 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:27:38 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:27:38 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137385855; transaction id 5905012 2020-02-01 1:27:38 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:27:38 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:27:38 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:27:38 0 [Note] Starting crash recovery... 2020-02-01 1:27:38 0 [Note] Crash recovery finished. 2020-02-01 1:27:38 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:27:38 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:27:38 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:27:38 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000156' at position 44093 2020-02-01 1:27:38 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:27:38 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:27:38 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000156' at position 44093 2020-02-01 1:27:38 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:27:38 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:27:38 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:27:38 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:27:38 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:27:39 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:27:39 2020-02-01 1:27:56 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:27:56 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000158' at position 245623 2020-02-01 1:27:56 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:28:02 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:28:04 10 [Note] Slave I/O thread killed during or after a reconnect done to recover from failed read 2020-02-01 1:28:04 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000158', position 245623 2020-02-01 1:28:06 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:28:07 17 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:28:07 17 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:28:09 20 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-02-01 1:28:09 20 [Warning] Checking table: './pizzahut/wp_options' 2020-02-01 1:28:10 23 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:28:10 23 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:28:24 33 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:28:24 33 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:28:24 34 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:28:24 34 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:28:24 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fdc8c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fdc90316cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x562cb9b27f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x562cb95be3af] sigaction.c:0(__restore_rt)[0x7fdd3da3d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x562cb94ec151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x562cb962a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x562cb93e0ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x562cb93d6d4b] /usr/sbin/mysqld(+0x6ca792)[0x562cb93e1792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x562cb94096da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x562cb940b65d] /usr/sbin/mysqld(+0x65e3f3)[0x562cb93753f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x562cb9374cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x562cb9408b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x562cb940b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x562cb940b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x562cb940c377] /usr/sbin/mysqld(+0x58fd84)[0x562cb92a6d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x562cb93b3b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x562cb93cbb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x562cb93cbcd2] /usr/sbin/mysqld(+0x6b55e7)[0x562cb93cc5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x562cb93cc677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x562cb93bb56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x562cb93bd07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x562cb9498afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x562cb9498bdd] pthread_create.c:0(start_thread)[0x7fdd3da35ea5] /lib64/libc.so.6(clone+0x6d)[0x7fdd3bdd68cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fdc8c00fe80): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 36 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:28:30 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:28:30 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:28:30 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:28:30 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:28:30 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:28:30 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:28:30 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:28:30 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:28:30 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:28:30 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137389239 2020-02-01 1:28:30 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:28:30 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:28:30 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:28:30 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:28:30 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:28:30 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:28:30 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:28:30 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137389248; transaction id 5905030 2020-02-01 1:28:30 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:28:30 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:28:30 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:28:30 0 [Note] Starting crash recovery... 2020-02-01 1:28:30 0 [Note] Crash recovery finished. 2020-02-01 1:28:30 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:28:30 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:28:30 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:28:30 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000158' at position 245623 2020-02-01 1:28:30 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:28:30 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:28:30 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 1:28:30 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 1:28:30 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 1:28:30 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000158' at position 245623 2020-02-01 1:28:30 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 1:28:30 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:28:30 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:28:30 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 1:28:30 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:28:30 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:28:30 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:28:31 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:28:31 2020-02-01 1:28:36 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:28:36 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000159' at position 68871 2020-02-01 1:28:36 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:28:42 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:28:56 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:28:57 15 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:29:00 16 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 1:29:09 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:29:12 22 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:29:12 23 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:29:13 24 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:29:22 27 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:29:36 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000159' at position 68871 2020-02-01 1:29:39 31 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-02-01 1:29:54 37 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:29:58 40 [ERROR] mysqld: Table './ulrich/wp_options' is marked as crashed and should be repaired 2020-02-01 1:30:01 43 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:30:10 46 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 1:30:10 47 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:30:12 48 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:30:32 56 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:30:40 58 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:30:42 62 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-02-01 1:31:07 71 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:31:10 72 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:31:20 77 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 1:31:26 81 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:31:42 87 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:31:52 92 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-02-01 1:31:56 94 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:32:12 105 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:32:26 111 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:32:30 113 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 1:32:37 118 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:32:37 119 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:32:37 120 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:32:37 121 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 1:32:37 122 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:32:37 15 [Warning] Aborted connection 15 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-02-01 1:32:37 20 [Warning] Aborted connection 20 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Table './ojsdata/sessions' is marked as crashed and should be re) 2020-02-01 1:32:37 24 [Warning] Aborted connection 24 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-02-01 1:32:37 23 [Warning] Aborted connection 23 to db: 'ksarn' user: 'ksarn' host: '10.0.59.19' (Table './ksarn/wp_options' is marked as crashed and should be re) 2020-02-01 1:32:38 16 [Warning] Aborted connection 16 to db: 'wpcomm306' user: 'wpcomm306' host: '10.0.59.19' (Table './wpcomm306/wp_options' is marked as crashed and should b) 2020-02-01 1:32:38 18 [Warning] Aborted connection 18 to db: 'wpcomm305' user: 'wpcomm305' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-02-01 1:32:38 22 [Warning] Aborted connection 22 to db: 'wpms' user: 'wpms' host: '10.0.59.19' (Table './wpms/wp_options' is marked as crashed and should be rep) 2020-02-01 1:32:38 27 [Warning] Aborted connection 27 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Table './ojsdata/sessions' is marked as crashed and should be re) 2020-02-01 1:32:47 128 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:32:52 133 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:32:56 134 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:33:02 139 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-02-01 1:33:05 37 [Warning] Aborted connection 37 to db: 'idtgroup' user: 'root' host: 'localhost' (Table './idtgroup/wp_usermeta' is marked as crashed and should b) 2020-02-01 1:33:09 144 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:33:12 145 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:33:17 148 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:33:26 153 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:33:47 160 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:33:47 40 [Warning] Checking table: './ulrich/wp_options' 2020-02-01 1:33:47 46 [Warning] Checking table: './wpcomm306/wp_options' 2020-02-01 1:33:47 48 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:33:47 144 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 1:33:47 133 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:33:47 139 [Warning] Checking table: './wpcomm303/wp_options' 2020-02-01 1:33:47 47 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:33:48 165 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:33:48 163 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:33:48 165 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:33:48 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=71 max_threads=202 thread_count=79 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fea48094138 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe997a22cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x564435ca7f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56443573e3af] sigaction.c:0(__restore_rt)[0x7fea53d03630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56443566c151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5644357aa545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564435560ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x564435556d4b] /usr/sbin/mysqld(+0x6ca792)[0x564435561792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5644355896da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56443558b65d] /usr/sbin/mysqld(+0x65e3f3)[0x5644354f53f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5644354f4cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x564435588b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56443558b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56443558b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56443558c377] /usr/sbin/mysqld(+0x58fd84)[0x564435426d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x564435533b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56443554bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56443554bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x56443554c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56443554c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56443553b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56443553d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x564435618afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x564435618bdd] pthread_create.c:0(start_thread)[0x7fea53cfbea5] /lib64/libc.so.6(clone+0x6d)[0x7fea5209c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fea480dd890): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 166 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:33:54 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:33:54 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:33:54 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:33:54 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:33:54 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:33:54 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:33:54 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:33:55 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:33:55 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:33:55 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137392893 2020-02-01 1:33:55 0 [Note] InnoDB: Starting final batch to recover 1 pages from redo log. 2020-02-01 1:33:55 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:33:55 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:33:55 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:33:55 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:33:55 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:33:55 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:33:55 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:33:55 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137393146; transaction id 5905101 2020-02-01 1:33:55 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:33:55 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:33:55 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:33:55 0 [Note] Starting crash recovery... 2020-02-01 1:33:55 0 [Note] Crash recovery finished. 2020-02-01 1:33:55 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:33:55 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:33:55 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:33:55 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000160' at position 346 2020-02-01 1:33:55 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:33:55 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:33:55 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000160' at position 346 2020-02-01 1:33:55 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 1:33:55 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 1:33:55 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 1:33:55 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 1:33:55 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:33:55 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:33:55 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 1:33:55 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:33:55 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:33:55 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:33:57 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:33:57 2020-02-01 1:34:19 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:34:19 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000160' at position 539399 2020-02-01 1:34:19 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:34:22 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:34:24 16 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-02-01 1:34:24 16 [Warning] Checking table: './openrecords/wp_options' 2020-02-01 1:34:25 18 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:34:44 28 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:34:44 28 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:34:44 29 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:34:44 29 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:34:44 29 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 1:34:44 29 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 1:34:44 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fe290000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe29e60ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x559e93781f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x559e932183af] sigaction.c:0(__restore_rt)[0x7fe342ca4630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x559e93146151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x559e93284545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x559e9303accf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x559e93030d4b] /usr/sbin/mysqld(+0x6ca792)[0x559e9303b792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x559e930636da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559e9306565d] /usr/sbin/mysqld(+0x65e3f3)[0x559e92fcf3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x559e92fcecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x559e93062b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x559e9306565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x559e93065761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x559e93066377] /usr/sbin/mysqld(+0x58fd84)[0x559e92f00d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x559e9300db6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x559e93025b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x559e93025cd2] /usr/sbin/mysqld(+0x6b55e7)[0x559e930265e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x559e93026677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x559e9301556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x559e9301707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x559e930f2afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x559e930f2bdd] pthread_create.c:0(start_thread)[0x7fe342c9cea5] /lib64/libc.so.6(clone+0x6d)[0x7fe34103d8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe29000fe80): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 30 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:34:50 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:34:50 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:34:50 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:34:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:34:50 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:34:50 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:34:50 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:34:50 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:34:50 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:34:50 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137396175 2020-02-01 1:34:50 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:34:50 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:34:50 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:34:50 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:34:50 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:34:50 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:34:50 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:34:50 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137396184; transaction id 5905112 2020-02-01 1:34:50 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:34:50 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:34:50 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:34:50 0 [Note] Starting crash recovery... 2020-02-01 1:34:50 0 [Note] Crash recovery finished. 2020-02-01 1:34:50 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:34:50 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:34:50 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:34:50 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000160' at position 539399 2020-02-01 1:34:50 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:34:50 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:34:50 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:34:50 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:34:50 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:34:50 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:34:50 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:34:50 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000160' at position 539399 2020-02-01 1:34:51 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:34:51 2020-02-01 1:35:06 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:35:06 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000161' at position 8411 2020-02-01 1:35:06 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:35:12 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:35:18 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:35:18 15 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 1:35:18 15 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 1:35:20 17 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-02-01 1:35:20 17 [Warning] Checking table: './wpstart/wp_options' 2020-02-01 1:35:40 40 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:35:40 40 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:35:40 41 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:35:40 41 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:35:41 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0460006bf8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f05101a5cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56305bf7ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56305ba163af] sigaction.c:0(__restore_rt)[0x7f052b228630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56305b944151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56305ba82545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56305b838ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56305b82ed4b] /usr/sbin/mysqld(+0x6ca792)[0x56305b839792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56305b8616da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56305b86365d] /usr/sbin/mysqld(+0x65e3f3)[0x56305b7cd3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56305b7cccb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56305b860b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56305b86365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56305b863761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56305b864377] /usr/sbin/mysqld(+0x58fd84)[0x56305b6fed84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56305b80bb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56305b823b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56305b823cd2] /usr/sbin/mysqld(+0x6b55e7)[0x56305b8245e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56305b824677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56305b81356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56305b81507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56305b8f0afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56305b8f0bdd] pthread_create.c:0(start_thread)[0x7f052b220ea5] /lib64/libc.so.6(clone+0x6d)[0x7f05295c18cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f0460010ea0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 42 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:35:47 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:35:47 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:35:47 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:35:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:35:47 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:35:47 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:35:47 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:35:47 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:35:47 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:35:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137410159 2020-02-01 1:35:47 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:35:47 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:35:47 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:35:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:35:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:35:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:35:47 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137410168; transaction id 5905123 2020-02-01 1:35:47 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:35:47 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:35:47 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:35:47 0 [Note] Starting crash recovery... 2020-02-01 1:35:47 0 [Note] Crash recovery finished. 2020-02-01 1:35:47 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:35:47 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:35:47 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:35:47 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000161' at position 8411 2020-02-01 1:35:47 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:35:47 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:35:47 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 1:35:47 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000161' at position 8411 2020-02-01 1:35:47 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 1:35:47 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 1:35:47 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 1:35:47 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:35:47 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:35:47 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 1:35:47 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:35:47 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:35:47 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:35:47 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:35:47 2020-02-01 1:35:57 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:35:57 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000162' at position 2513 2020-02-01 1:35:57 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:36:02 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:36:13 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:36:16 16 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 1:36:16 16 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 1:36:18 17 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:36:18 17 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 1:36:21 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:36:21 20 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:36:21 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:36:21 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:36:21 21 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 1:36:21 21 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 1:36:21 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f8594000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f863815fcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56344e813f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56344e2aa3af] sigaction.c:0(__restore_rt)[0x7f865349d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56344e1d8151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56344e316545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56344e0ccccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56344e0c2d4b] /usr/sbin/mysqld(+0x6ca792)[0x56344e0cd792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56344e0f56da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56344e0f765d] /usr/sbin/mysqld(+0x65e3f3)[0x56344e0613f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56344e060cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56344e0f4b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56344e0f765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56344e0f7761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56344e0f8377] /usr/sbin/mysqld(+0x58fd84)[0x56344df92d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56344e09fb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56344e0b7b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56344e0b7cd2] /usr/sbin/mysqld(+0x6b55e7)[0x56344e0b85e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56344e0b8677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56344e0a756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56344e0a907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56344e184afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56344e184bdd] pthread_create.c:0(start_thread)[0x7f8653495ea5] /lib64/libc.so.6(clone+0x6d)[0x7f86518368cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f859400fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:36:27 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:36:27 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:36:27 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:36:27 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:36:27 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:36:27 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:36:27 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:36:28 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:36:28 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:36:28 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137412978 2020-02-01 1:36:28 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:36:28 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:36:28 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:36:28 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:36:28 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:36:28 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:36:28 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:36:28 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137412987; transaction id 5905134 2020-02-01 1:36:28 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:36:28 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:36:28 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:36:28 0 [Note] Starting crash recovery... 2020-02-01 1:36:28 0 [Note] Crash recovery finished. 2020-02-01 1:36:28 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:36:28 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:36:28 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:36:28 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000162' at position 2513 2020-02-01 1:36:28 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:36:28 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:36:28 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:36:28 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:36:28 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:36:28 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:36:28 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:36:28 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000162' at position 2513 2020-02-01 1:36:29 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:36:29 2020-02-01 1:36:31 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:36:31 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000163' at position 1410 2020-02-01 1:36:31 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:36:36 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:36:51 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown 2020-02-01 1:36:51 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-02-01 1:36:51 0 [Note] InnoDB: FTS optimize thread exiting. 2020-02-01 1:36:51 10 [Note] Slave I/O thread killed during or after a reconnect done to recover from failed read 2020-02-01 1:36:51 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000163', position 1410 2020-02-01 1:36:51 0 [Note] InnoDB: Starting shutdown... 2020-02-01 1:36:51 0 [Note] InnoDB: Dumping buffer pool(s) to /data/mysql/ib_buffer_pool 2020-02-01 1:36:51 0 [Note] InnoDB: Buffer pool(s) dump completed at 200201 1:36:51 2020-02-01 1:36:52 0 [Note] InnoDB: Shutdown completed; log sequence number 5137415850; transaction id 5905143 2020-02-01 1:36:52 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:36:52 0 [Note] /usr/sbin/mysqld: Shutdown complete 2020-02-01 1:36:52 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:36:52 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:36:52 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:36:52 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:36:52 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:36:52 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:36:52 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:36:52 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:36:52 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:36:52 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:36:52 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:36:52 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:36:52 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:36:52 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:36:52 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137415850; transaction id 5905145 2020-02-01 1:36:52 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:36:52 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:36:52 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:36:52 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:36:52 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:36:52 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000163' at position 1410 2020-02-01 1:36:52 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:36:52 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:36:52 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:36:52 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:36:52 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:36:52 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:36:52 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:36:52 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000163' at position 1410 2020-02-01 1:36:53 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:36:53 2020-02-01 1:36:57 10 [Note] Slave: received end packet from server, apparent master shutdown: 2020-02-01 1:36:57 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000164' at position 346 2020-02-01 1:36:57 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:37:04 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000193, 1464) 2020-02-01 1:37:11 10 [Note] Slave I/O thread killed during or after a reconnect done to recover from failed read 2020-02-01 1:37:11 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000164', position 346 2020-02-01 1:37:46 19 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 1:37:46 19 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 1:37:46 19 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:37:46 19 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 1:37:49 19 [ERROR] mysqld: Table './itstraining/wp_options' is marked as crashed and should be repaired 2020-02-01 1:37:49 19 [Warning] Checking table: './itstraining/wp_options' 2020-02-01 1:37:51 19 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:37:51 19 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:37:53 19 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 1:37:53 19 [Warning] Checking table: './ojsdata/controlled_vocabs' 2020-02-01 1:37:54 19 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:37:54 19 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:37:54 19 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:37:54 19 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:37:56 19 [ERROR] mysqld: Table './openrecords/wp_options' is marked as crashed and should be repaired 2020-02-01 1:37:56 19 [Warning] Checking table: './openrecords/wp_options' 2020-02-01 1:37:56 19 [ERROR] mysqld: Table './pizzahut/wp_options' is marked as crashed and should be repaired 2020-02-01 1:37:56 19 [Warning] Checking table: './pizzahut/wp_options' 2020-02-01 1:38:08 19 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-02-01 1:38:08 19 [Warning] Checking table: './shockeripe/wp_options' 2020-02-01 1:38:09 19 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-02-01 1:38:09 19 [Warning] Checking table: './wpcomm305/comm305_options' 2020-02-01 1:38:11 19 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:38:11 19 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:38:11 19 [ERROR] mysqld: Table './wpms/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:38:11 19 [Warning] Checking table: './wpms/wp_usermeta' 2020-02-01 1:38:13 19 [ERROR] mysqld: Table './wpsem/wp_options' is marked as crashed and should be repaired 2020-02-01 1:38:13 19 [Warning] Checking table: './wpsem/wp_options' 2020-02-01 1:38:13 19 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-02-01 1:38:13 19 [Warning] Checking table: './wpstart/wp_options' 2020-02-01 1:38:14 19 [ERROR] mysqld: Table './wpwsutv/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:38:14 19 [Warning] Checking table: './wpwsutv/wp_usermeta' 2020-02-01 1:43:03 30 [Note] Start binlog_dump to slave_server(1), pos(, 4) 2020-02-01 1:43:28 32 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 200201 1:43:33 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=202 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc914000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc922058cf0 thread_stack 0x49000 *** buffer overflow detected ***: /usr/sbin/mysqld terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7fc9c48ea567] /lib64/libc.so.6(+0x1166e2)[0x7fc9c48e86e2] /lib64/libc.so.6(+0x1184c7)[0x7fc9c48ea4c7] /usr/sbin/mysqld(my_addr_resolve+0xda)[0x556eebb0994a] /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x556eebaf20c2] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x556eeb5883af] /lib64/libpthread.so.0(+0xf630)[0x7fc9c6537630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x556eeb4b6151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x556eeb5f4545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x556eeb3aaccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x556eeb3a0d4b] /usr/sbin/mysqld(+0x6ca792)[0x556eeb3ab792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x556eeb3d36da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556eeb3d565d] /usr/sbin/mysqld(+0x65e3f3)[0x556eeb33f3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x556eeb33ecb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x556eeb3d2b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x556eeb3d565d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x556eeb3d5761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x556eeb3d6377] /usr/sbin/mysqld(+0x58fd84)[0x556eeb270d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x556eeb37db6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x556eeb395b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x556eeb395cd2] /usr/sbin/mysqld(+0x6b55e7)[0x556eeb3965e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x556eeb396677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x556eeb38556d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x556eeb38707c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x556eeb462afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x556eeb462bdd] /lib64/libpthread.so.0(+0x7ea5)[0x7fc9c652fea5] /lib64/libc.so.6(clone+0x6d)[0x7fc9c48d08cd] ======= Memory map: ======== 556eeace1000-556eec12f000 r-xp 00000000 fd:00 2101454 /usr/sbin/mysqld 556eec32e000-556eec45a000 r--p 0144d000 fd:00 2101454 /usr/sbin/mysqld 556eec45a000-556eec514000 rw-p 01579000 fd:00 2101454 /usr/sbin/mysqld 556eec514000-556eecdc3000 rw-p 00000000 00:00 0 556eed006000-556ef06dd000 rw-p 00000000 00:00 0 [heap] 7fc8deffe000-7fc8defff000 ---p 00000000 00:00 0 7fc8defff000-7fc8df7ff000 rw-p 00000000 00:00 0 7fc8df7ff000-7fc8df800000 ---p 00000000 00:00 0 7fc8df800000-7fc8e0000000 rw-p 00000000 00:00 0 7fc8e0000000-7fc8e0022000 rw-p 00000000 00:00 0 7fc8e0022000-7fc8e4000000 ---p 00000000 00:00 0 7fc8e4000000-7fc8e4029000 rw-p 00000000 00:00 0 7fc8e4029000-7fc8e8000000 ---p 00000000 00:00 0 7fc8e8000000-7fc8e8021000 rw-p 00000000 00:00 0 7fc8e8021000-7fc8ec000000 ---p 00000000 00:00 0 7fc8ec000000-7fc8ec123000 rw-p 00000000 00:00 0 7fc8ec123000-7fc8f0000000 ---p 00000000 00:00 0 7fc8f0000000-7fc8f0028000 rw-p 00000000 00:00 0 7fc8f0028000-7fc8f4000000 ---p 00000000 00:00 0 7fc8f4000000-7fc8f4043000 rw-p 00000000 00:00 0 7fc8f4043000-7fc8f8000000 ---p 00000000 00:00 0 7fc8f8000000-7fc8f8021000 rw-p 00000000 00:00 0 7fc8f8021000-7fc8fc000000 ---p 00000000 00:00 0 7fc8fc000000-7fc8fc040000 rw-p 00000000 00:00 0 7fc8fc040000-7fc900000000 ---p 00000000 00:00 0 7fc900000000-7fc903ff9000 rw-p 00000000 00:00 0 7fc903ff9000-7fc904000000 ---p 00000000 00:00 0 7fc904000000-7fc9044f1000 rw-p 00000000 00:00 0 7fc9044f1000-7fc908000000 ---p 00000000 00:00 0 7fc908000000-7fc908037000 rw-p 00000000 00:00 0 7fc908037000-7fc90c000000 ---p 00000000 00:00 0 7fc90c000000-7fc90c36e000 rw-p 00000000 00:00 0 7fc90c36e000-7fc910000000 ---p 00000000 00:00 0 7fc9103dd000-7fc9103ef000 r-xp 00000000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7fc9103ef000-7fc9105ee000 ---p 00012000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7fc9105ee000-7fc9105f1000 r--p 00011000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7fc9105f1000-7fc9105f2000 rw-p 00014000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7fc9105f2000-7fc9105f8000 r-xp 00000000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7fc9105f8000-7fc9107f7000 ---p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7fc9107f7000-7fc9107f8000 r--p 00005000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7fc9107f8000-7fc9107f9000 rw-p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7fc9107f9000-7fc9107fa000 ---p 00000000 00:00 0 7fc9107fa000-7fc910ffa000 rw-p 00000000 00:00 0 7fc910ffa000-7fc910ffb000 ---p 00000000 00:00 0 7fc910ffb000-7fc9117fb000 rw-p 00000000 00:00 0 7fc9117fb000-7fc9117fc000 ---p 00000000 00:00 0 7fc9117fc000-7fc911ffc000 rw-p 00000000 00:00 0 7fc911ffc000-7fc911ffd000 ---p 00000000 00:00 0 7fc911ffd000-7fc9127fd000 rw-p 00000000 00:00 0 7fc9127fd000-7fc9127fe000 ---p 00000000 00:00 0 7fc9127fe000-7fc912ffe000 rw-p 00000000 00:00 0 7fc912ffe000-7fc912fff000 ---p 00000000 00:00 0 7fc912fff000-7fc9137ff000 rw-p 00000000 00:00 0 7fc9137ff000-7fc913800000 ---p 00000000 00:00 0 7fc913800000-7fc914000000 rw-p 00000000 00:00 0 7fc914000000-7fc914080000 rw-p 00000000 00:00 0 7fc914080000-7fc918000000 ---p 00000000 00:00 0 7fc9180b7000-7fc9180b8000 ---p 00000000 00:00 0 7fc9180b8000-7fc919768000 rw-p 00000000 00:00 0 7fc919768000-7fc919769000 ---p 00000000 00:00 0 7fc919769000-7fc919f69000 rw-p 00000000 00:00 0 7fc919f69000-7fc919f6a000 ---p 00000000 00:00 0 7fc919f6a000-7fc91a76a000 rw-p 00000000 00:00 0 7fc91a76a000-7fc91a76b000 ---p 00000000 00:00 0 7fc91a76b000-7fc91af6b000 rw-p 00000000 00:00 0 7fc91af6b000-7fc91af6c000 ---p 00000000 00:00 0 7fc91af6c000-7fc91b76c000 rw-p 00000000 00:00 0 7fc91b76c000-7fc91b76d000 ---p 00000000 00:00 0 7fc91b76d000-7fc91bf6d000 rw-p 00000000 00:00 0 7fc91bf6d000-7fc91bf6e000 ---p 00000000 00:00 0 7fc91bf6e000-7fc91c76e000 rw-p 00000000 00:00 0 7fc91c76e000-7fc91c76f000 ---p 00000000 00:00 0 7fc91c76f000-7fc91cf6f000 rw-p 00000000 00:00 0 7fc91cf6f000-7fc91cf70000 ---p 00000000 00:00 0 7fc91cf70000-7fc91d770000 rw-p 00000000 00:00 0 7fc91d770000-7fc91d771000 ---p 00000000 00:00 0 7fc91d771000-7fc91df71000 rw-p 00000000 00:00 0 7fc91df71000-7fc91df72000 ---p 00000000 00:00 0 7fc91df72000-7fc91e772000 rw-p 00000000 00:00 0 7fc91e772000-7fc91e773000 ---p 00000000 00:00 0 7fc91e773000-7fc91ef73000 rw-p 00000000 00:00 0 7fc91ef73000-7fc91ef74000 ---p 00000000 00:00 0 7fc91ef74000-7fc91f774000 rw-p 00000000 00:00 0 7fc91f774000-7fc91f775000 ---p 00000000 00:00 0 7fc91f775000-7fc91ff75000 rw-p 00000000 00:00 0 7fc91ff75000-7fc91ff76000 ---p 00000000 00:00 0 7fc91ff76000-7fc920776000 rw-p 00000000 00:00 0 7fc920776000-7fc920777000 ---p 00000000 00:00 0 7fc920777000-7fc921ead000 rw-p 00000000 00:00 0 7fc92200f000-7fc922010000 ---p 00000000 00:00 0 7fc922010000-7fc92205a000 rw-p 00000000 00:00 0 7fc92205a000-7fc922066000 r-xp 00000000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7fc922066000-7fc922265000 ---p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7fc922265000-7fc922266000 r--p 0000b000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7fc922266000-7fc922267000 rw-p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7fc922267000-7fc92226d000 rw-p 00000000 00:00 0 7fc92226d000-7fc92226e000 ---p 00000000 00:00 0 7fc92226e000-7fc922a6e000 rw-p 00000000 00:00 0 7fc922a6e000-7fc922a6f000 ---p 00000000 00:00 0 7fc922a6f000-7fc92326f000 rw-p 00000000 00:00 0 7fc92326f000-7fc923270000 ---p 00000000 00:00 0 7fc923270000-7fc923a70000 rw-p 00000000 00:00 0 7fc923a70000-7fc923a71000 ---p 00000000 00:00 0 7fc923a71000-7fc924271000 rw-p 00000000 00:00 0 7fc924271000-7fc926271000 rw-p 00000000 00:00 0 7fc926271000-7fc928000000 rw-p 00000000 00:00 0 7fc928000000-7fc9a8000000 rw-p 00000000 00:00 0 7fc9a8000000-7fc9a80de000 rw-p 00000000 00:00 0 7fc9a80de000-7fc9ac000000 ---p 00000000 00:00 0 7fc9ac030000-7fc9ac031000 ---p 00000000 00:00 0 7fc9ac031000-7fc9ac07b000 rw-p 00000000 00:00 0 7fc9ac07b000-7fc9ac07c000 ---p 00000000 00:00 0 7fc9ac07c000-7fc9ac0c6000 rw-p 00000000 00:00 0 7fc9ac0c6000-7fc9ac0c7000 ---p 00000000 00:00 0 7fc9ac0c7000-7fc9bc000000 rw-p 00000000 00:00 0 7fc9bc000000-7fc9bc021000 rw-p 00000000 00:00 0 7fc9bc021000-7fc9c0000000 ---p 00000000 00:00 0 7fc9c0004000-7fc9c0005000 ---p 00000000 00:00 0 7fc9c0005000-7fc9c004f000 rw-p 00000000 00:00 0 7fc9c0057000-7fc9c0058000 ---p 00000000 00:00 0 7fc9c0058000-7fc9c00a2000 rw-p 00000000 00:00 0 7fc9c00a2000-7fc9c00a3000 ---p 00000000 00:00 0 7fc9c00a3000-7fc9c00ed000 rw-p 00000000 00:00 0 7fc9c00ed000-7fc9c00ee000 ---p 00000000 00:00 0 7fc9c00ee000-7fc9c0539000 rw-p 00000000 00:00 0 7fc9c0539000-7fc9c053a000 ---p 00000000 00:00 0 7fc9c053a000-7fc9c146f000 rw-p 00000000 00:00 0 7fc9c146f000-7fc9c1470000 ---p 00000000 00:00 0 7fc9c1470000-7fc9c1c70000 rw-p 00000000 00:00 0 7fc9c1c70000-7fc9c1c73000 r-xp 00000000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7fc9c1c73000-7fc9c1e72000 ---p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7fc9c1e72000-7fc9c1e73000 r--p 00002000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7fc9c1e73000-7fc9c1e74000 rw-p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7fc9c1e74000-7fc9c1e82000 r-xp 00000000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7fc9c1e82000-7fc9c2082000 ---p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7fc9c2082000-7fc9c2083000 r--p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7fc9c2083000-7fc9c2084000 rw-p 0000f000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7fc9c2084000-7fc9c2093000 r-xp 00000000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7fc9c2093000-7fc9c2292000 ---p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7fc9c2292000-7fc9c2293000 r--p 0000e000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7fc9c2293000-7fc9c2294000 rw-p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7fc9c2294000-7fc9c22ab000 r-xp 00000000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7fc9c22ab000-7fc9c24aa000 ---p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7fc9c24aa000-7fc9c24ab000 r--p 00016000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7fc9c24ab000-7fc9c24ac000 rw-p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7fc9c24ac000-7fc9c250c000 r-xp 00000000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7fc9c250c000-7fc9c270c000 ---p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7fc9c270c000-7fc9c270d000 r--p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7fc9c270d000-7fc9c270e000 rw-p 00061000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7fc9c270e000-7fc9c2712000 r-xp 00000000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7fc9c2712000-7fc9c2911000 ---p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7fc9c2911000-7fc9c2912000 r--p 00003000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7fc9c2912000-7fc9c2913000 rw-p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7fc9c2913000-7fc9c2944000 r-xp 00000000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7fc9c2944000-7fc9c2b43000 ---p 00031000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7fc9c2b43000-7fc9c2b45000 r--p 00030000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7fc9c2b45000-7fc9c2b46000 rw-p 00032000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7fc9c2b46000-7fc9c2b49000 r-xp 00000000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7fc9c2b49000-7fc9c2d48000 ---p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7fc9c2d48000-7fc9c2d49000 r--p 00002000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7fc9c2d49000-7fc9c2d4a000 rw-p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7fc9c2d4a000-7fc9c2e23000 r-xp 00000000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7fc9c2e23000-7fc9c3022000 ---p 000d9000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7fc9c3022000-7fc9c3030000 r--p 000d8000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7fc9c3030000-7fc9c3033000 rw-p 000e6000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7fc9c3033000-7fc9c307d000 r-xp 00000000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7fc9c307d000-7fc9c327d000 ---p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7fc9c327d000-7fc9c327e000 r--p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7fc9c327e000-7fc9c3280000 rw-p 0004b000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7fc9c3280000-7fc9c3295000 r-xp 00000000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fc9c3295000-7fc9c3494000 ---p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fc9c3494000-7fc9c3495000 r--p 00014000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fc9c3495000-7fc9c3496000 rw-p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fc9c3496000-7fc9c34e4000 r-xp 00000000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7fc9c34e4000-7fc9c36e4000 ---p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7fc9c36e4000-7fc9c36e6000 r--p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7fc9c36e6000-7fc9c36e7000 rw-p 00050000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7fc9c36e7000-7fc9c36fd000 r-xp 00000000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7fc9c36fd000-7fc9c38fc000 ---p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7fc9c38fc000-7fc9c38fd000 r--p 00015000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7fc9c38fd000-7fc9c38fe000 rw-p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7fc9c38fe000-7fc9c3900000 rw-p 00000000 00:00 0 7fc9c3900000-7fc9c3904000 r-xp 00000000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7fc9c3904000-7fc9c3b03000 ---p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7fc9c3b03000-7fc9c3b04000 r--p 00003000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7fc9c3b04000-7fc9c3b05000 rw-p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7fc9c3b05000-7fc9c3b82000 r-xp 00000000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7fc9c3b82000-7fc9c3d81000 ---p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7fc9c3d81000-7fc9c3d82000 r--p 0007c000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7fc9c3d82000-7fc9c3d85000 rw-p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7fc9c3d85000-7fc9c3d86000 rw-p 00000000 00:00 0 7fc9c3d86000-7fc9c3d9a000 r-xp 00000000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7fc9c3d9a000-7fc9c3f99000 ---p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7fc9c3f99000-7fc9c3f9a000 r--p 00013000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7fc9c3f9a000-7fc9c3f9b000 rw-p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7fc9c3f9b000-7fc9c3fbf000 r-xp 00000000 fd:00 2101234 /usr/lib64/libselinux.so.1 7fc9c3fbf000-7fc9c41be000 ---p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7fc9c41be000-7fc9c41bf000 r--p 00023000 fd:00 2101234 /usr/lib64/libselinux.so.1 7fc9c41bf000-7fc9c41c0000 rw-p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7fc9c41c0000-7fc9c41c2000 rw-p 00000000 00:00 0 7fc9c41c2000-7fc9c41c9000 r-xp 00000000 fd:00 2101451 /usr/lib64/librt-2.17.so 7fc9c41c9000-7fc9c43c8000 ---p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7fc9c43c8000-7fc9c43c9000 r--p 00006000 fd:00 2101451 /usr/lib64/librt-2.17.so 7fc9c43c9000-7fc9c43ca000 rw-p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7fc9c43ca000-7fc9c43ce000 r-xp 00000000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7fc9c43ce000-7fc9c45cd000 ---p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7fc9c45cd000-7fc9c45ce000 r--p 00003000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7fc9c45ce000-7fc9c45cf000 rw-p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7fc9c45cf000-7fc9c45d1000 r-xp 00000000 fd:00 2227367 /usr/lib64/libfreebl3.so 7fc9c45d1000-7fc9c47d0000 ---p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7fc9c47d0000-7fc9c47d1000 r--p 00001000 fd:00 2227367 /usr/lib64/libfreebl3.so 7fc9c47d1000-7fc9c47d2000 rw-p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7fc9c47d2000-7fc9c4995000 r-xp 00000000 fd:00 2097970 /usr/lib64/libc-2.17.so 7fc9c4995000-7fc9c4b95000 ---p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7fc9c4b95000-7fc9c4b99000 r--p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7fc9c4b99000-7fc9c4b9b000 rw-p 001c7000 fd:00 2097970 /usr/lib64/libc-2.17.so 7fc9c4b9b000-7fc9c4ba0000 rw-p 00000000 00:00 0 7fc9c4ba0000-7fc9c4ca1000 r-xp 00000000 fd:00 2097978 /usr/lib64/libm-2.17.so 7fc9c4ca1000-7fc9c4ea0000 ---p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7fc9c4ea0000-7fc9c4ea1000 r--p 00100000 fd:00 2097978 /usr/lib64/libm-2.17.so 7fc9c4ea1000-7fc9c4ea2000 rw-p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7fc9c4ea2000-7fc9c4f8b000 r-xp 00000000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7fc9c4f8b000-7fc9c518a000 ---p 000e9000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7fc9c518a000-7fc9c5192000 r--p 000e8000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7fc9c5192000-7fc9c5194000 rw-p 000f0000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7fc9c5194000-7fc9c51a9000 rw-p 00000000 00:00 0 7fc9c51a9000-7fc9c51ab000 r-xp 00000000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7fc9c51ab000-7fc9c53ab000 ---p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7fc9c53ab000-7fc9c53ac000 r--p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7fc9c53ac000-7fc9c53ad000 rw-p 00003000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7fc9c53ad000-7fc9c55e3000 r-xp 00000000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7fc9c55e3000-7fc9c57e3000 ---p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7fc9c57e3000-7fc9c57ff000 r--p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7fc9c57ff000-7fc9c580c000 rw-p 00252000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7fc9c580c000-7fc9c5810000 rw-p 00000000 00:00 0 7fc9c5810000-7fc9c5877000 r-xp 00000000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7fc9c5877000-7fc9c5a77000 ---p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7fc9c5a77000-7fc9c5a7b000 r--p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7fc9c5a7b000-7fc9c5a82000 rw-p 0006b000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7fc9c5a82000-7fc9c5ab1000 r-xp 00000000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7fc9c5ab1000-7fc9c5cb1000 ---p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7fc9c5cb1000-7fc9c5cb2000 r--p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7fc9c5cb2000-7fc9c5cb3000 rw-p 00030000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7fc9c5cb3000-7fc9c5cbb000 r-xp 00000000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7fc9c5cbb000-7fc9c5eba000 ---p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7fc9c5eba000-7fc9c5ebb000 r--p 00007000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7fc9c5ebb000-7fc9c5ebc000 rw-p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7fc9c5ebc000-7fc9c5eea000 rw-p 00000000 00:00 0 7fc9c5eea000-7fc9c5eff000 r-xp 00000000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7fc9c5eff000-7fc9c60fe000 ---p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7fc9c60fe000-7fc9c60ff000 r--p 00014000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7fc9c60ff000-7fc9c6100000 rw-p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7fc9c6100000-7fc9c6101000 r-xp 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7fc9c6101000-7fc9c6300000 ---p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7fc9c6300000-7fc9c6301000 r--p 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7fc9c6301000-7fc9c6302000 rw-p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7fc9c6302000-7fc9c6327000 r-xp 00000000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7fc9c6327000-7fc9c6526000 ---p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7fc9c6526000-7fc9c6527000 r--p 00024000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7fc9c6527000-7fc9c6528000 rw-p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7fc9c6528000-7fc9c653f000 r-xp 00000000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7fc9c653f000-7fc9c673e000 ---p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7fc9c673e000-7fc9c673f000 r--p 00016000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7fc9c673f000-7fc9c6740000 rw-p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7fc9c6740000-7fc9c6744000 rw-p 00000000 00:00 0 7fc9c6744000-7fc9c6766000 r-xp 00000000 fd:00 2227369 /usr/lib64/ld-2.17.so 7fc9c676a000-7fc9c676b000 ---p 00000000 00:00 0 7fc9c676b000-7fc9c67b6000 rw-p 00000000 00:00 0 7fc9c67b6000-7fc9c67b9000 rw-s 00000000 00:0b 90136 /[aio] (deleted) 7fc9c67b9000-7fc9c67bc000 rw-s 00000000 00:0b 90135 /[aio] (deleted) 7fc9c67bc000-7fc9c67bf000 rw-s 00000000 00:0b 90134 /[aio] (deleted) 7fc9c67bf000-7fc9c67c2000 rw-s 00000000 00:0b 90133 /[aio] (deleted) 7fc9c67c2000-7fc9c67c5000 rw-s 00000000 00:0b 90132 /[aio] (deleted) 7fc9c67c5000-7fc9c67c8000 rw-s 00000000 00:0b 90131 /[aio] (deleted) 7fc9c67c8000-7fc9c67cb000 rw-s 00000000 00:0b 90130 /[aio] (deleted) 7fc9c67cb000-7fc9c67ce000 rw-s 00000000 00:0b 90129 /[aio] (deleted) 7fc9c67ce000-7fc9c695c000 rw-p 00000000 00:00 0 7fc9c695c000-7fc9c695d000 rw-s 00000000 00:0b 90137 /[aio] (deleted) 7fc9c695d000-7fc9c6960000 rw-s 00000000 00:0b 90128 /[aio] (deleted) 7fc9c6960000-7fc9c6963000 rw-s 00000000 00:0b 90127 /[aio] (deleted) 7fc9c6963000-7fc9c6964000 rw-s 00000000 00:0b 90126 /[aio] (deleted) 7fc9c6964000-7fc9c6965000 rw-p 00000000 00:00 0 7fc9c6965000-7fc9c6966000 r--p 00021000 fd:00 2227369 /usr/lib64/ld-2.17.so 7fc9c6966000-7fc9c6967000 rw-p 00022000 fd:00 2227369 /usr/lib64/ld-2.17.so 7fc9c6967000-7fc9c6968000 rw-p 00000000 00:00 0 7ffeb8abe000-7ffeb8adf000 rw-p 00000000 00:00 0 [stack] 7ffeb8b0d000-7ffeb8b0f000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Fatal signal 6 while backtracing 2020-02-01 1:43:39 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:43:39 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:43:39 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:43:39 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:43:39 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:43:39 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:43:39 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:43:39 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:43:39 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:43:39 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137419205 2020-02-01 1:43:39 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:43:40 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:43:40 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:43:40 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:43:40 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:43:40 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:43:40 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:43:40 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137419214; transaction id 5905156 2020-02-01 1:43:40 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:43:40 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:43:40 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:43:40 0 [Note] Starting crash recovery... 2020-02-01 1:43:40 0 [Note] Crash recovery finished. 2020-02-01 1:43:40 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:43:40 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:43:40 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:43:40 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000164' at position 346 2020-02-01 1:43:40 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:43:40 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:43:40 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:43:40 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:43:40 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:43:40 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:43:40 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:43:40 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000164' at position 346 2020-02-01 1:43:41 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:43:41 2020-02-01 1:44:03 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:44:03 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000166' at position 402570847 2020-02-01 1:44:03 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:44:09 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:44:10 14 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:44:10 14 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 1:44:10 19 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000001, 211055) 2020-02-01 1:44:14 25 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:44:14 25 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:44:14 26 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:44:14 26 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:44:14 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=202 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ffa54000ac8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ffb18057cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x558652e5ff2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5586528f63af] sigaction.c:0(__restore_rt)[0x7ffb1e39e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x558652824151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x558652962545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x558652718ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55865270ed4b] /usr/sbin/mysqld(+0x6ca792)[0x558652719792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5586527416da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55865274365d] /usr/sbin/mysqld(+0x65e3f3)[0x5586526ad3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5586526accb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x558652740b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55865274365d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x558652743761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x558652744377] /usr/sbin/mysqld(+0x58fd84)[0x5586525ded84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5586526ebb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x558652703b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x558652703cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5586527045e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x558652704677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5586526f356d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5586526f507c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5586527d0afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5586527d0bdd] pthread_create.c:0(start_thread)[0x7ffb1e396ea5] /lib64/libc.so.6(clone+0x6d)[0x7ffb1c7378cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ffa5400feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 27 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:44:20 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:44:20 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:44:20 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:44:20 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:44:20 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:44:20 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:44:20 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:44:21 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:44:21 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:44:21 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137422040 2020-02-01 1:44:21 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:44:21 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:44:21 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:44:21 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:44:21 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:44:21 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:44:21 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:44:21 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137422049; transaction id 5905167 2020-02-01 1:44:21 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:44:21 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:44:21 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:44:21 0 [Note] Starting crash recovery... 2020-02-01 1:44:21 0 [Note] Crash recovery finished. 2020-02-01 1:44:21 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:44:21 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:44:21 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:44:21 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000166' at position 402570847 2020-02-01 1:44:21 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:44:21 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:44:21 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:44:21 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:44:21 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:44:21 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:44:21 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:44:21 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000166' at position 402570847 2020-02-01 1:44:22 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:44:22 2020-02-01 1:44:49 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 200201 1:44:49 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=1 max_threads=202 thread_count=9 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fcc14006bf8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fccd80bccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5647e572af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5647e51c13af] sigaction.c:0(__restore_rt)[0x7fccde9c9630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5647e50ef151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5647e522d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5647e4fe3ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5647e4fd9d4b] /usr/sbin/mysqld(+0x6ca792)[0x5647e4fe4792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5647e500c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5647e500e65d] /usr/sbin/mysqld(+0x65e3f3)[0x5647e4f783f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5647e4f77cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5647e500bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5647e500e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5647e500e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5647e500f377] /usr/sbin/mysqld(+0x58fd84)[0x5647e4ea9d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5647e4fb6b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5647e4fceb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5647e4fcecd2] /usr/sbin/mysqld(+0x6b55e7)[0x5647e4fcf5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5647e4fcf677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5647e4fbe56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5647e4fc007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5647e509bafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5647e509bbdd] pthread_create.c:0(start_thread)[0x7fccde9c1ea5] /lib64/libc.so.6(clone+0x6d)[0x7fccdcd628cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fcc14010ea0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 14 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:44:55 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:44:55 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:44:55 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:44:55 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:44:55 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:44:55 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:44:55 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:44:55 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:44:55 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:44:55 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137424987 2020-02-01 1:44:55 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:44:55 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:44:55 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:44:55 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:44:55 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:44:55 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:44:55 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:44:55 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137424996; transaction id 5905178 2020-02-01 1:44:55 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:44:55 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:44:55 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:44:55 0 [Note] Starting crash recovery... 2020-02-01 1:44:55 0 [Note] Crash recovery finished. 2020-02-01 1:44:55 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:44:55 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:44:55 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:44:55 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000167' at position 45851 2020-02-01 1:44:55 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:44:55 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:44:55 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000167' at position 45851 2020-02-01 1:44:55 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:44:55 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:44:55 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:44:55 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:44:55 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:44:56 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:44:56 2020-02-01 1:45:15 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000002, 282837) 2020-02-01 1:45:24 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:45:28 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:45:28 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:45:28 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:45:28 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:45:28 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0820000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f08c4139cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56487deddf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56487d9743af] sigaction.c:0(__restore_rt)[0x7f08dff8d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56487d8a2151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56487d9e0545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56487d796ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56487d78cd4b] /usr/sbin/mysqld(+0x6ca792)[0x56487d797792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56487d7bf6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56487d7c165d] /usr/sbin/mysqld(+0x65e3f3)[0x56487d72b3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56487d72acb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56487d7beb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56487d7c165d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56487d7c1761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56487d7c2377] /usr/sbin/mysqld(+0x58fd84)[0x56487d65cd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56487d769b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56487d781b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56487d781cd2] /usr/sbin/mysqld(+0x6b55e7)[0x56487d7825e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56487d782677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56487d77156d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56487d77307c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56487d84eafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56487d84ebdd] pthread_create.c:0(start_thread)[0x7f08dff85ea5] /lib64/libc.so.6(clone+0x6d)[0x7f08de3268cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f082000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:45:34 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:45:34 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:45:34 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:45:34 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:45:34 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:45:34 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:45:34 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:45:34 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:45:34 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:45:34 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137427834 2020-02-01 1:45:34 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:45:34 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:45:34 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:45:34 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:45:34 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:45:34 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:45:34 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:45:35 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137427843; transaction id 5905189 2020-02-01 1:45:35 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:45:35 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:45:35 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:45:35 0 [Note] Starting crash recovery... 2020-02-01 1:45:35 0 [Note] Crash recovery finished. 2020-02-01 1:45:35 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:45:35 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:45:35 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:45:35 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000167' at position 157090 2020-02-01 1:45:35 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:45:35 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:45:35 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:45:35 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:45:35 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:45:35 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:45:35 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:45:35 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:45:35 2020-02-01 1:45:36 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000167' at position 157090 2020-02-01 1:45:59 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:46:02 14 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-02-01 1:46:02 14 [Warning] Checking table: './wpstart/wp_options' 2020-02-01 1:46:14 27 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:46:14 27 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:46:29 42 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000004, 1464) 2020-02-01 1:46:32 44 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:46:32 44 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:46:32 45 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:46:32 45 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:46:32 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9b88006bf8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f9c4408ccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56401a05af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x564019af13af] sigaction.c:0(__restore_rt)[0x7f9c4abc5630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x564019a1f151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x564019b5d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x564019913ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x564019909d4b] /usr/sbin/mysqld(+0x6ca792)[0x564019914792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56401993c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56401993e65d] /usr/sbin/mysqld(+0x65e3f3)[0x5640198a83f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5640198a7cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56401993bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56401993e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56401993e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56401993f377] /usr/sbin/mysqld(+0x58fd84)[0x5640197d9d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5640198e6b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5640198feb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5640198fecd2] /usr/sbin/mysqld(+0x6b55e7)[0x5640198ff5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5640198ff677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5640198ee56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5640198f007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5640199cbafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5640199cbbdd] pthread_create.c:0(start_thread)[0x7f9c4abbdea5] /lib64/libc.so.6(clone+0x6d)[0x7f9c48f5e8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f9b88010ea0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 46 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:46:38 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:46:38 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:46:38 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:46:38 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:46:38 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:46:38 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:46:38 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:46:39 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:46:39 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:46:39 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137430697 2020-02-01 1:46:39 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:46:39 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:46:39 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:46:39 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:46:39 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:46:39 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:46:39 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:46:39 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137430706; transaction id 5905200 2020-02-01 1:46:39 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:46:39 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:46:39 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:46:39 0 [Note] Starting crash recovery... 2020-02-01 1:46:39 0 [Note] Crash recovery finished. 2020-02-01 1:46:39 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:46:39 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:46:39 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:46:39 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000167' at position 190442 2020-02-01 1:46:39 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:46:39 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:46:39 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000167' at position 190442 2020-02-01 1:46:39 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 1:46:39 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 1:46:39 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 1:46:39 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 1:46:39 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:46:39 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:46:39 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 1:46:39 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:46:39 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:46:39 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:46:40 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:46:40 2020-02-01 1:47:01 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:47:01 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000167' at position 199803 2020-02-01 1:47:01 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:47:05 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:47:07 14 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000005, 584135) 2020-02-01 1:47:18 18 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:47:18 18 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 1:47:20 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:47:20 20 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:47:20 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:47:20 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:47:20 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc3f8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc49c0fccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561a19ab8f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x561a1954f3af] sigaction.c:0(__restore_rt)[0x7fc4b8765630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x561a1947d151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x561a195bb545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x561a19371ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x561a19367d4b] /usr/sbin/mysqld(+0x6ca792)[0x561a19372792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x561a1939a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x561a1939c65d] /usr/sbin/mysqld(+0x65e3f3)[0x561a193063f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x561a19305cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x561a19399b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x561a1939c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x561a1939c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x561a1939d377] /usr/sbin/mysqld(+0x58fd84)[0x561a19237d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x561a19344b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x561a1935cb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x561a1935ccd2] /usr/sbin/mysqld(+0x6b55e7)[0x561a1935d5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x561a1935d677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x561a1934c56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x561a1934e07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x561a19429afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x561a19429bdd] pthread_create.c:0(start_thread)[0x7fc4b875dea5] /lib64/libc.so.6(clone+0x6d)[0x7fc4b6afe8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc3f800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:47:25 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:47:25 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:47:25 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:47:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:47:25 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:47:25 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:47:25 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:47:26 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:47:26 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:47:26 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137433544 2020-02-01 1:47:26 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:47:26 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:47:26 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:47:26 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:47:26 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:47:26 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:47:26 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:47:26 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137433553; transaction id 5905211 2020-02-01 1:47:26 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:47:26 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:47:26 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:47:26 0 [Note] Starting crash recovery... 2020-02-01 1:47:26 0 [Note] Crash recovery finished. 2020-02-01 1:47:26 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:47:26 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:47:26 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:47:26 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000167' at position 199803 2020-02-01 1:47:26 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:47:26 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:47:26 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000167' at position 199803 2020-02-01 1:47:26 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:47:26 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:47:26 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:47:26 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:47:26 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:47:26 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:47:26 2020-02-01 1:47:39 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:47:39 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000168' at position 1464 2020-02-01 1:47:40 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:47:45 17 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000006, 1464) 2020-02-01 1:47:55 18 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:47:56 24 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-02-01 1:47:56 24 [Warning] Checking table: './wpcomm305/comm305_options' 2020-02-01 1:48:02 29 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:48:02 29 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:48:02 30 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:48:02 30 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:48:02 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=202 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f06a8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0744090cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x557e83996f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x557e8342d3af] sigaction.c:0(__restore_rt)[0x7f075f855630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x557e8335b151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x557e83499545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x557e8324fccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x557e83245d4b] /usr/sbin/mysqld(+0x6ca792)[0x557e83250792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x557e832786da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557e8327a65d] /usr/sbin/mysqld(+0x65e3f3)[0x557e831e43f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x557e831e3cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x557e83277b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x557e8327a65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x557e8327a761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x557e8327b377] /usr/sbin/mysqld(+0x58fd84)[0x557e83115d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x557e83222b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x557e8323ab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x557e8323acd2] /usr/sbin/mysqld(+0x6b55e7)[0x557e8323b5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x557e8323b677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x557e8322a56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x557e8322c07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x557e83307afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x557e83307bdd] pthread_create.c:0(start_thread)[0x7f075f84dea5] /lib64/libc.so.6(clone+0x6d)[0x7f075dbee8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f06a800feb0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 31 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:48:08 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:48:08 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:48:08 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:48:08 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:48:08 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:48:08 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:48:08 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:48:09 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:48:09 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:48:09 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137436407 2020-02-01 1:48:09 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:48:09 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:48:09 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:48:09 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:48:09 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:48:09 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:48:09 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:48:09 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137436416; transaction id 5905222 2020-02-01 1:48:09 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:48:09 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:48:09 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:48:09 0 [Note] Starting crash recovery... 2020-02-01 1:48:09 0 [Note] Crash recovery finished. 2020-02-01 1:48:09 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:48:09 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:48:09 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:48:09 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000168' at position 1464 2020-02-01 1:48:09 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:48:09 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:48:09 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 1:48:09 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 1:48:09 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 1:48:09 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 1:48:09 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:48:09 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:48:09 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 1:48:09 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:48:09 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:48:09 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:48:09 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:48:09 2020-02-01 1:48:10 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000168' at position 1464 2020-02-01 1:48:12 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:48:12 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000169' at position 1464 2020-02-01 1:48:12 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:48:18 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000007, 26710) 2020-02-01 1:48:35 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:48:36 15 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:48:36 15 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:48:36 16 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:48:36 16 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:48:36 17 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:48:36 17 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:48:36 16 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 1:48:36 16 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 1:48:36 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=202 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fd51c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fd533ffacf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b553ad3f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55b55356a3af] sigaction.c:0(__restore_rt)[0x7fd5d872e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55b553498151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55b5535d6545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55b55338cccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55b553382d4b] /usr/sbin/mysqld(+0x6ca792)[0x55b55338d792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55b5533b56da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b5533b765d] /usr/sbin/mysqld(+0x65e3f3)[0x55b5533213f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55b553320cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55b5533b4b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55b5533b765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55b5533b7761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55b5533b8377] /usr/sbin/mysqld(+0x58fd84)[0x55b553252d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55b55335fb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55b553377b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55b553377cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55b5533785e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55b553378677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55b55336756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55b55336907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55b553444afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55b553444bdd] pthread_create.c:0(start_thread)[0x7fd5d8726ea5] /lib64/libc.so.6(clone+0x6d)[0x7fd5d6ac78cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fd51c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:48:41 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:48:41 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:48:41 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:48:41 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:48:41 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:48:41 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:48:41 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:48:42 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:48:42 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:48:42 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137439322 2020-02-01 1:48:42 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:48:42 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:48:42 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:48:42 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:48:42 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:48:42 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:48:42 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:48:42 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137439331; transaction id 5905234 2020-02-01 1:48:42 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:48:42 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:48:42 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:48:42 0 [Note] Starting crash recovery... 2020-02-01 1:48:42 0 [Note] Crash recovery finished. 2020-02-01 1:48:42 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:48:42 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:48:42 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:48:42 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000169' at position 1464 2020-02-01 1:48:42 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:48:42 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:48:42 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000169' at position 1464 2020-02-01 1:48:42 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:48:42 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:48:42 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:48:42 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:48:42 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:48:42 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:48:42 2020-02-01 1:48:59 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:48:59 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000170' at position 189970 2020-02-01 1:48:59 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:49:05 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000008, 2513) 2020-02-01 1:49:10 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:49:13 17 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:49:13 17 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:49:13 18 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:49:13 18 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 1:49:13 18 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 1:49:13 18 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 1:49:14 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f93e8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f947d4b4cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5654987a3f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56549823a3af] sigaction.c:0(__restore_rt)[0x7f94996fe630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x565498168151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5654982a6545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56549805cccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x565498052d4b] /usr/sbin/mysqld(+0x6ca792)[0x56549805d792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5654980856da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56549808765d] /usr/sbin/mysqld(+0x65e3f3)[0x565497ff13f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x565497ff0cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x565498084b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56549808765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x565498087761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x565498088377] /usr/sbin/mysqld(+0x58fd84)[0x565497f22d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56549802fb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x565498047b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x565498047cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5654980485e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x565498048677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56549803756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56549803907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x565498114afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x565498114bdd] pthread_create.c:0(start_thread)[0x7f94996f6ea5] /lib64/libc.so.6(clone+0x6d)[0x7f9497a978cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f93e800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 19 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:49:19 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:49:19 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:49:19 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:49:19 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:49:19 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:49:19 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:49:19 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:49:20 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:49:20 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:49:20 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137443767 2020-02-01 1:49:20 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:49:20 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:49:20 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:49:20 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:49:20 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:49:20 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:49:20 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:49:20 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137443776; transaction id 5905245 2020-02-01 1:49:20 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:49:20 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:49:20 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:49:20 0 [Note] Starting crash recovery... 2020-02-01 1:49:20 0 [Note] Crash recovery finished. 2020-02-01 1:49:20 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:49:20 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:49:20 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:49:20 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000170' at position 189970 2020-02-01 1:49:20 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:49:20 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:49:20 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000170' at position 189970 2020-02-01 1:49:20 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:49:20 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:49:20 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:49:20 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:49:20 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:49:21 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:49:21 2020-02-01 1:49:40 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:49:40 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000171' at position 29263 2020-02-01 1:49:41 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:49:46 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:49:46 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000009, 2513) 2020-02-01 1:49:47 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:49:47 16 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:49:47 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:49:47 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:49:47 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=202 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fc3fc000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fc41a60ecf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55636d924f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55636d3bb3af] sigaction.c:0(__restore_rt)[0x7fc4bed2b630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55636d2e9151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55636d427545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55636d1ddccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55636d1d3d4b] /usr/sbin/mysqld(+0x6ca792)[0x55636d1de792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55636d2066da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55636d20865d] /usr/sbin/mysqld(+0x65e3f3)[0x55636d1723f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55636d171cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55636d205b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55636d20865d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55636d208761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55636d209377] /usr/sbin/mysqld(+0x58fd84)[0x55636d0a3d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55636d1b0b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55636d1c8b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55636d1c8cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55636d1c95e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55636d1c9677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55636d1b856d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55636d1ba07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55636d295afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55636d295bdd] pthread_create.c:0(start_thread)[0x7fc4bed23ea5] /lib64/libc.so.6(clone+0x6d)[0x7fc4bd0c48cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fc3fc00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:49:53 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:49:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:49:53 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:49:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:49:53 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:49:53 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:49:53 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:49:53 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:49:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:49:53 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137446628 2020-02-01 1:49:53 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:49:53 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:49:53 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:49:53 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:49:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:49:53 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:49:53 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:49:53 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137446637; transaction id 5905256 2020-02-01 1:49:53 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:49:53 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:49:53 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:49:53 0 [Note] Starting crash recovery... 2020-02-01 1:49:53 0 [Note] Crash recovery finished. 2020-02-01 1:49:53 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:49:53 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:49:53 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:49:53 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000171' at position 29263 2020-02-01 1:49:53 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:49:53 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:49:53 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:49:53 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:49:53 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:49:53 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:49:53 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:49:53 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000171' at position 29263 2020-02-01 1:49:54 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:49:54 2020-02-01 1:50:21 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:50:23 14 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 1:50:23 14 [Warning] Checking table: './wpms/wp_options' 2020-02-01 1:50:30 16 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:50:30 16 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:50:30 17 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:50:30 17 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:50:31 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f4108000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f41ac15acf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55bcae223f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55bcadcba3af] sigaction.c:0(__restore_rt)[0x7f41c7261630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55bcadbe8151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55bcadd26545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55bcadadcccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55bcadad2d4b] /usr/sbin/mysqld(+0x6ca792)[0x55bcadadd792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55bcadb056da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55bcadb0765d] /usr/sbin/mysqld(+0x65e3f3)[0x55bcada713f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55bcada70cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55bcadb04b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55bcadb0765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55bcadb07761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55bcadb08377] /usr/sbin/mysqld(+0x58fd84)[0x55bcad9a2d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55bcadaafb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55bcadac7b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55bcadac7cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55bcadac85e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55bcadac8677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55bcadab756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55bcadab907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55bcadb94afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55bcadb94bdd] pthread_create.c:0(start_thread)[0x7f41c7259ea5] /lib64/libc.so.6(clone+0x6d)[0x7f41c55fa8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f410800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 18 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:50:36 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:50:36 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:50:36 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:50:36 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:50:36 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:50:36 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:50:36 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:50:37 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:50:37 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:50:37 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137449491 2020-02-01 1:50:37 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:50:37 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:50:37 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:50:37 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:50:37 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:50:37 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:50:37 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:50:37 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137449500; transaction id 5905267 2020-02-01 1:50:37 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:50:37 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:50:37 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:50:37 0 [Note] Starting crash recovery... 2020-02-01 1:50:37 0 [Note] Crash recovery finished. 2020-02-01 1:50:37 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:50:37 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:50:37 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:50:37 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000172' at position 13083 2020-02-01 1:50:37 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:50:37 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:50:37 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000172' at position 13083 2020-02-01 1:50:37 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:50:37 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:50:37 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:50:37 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:50:37 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:50:38 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:50:38 2020-02-01 1:50:40 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:50:40 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000172' at position 14201 2020-02-01 1:50:40 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:50:46 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000010, 6961) 2020-02-01 1:51:06 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:51:08 14 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-02-01 1:51:08 14 [Warning] Checking table: './wpstart/wp_options' 2020-02-01 1:51:13 23 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:51:13 23 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:51:13 24 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:51:13 24 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:51:13 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=202 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fec08000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fecaca92cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5610d7d9cf2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5610d78333af] sigaction.c:0(__restore_rt)[0x7fecc8cc6630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5610d7761151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5610d789f545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5610d7655ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5610d764bd4b] /usr/sbin/mysqld(+0x6ca792)[0x5610d7656792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5610d767e6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5610d768065d] /usr/sbin/mysqld(+0x65e3f3)[0x5610d75ea3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5610d75e9cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5610d767db24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5610d768065d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5610d7680761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5610d7681377] /usr/sbin/mysqld(+0x58fd84)[0x5610d751bd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5610d7628b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5610d7640b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5610d7640cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5610d76415e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5610d7641677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5610d763056d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5610d763207c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5610d770dafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5610d770dbdd] pthread_create.c:0(start_thread)[0x7fecc8cbeea5] /lib64/libc.so.6(clone+0x6d)[0x7fecc705f8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fec0800fe70): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 25 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:51:19 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:51:19 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:51:19 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:51:19 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:51:19 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:51:19 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:51:19 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:51:20 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:51:20 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:51:20 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137452338 2020-02-01 1:51:20 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:51:20 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:51:20 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:51:20 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:51:20 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:51:20 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:51:20 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:51:20 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137452347; transaction id 5905278 2020-02-01 1:51:20 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:51:20 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:51:20 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:51:20 0 [Note] Starting crash recovery... 2020-02-01 1:51:20 0 [Note] Crash recovery finished. 2020-02-01 1:51:20 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:51:20 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:51:20 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:51:20 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000172' at position 14201 2020-02-01 1:51:20 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:51:20 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:51:20 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 1:51:20 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 1:51:20 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 1:51:20 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 1:51:20 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:51:20 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:51:20 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 1:51:20 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:51:20 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:51:20 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:51:20 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:51:20 2020-02-01 1:51:21 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000172' at position 14201 2020-02-01 1:51:25 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:51:25 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000173' at position 2513 2020-02-01 1:51:25 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:51:31 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000012, 375803) 2020-02-01 1:51:46 13 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:51:48 15 [ERROR] mysqld: Table './idtgroup/wp_usermeta' is marked as crashed and should be repaired 2020-02-01 1:51:48 15 [Warning] Checking table: './idtgroup/wp_usermeta' 2020-02-01 1:51:49 17 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 1:51:49 17 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 1:52:00 26 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:52:00 26 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:52:00 27 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:52:00 27 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 1:52:01 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=202 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f21f8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f229519ccf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55f41a69af2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55f41a1313af] sigaction.c:0(__restore_rt)[0x7f22b13cf630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55f41a05f151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55f41a19d545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55f419f53ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55f419f49d4b] /usr/sbin/mysqld(+0x6ca792)[0x55f419f54792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55f419f7c6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55f419f7e65d] /usr/sbin/mysqld(+0x65e3f3)[0x55f419ee83f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55f419ee7cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55f419f7bb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55f419f7e65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55f419f7e761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55f419f7f377] /usr/sbin/mysqld(+0x58fd84)[0x55f419e19d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55f419f26b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55f419f3eb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55f419f3ecd2] /usr/sbin/mysqld(+0x6b55e7)[0x55f419f3f5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55f419f3f677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55f419f2e56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55f419f3007c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55f41a00bafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55f41a00bbdd] pthread_create.c:0(start_thread)[0x7f22b13c7ea5] /lib64/libc.so.6(clone+0x6d)[0x7f22af7688cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f21f800fe70): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 29 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:52:07 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:52:07 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:52:07 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:52:07 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:52:07 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:52:07 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:52:07 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:52:07 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:52:07 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:52:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137455318 2020-02-01 1:52:07 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:52:07 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:52:07 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:52:07 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:52:07 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:52:07 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:52:07 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137455327; transaction id 5905289 2020-02-01 1:52:07 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:52:07 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:52:07 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:52:07 0 [Note] Starting crash recovery... 2020-02-01 1:52:07 0 [Note] Crash recovery finished. 2020-02-01 1:52:07 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:52:07 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:52:07 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:52:07 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000173' at position 2513 2020-02-01 1:52:07 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:52:07 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:52:07 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:52:07 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:52:07 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:52:07 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:52:07 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:52:07 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000173' at position 2513 2020-02-01 1:52:08 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:52:08 2020-02-01 1:52:13 17 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2020-02-01 1:52:15 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:52:15 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000174' at position 42695 2020-02-01 1:52:15 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:52:21 18 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000013, 128966) 2020-02-01 1:52:24 19 [ERROR] Got an error from thread_id=19, /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.10/storage/myisam/ha_myisam.cc:1080 2020-02-01 1:52:24 19 [ERROR] MySQL thread id 19, OS thread handle 139913276372736, query id 523 localhost root Checking table CHECK TABLE `sessions` 2020-02-01 1:52:36 20 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:53:05 31 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:53:05 31 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 1:53:05 31 [Warning] Recovering table: './ojsdata/sessions' 2020-02-01 1:53:05 31 [Note] mysqld: Making backup of data file ./ojsdata/sessions.MYD with extension '-200201015305.BAK' 2020-02-01 1:53:05 31 [Note] Found 18073 of 18072 rows when repairing './ojsdata/sessions' 200201 1:53:05 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=5 max_threads=202 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f3f7c000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4019151cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c0fd8eef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c0fd3853af] sigaction.c:0(__restore_rt)[0x7f4035408630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c0fd2b3151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c0fd3f1545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c0fd1a7ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c0fd19dd4b] /usr/sbin/mysqld(+0x6ca792)[0x55c0fd1a8792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c0fd1d06da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c0fd1d265d] /usr/sbin/mysqld(+0x65e3f3)[0x55c0fd13c3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c0fd13bcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c0fd1cfb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c0fd1d265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c0fd1d2761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c0fd1d3377] /usr/sbin/mysqld(+0x58fd84)[0x55c0fd06dd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c0fd17ab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c0fd192b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c0fd192cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c0fd1935e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c0fd193677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c0fd18256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c0fd18407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c0fd25fafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c0fd25fbdd] pthread_create.c:0(start_thread)[0x7f4035400ea5] /lib64/libc.so.6(clone+0x6d)[0x7f40337a18cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f3f7c00fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 33 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:53:12 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:53:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:53:12 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:53:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:53:12 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:53:13 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:53:13 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:53:13 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:53:13 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:53:13 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137460593 2020-02-01 1:53:13 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:53:13 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:53:13 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:53:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:53:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:53:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:53:13 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:53:13 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137460602; transaction id 5905300 2020-02-01 1:53:13 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:53:13 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:53:13 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:53:13 0 [Note] Starting crash recovery... 2020-02-01 1:53:13 0 [Note] Crash recovery finished. 2020-02-01 1:53:13 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:53:13 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:53:13 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:53:13 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000174' at position 42695 2020-02-01 1:53:13 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:53:13 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:53:13 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:53:13 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:53:13 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:53:13 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:53:13 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:53:13 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000174' at position 42695 2020-02-01 1:53:13 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:53:13 2020-02-01 1:53:41 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 200201 1:54:13 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f4808000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f48a9469cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a4fc942f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a4fc3d93af] sigaction.c:0(__restore_rt)[0x7f48c5668630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55a4fc307151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55a4fc445545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55a4fc1fbccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55a4fc1f1d4b] /usr/sbin/mysqld(+0x6ca792)[0x55a4fc1fc792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55a4fc2246da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a4fc22665d] /usr/sbin/mysqld(+0x65e3f3)[0x55a4fc1903f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55a4fc18fcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55a4fc223b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55a4fc22665d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55a4fc226761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55a4fc227377] /usr/sbin/mysqld(+0x58fd84)[0x55a4fc0c1d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55a4fc1ceb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55a4fc1e6b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55a4fc1e6cd2] /usr/sbin/mysqld(+0x6b55e7)[0x55a4fc1e75e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55a4fc1e7677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55a4fc1d656d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a4fc1d807c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55a4fc2b3afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a4fc2b3bdd] pthread_create.c:0(start_thread)[0x7f48c5660ea5] /lib64/libc.so.6(clone+0x6d)[0x7f48c3a018cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f480800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 27 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 1:54:19 0 [Note] mysqld: Aria engine: starting recovery recovered pages: 0% 10% 21% 32% 44% 56% 68% 82% 97% 100% (0.0 seconds); tables to flush: 9 8 7 6 5 4 3 2 1 0 (0.0 seconds); 2020-02-01 1:54:19 0 [Note] mysqld: Aria engine: recovery done 2020-02-01 1:54:19 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:54:19 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:54:19 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:54:19 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:54:19 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:54:19 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:54:19 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:54:19 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:54:19 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:54:19 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137463710 2020-02-01 1:54:19 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 1:54:20 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:54:20 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:54:20 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:54:20 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:54:20 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:54:20 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 1:54:20 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137463719; transaction id 5905311 2020-02-01 1:54:20 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:54:20 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/plugin' 2020-02-01 1:54:20 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 1:54:20 0 [Note] Starting crash recovery... 2020-02-01 1:54:20 0 [Note] Crash recovery finished. 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/servers' 2020-02-01 1:54:20 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/db' 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/proxies_priv' 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/roles_mapping' 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/global_priv' 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/time_zone_leap_second' 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/time_zone_name' 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/time_zone' 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/time_zone_transition_type' 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/time_zone_transition' 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/tables_priv' 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/columns_priv' 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/procs_priv' 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/func' 2020-02-01 1:54:20 0 [Note] Zerofilling moved table: './mysql/event' 2020-02-01 1:54:20 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:54:20 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:54:20 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000175' at position 95866 2020-02-01 1:54:20 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:54:20 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:54:20 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000175' at position 95866 2020-02-01 1:54:20 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:54:20 2020-02-01 1:54:20 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:54:20 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:54:20 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:54:20 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:54:20 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:54:26 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:54:26 10 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'dbsvc-527-bin.000175' at position 110692 2020-02-01 1:54:26 10 [ERROR] Slave I/O: error reconnecting to master 'replication@dbsvc-527.wichita.edu:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'dbsvc-527.wichita.edu' (111 "Connection refused"), Internal MariaDB error code: 2003 2020-02-01 1:54:31 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000014, 72271) 2020-02-01 1:54:47 16 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 1:54:50 19 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:54:52 22 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:54:59 27 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 1:55:21 33 [Warning] Aborted connection 33 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Got an error reading communication packets) 2020-02-01 1:55:21 31 [Warning] Aborted connection 31 to db: 'wpwsutv' user: 'wpwsutv' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-02-01 1:55:21 17 [Warning] Aborted connection 17 to db: 'kfbf' user: 'kfbf' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-02-01 1:55:21 22 [Warning] Aborted connection 22 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Table './ojsdata/sessions' is marked as crashed and should be re) 2020-02-01 1:55:21 20 [Warning] Aborted connection 20 to db: 'idtgroup' user: 'idtgroup' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-02-01 1:55:21 27 [Warning] Aborted connection 27 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Table './ojsdata/sessions' is marked as crashed and should be re) 2020-02-01 1:55:21 24 [Warning] Aborted connection 24 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-02-01 1:55:22 19 [Warning] Aborted connection 19 to db: 'ojsdata' user: 'ojsuser' host: '10.0.59.19' (Table './ojsdata/sessions' is marked as crashed and should be re) 2020-02-01 1:55:22 34 [Warning] Aborted connection 34 to db: 'wpcomm303' user: 'wpcomm303' host: '10.0.59.19' (Lock wait timeout exceeded; try restarting transaction) 2020-02-01 1:55:31 10 [Note] Slave: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication resumed in log 'dbsvc-527-bin.000175' at position 110692 2020-02-01 1:55:33 14 [Warning] Aborted connection 14 to db: 'mysql' user: 'root' host: 'localhost' (Old key file for table 'proc'; repair it!) 2020-02-01 1:55:47 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown 2020-02-01 1:55:47 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-02-01 1:55:47 0 [Note] InnoDB: FTS optimize thread exiting. 2020-02-01 1:55:47 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000175', position 110692 2020-02-01 1:55:49 0 [Note] InnoDB: Starting shutdown... 2020-02-01 1:55:49 0 [Note] InnoDB: Dumping buffer pool(s) to /data/mysql/ib_buffer_pool 2020-02-01 1:55:49 0 [Note] InnoDB: Buffer pool(s) dump completed at 200201 1:55:49 2020-02-01 1:55:50 0 [Note] InnoDB: Shutdown completed; log sequence number 5137467009; transaction id 5905433 2020-02-01 1:55:50 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 1:55:50 0 [Note] /usr/sbin/mysqld: Shutdown complete 2020-02-01 1:55:50 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 1:55:50 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 1:55:50 0 [Note] InnoDB: Uses event mutexes 2020-02-01 1:55:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 1:55:50 0 [Note] InnoDB: Number of pools: 1 2020-02-01 1:55:50 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 1:55:50 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 1:55:50 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 1:55:50 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 1:55:50 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 1:55:50 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 1:55:50 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 1:55:50 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 1:55:50 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137467009; transaction id 5905435 2020-02-01 1:55:50 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 1:55:50 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 1:55:50 0 [Note] Server socket created on IP: '::'. 2020-02-01 1:55:50 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 1:55:50 0 [Note] Added new Master_info '' to hash table 2020-02-01 1:55:50 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000175' at position 110692 2020-02-01 1:55:50 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 1:55:50 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 1:55:50 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 1:55:50 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 1:55:50 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 1:55:50 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 1:55:50 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 1:55:50 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000175' at position 110692 2020-02-01 1:55:51 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 1:55:51 2020-02-01 1:56:02 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 1:56:02 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000176', position 346 2020-02-01 1:56:34 13 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 1:57:18 13 [Warning] Aborted connection 13 to db: 'ojsdata' user: 'root' host: 'localhost' (Table './ojsdata/scheduled_tasks' is marked as crashed and shoul) 2020-02-01 1:57:44 19 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 2:03:59 19 [Warning] Aborted connection 19 to db: 'ojsdata' user: 'root' host: 'localhost' (Table './ojsdata/scheduled_tasks' is marked as crashed and shoul) 2020-02-01 2:04:23 32 [Warning] Aborted connection 32 to db: 'auditcalc' user: 'root' host: 'localhost' (Got an error writing communication packets) 2020-02-01 2:04:30 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown 2020-02-01 2:04:30 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-02-01 2:04:30 0 [Note] InnoDB: FTS optimize thread exiting. 2020-02-01 2:04:31 0 [Note] InnoDB: Starting shutdown... 2020-02-01 2:04:31 0 [Note] InnoDB: Dumping buffer pool(s) to /data/mysql/ib_buffer_pool 2020-02-01 2:04:31 0 [Note] InnoDB: Buffer pool(s) dump completed at 200201 2:04:31 2020-02-01 2:04:32 0 [Note] InnoDB: Shutdown completed; log sequence number 5137470324; transaction id 5905582 2020-02-01 2:04:32 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 2:04:32 0 [Note] /usr/sbin/mysqld: Shutdown complete 2020-02-01 2:05:05 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 2:05:05 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 2:05:05 0 [Note] InnoDB: Uses event mutexes 2020-02-01 2:05:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 2:05:05 0 [Note] InnoDB: Number of pools: 1 2020-02-01 2:05:05 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 2:05:05 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 2:05:05 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 2:05:05 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 2:05:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 2:05:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 2:05:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 2:05:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 2:05:05 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 2:05:06 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137470324; transaction id 5905446 2020-02-01 2:05:06 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 2:05:06 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 2:05:06 0 [Note] Server socket created on IP: '::'. 2020-02-01 2:05:06 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 2:05:06 0 [Note] Added new Master_info '' to hash table 2020-02-01 2:05:06 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000176' at position 346 2020-02-01 2:05:06 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 2:05:06 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 2:05:06 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 2:05:06 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 2:05:06 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 2:05:06 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 2:05:06 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 2:05:06 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 2:05:06 2020-02-01 2:05:07 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000176' at position 346 2020-02-01 2:05:23 10 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=2013) 2020-02-01 2:05:23 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000176', position 346 2020-02-01 2:08:08 21 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000014, 72271) 2020-02-01 2:11:35 26 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000001, 332) 2020-02-01 2:12:57 32 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 200201 2:13:03 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=10 max_threads=202 thread_count=17 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f1270000b68 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f12b36a7cf0 thread_stack 0x49000 *** buffer overflow detected ***: /usr/sbin/mysqld terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7f1356232567] /lib64/libc.so.6(+0x1166e2)[0x7f13562306e2] /lib64/libc.so.6(+0x1184c7)[0x7f13562324c7] /usr/sbin/mysqld(my_addr_resolve+0xda)[0x55989ee7f94a] /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x55989ee680c2] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55989e8fe3af] /lib64/libpthread.so.0(+0xf630)[0x7f1357e7f630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55989e82c151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55989e96a545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55989e720ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55989e716d4b] /usr/sbin/mysqld(+0x6ca792)[0x55989e721792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55989e7496da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55989e74b65d] /usr/sbin/mysqld(+0x65e3f3)[0x55989e6b53f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55989e6b4cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55989e748b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55989e74b65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55989e74b761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55989e74c377] /usr/sbin/mysqld(+0x58fd84)[0x55989e5e6d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55989e6f3b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55989e70bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55989e70bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55989e70c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55989e70c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55989e6fb56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55989e6fd07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55989e7d8afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55989e7d8bdd] /lib64/libpthread.so.0(+0x7ea5)[0x7f1357e77ea5] /lib64/libc.so.6(clone+0x6d)[0x7f13562188cd] ======= Memory map: ======== 55989e057000-55989f4a5000 r-xp 00000000 fd:00 2101454 /usr/sbin/mysqld 55989f6a4000-55989f7d0000 r--p 0144d000 fd:00 2101454 /usr/sbin/mysqld 55989f7d0000-55989f88a000 rw-p 01579000 fd:00 2101454 /usr/sbin/mysqld 55989f88a000-5598a0139000 rw-p 00000000 00:00 0 5598a1697000-5598a49a3000 rw-p 00000000 00:00 0 [heap] 7f126c000000-7f126c3ec000 rw-p 00000000 00:00 0 7f126c3ec000-7f1270000000 ---p 00000000 00:00 0 7f1270000000-7f12703f0000 rw-p 00000000 00:00 0 7f12703f0000-7f1274000000 ---p 00000000 00:00 0 7f1274000000-7f1274021000 rw-p 00000000 00:00 0 7f1274021000-7f1278000000 ---p 00000000 00:00 0 7f1278000000-7f1278021000 rw-p 00000000 00:00 0 7f1278021000-7f127c000000 ---p 00000000 00:00 0 7f127c000000-7f127c027000 rw-p 00000000 00:00 0 7f127c027000-7f1280000000 ---p 00000000 00:00 0 7f1284000000-7f1284021000 rw-p 00000000 00:00 0 7f1284021000-7f1288000000 ---p 00000000 00:00 0 7f1288000000-7f1288123000 rw-p 00000000 00:00 0 7f1288123000-7f128c000000 ---p 00000000 00:00 0 7f128c000000-7f128c03e000 rw-p 00000000 00:00 0 7f128c03e000-7f1290000000 ---p 00000000 00:00 0 7f1290000000-7f1290021000 rw-p 00000000 00:00 0 7f1290021000-7f1294000000 ---p 00000000 00:00 0 7f1294000000-7f12943ce000 rw-p 00000000 00:00 0 7f12943ce000-7f1298000000 ---p 00000000 00:00 0 7f1298000000-7f129bef2000 rw-p 00000000 00:00 0 7f129bef2000-7f129c000000 ---p 00000000 00:00 0 7f129c000000-7f129c3cd000 rw-p 00000000 00:00 0 7f129c3cd000-7f12a0000000 ---p 00000000 00:00 0 7f12a0c6c000-7f12a0c7e000 r-xp 00000000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f12a0c7e000-7f12a0e7d000 ---p 00012000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f12a0e7d000-7f12a0e80000 r--p 00011000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f12a0e80000-7f12a0e81000 rw-p 00014000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f12a0e81000-7f12a0e82000 ---p 00000000 00:00 0 7f12a0e82000-7f12a1682000 rw-p 00000000 00:00 0 7f12a1682000-7f12a1683000 ---p 00000000 00:00 0 7f12a1683000-7f12a1e83000 rw-p 00000000 00:00 0 7f12a1e83000-7f12a1e84000 ---p 00000000 00:00 0 7f12a1e84000-7f12a2684000 rw-p 00000000 00:00 0 7f12a2684000-7f12a2685000 ---p 00000000 00:00 0 7f12a2685000-7f12a2e85000 rw-p 00000000 00:00 0 7f12a2e85000-7f12a2e86000 ---p 00000000 00:00 0 7f12a2e86000-7f12a3686000 rw-p 00000000 00:00 0 7f12a3686000-7f12a3687000 ---p 00000000 00:00 0 7f12a3687000-7f12a3e87000 rw-p 00000000 00:00 0 7f12a3e87000-7f12a3e88000 ---p 00000000 00:00 0 7f12a3e88000-7f12a4688000 rw-p 00000000 00:00 0 7f12a4788000-7f12a4789000 ---p 00000000 00:00 0 7f12a4789000-7f12a4f89000 rw-p 00000000 00:00 0 7f12a4f89000-7f12a4f8a000 ---p 00000000 00:00 0 7f12a4f8a000-7f12a578a000 rw-p 00000000 00:00 0 7f12a578a000-7f12a578b000 ---p 00000000 00:00 0 7f12a578b000-7f12a5f8b000 rw-p 00000000 00:00 0 7f12a5f8b000-7f12a5f8c000 ---p 00000000 00:00 0 7f12a5f8c000-7f12a6ffe000 rw-p 00000000 00:00 0 7f12a6ffe000-7f12a6fff000 ---p 00000000 00:00 0 7f12a6fff000-7f12a77ff000 rw-p 00000000 00:00 0 7f12a77ff000-7f12a7800000 ---p 00000000 00:00 0 7f12a7800000-7f12a8000000 rw-p 00000000 00:00 0 7f12a8000000-7f12a83d1000 rw-p 00000000 00:00 0 7f12a83d1000-7f12ac000000 ---p 00000000 00:00 0 7f12ac048000-7f12ac04e000 r-xp 00000000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f12ac04e000-7f12ac24d000 ---p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f12ac24d000-7f12ac24e000 r--p 00005000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f12ac24e000-7f12ac24f000 rw-p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f12ac24f000-7f12ac688000 rw-p 00000000 00:00 0 7f12ac688000-7f12ac689000 ---p 00000000 00:00 0 7f12ac689000-7f12ace89000 rw-p 00000000 00:00 0 7f12ace89000-7f12ace8a000 ---p 00000000 00:00 0 7f12ace8a000-7f12ad68a000 rw-p 00000000 00:00 0 7f12ad68a000-7f12ad68b000 ---p 00000000 00:00 0 7f12ad68b000-7f12ade8b000 rw-p 00000000 00:00 0 7f12ade8b000-7f12ade8c000 ---p 00000000 00:00 0 7f12ade8c000-7f12ae68c000 rw-p 00000000 00:00 0 7f12ae68c000-7f12ae68d000 ---p 00000000 00:00 0 7f12ae68d000-7f12aee8d000 rw-p 00000000 00:00 0 7f12aee8d000-7f12aee8e000 ---p 00000000 00:00 0 7f12aee8e000-7f12af68e000 rw-p 00000000 00:00 0 7f12af68e000-7f12af68f000 ---p 00000000 00:00 0 7f12af68f000-7f12afe8f000 rw-p 00000000 00:00 0 7f12afe8f000-7f12afe90000 ---p 00000000 00:00 0 7f12afe90000-7f12b0690000 rw-p 00000000 00:00 0 7f12b0690000-7f12b0691000 ---p 00000000 00:00 0 7f12b0691000-7f12b0e91000 rw-p 00000000 00:00 0 7f12b0e91000-7f12b0e92000 ---p 00000000 00:00 0 7f12b0e92000-7f12b1692000 rw-p 00000000 00:00 0 7f12b1692000-7f12b1693000 ---p 00000000 00:00 0 7f12b1693000-7f12b1e93000 rw-p 00000000 00:00 0 7f12b1e93000-7f12b1e94000 ---p 00000000 00:00 0 7f12b1e94000-7f12b35ca000 rw-p 00000000 00:00 0 7f12b3613000-7f12b3614000 ---p 00000000 00:00 0 7f12b3614000-7f12b365e000 rw-p 00000000 00:00 0 7f12b365e000-7f12b365f000 ---p 00000000 00:00 0 7f12b365f000-7f12b36a9000 rw-p 00000000 00:00 0 7f12b36a9000-7f12b36aa000 ---p 00000000 00:00 0 7f12b36aa000-7f12b36f4000 rw-p 00000000 00:00 0 7f12b36f4000-7f12b36f5000 ---p 00000000 00:00 0 7f12b36f5000-7f12b373f000 rw-p 00000000 00:00 0 7f12b373f000-7f12b3740000 ---p 00000000 00:00 0 7f12b3740000-7f12b378a000 rw-p 00000000 00:00 0 7f12b378a000-7f12b378b000 ---p 00000000 00:00 0 7f12b378b000-7f12b3f8b000 rw-p 00000000 00:00 0 7f12b3f8b000-7f12b3f8c000 ---p 00000000 00:00 0 7f12b3f8c000-7f12b478c000 rw-p 00000000 00:00 0 7f12b478c000-7f12b478d000 ---p 00000000 00:00 0 7f12b478d000-7f12b4f8d000 rw-p 00000000 00:00 0 7f12b4f8d000-7f12b4f8e000 ---p 00000000 00:00 0 7f12b4f8e000-7f12b578e000 rw-p 00000000 00:00 0 7f12b578e000-7f12b778e000 rw-p 00000000 00:00 0 7f12b778e000-7f12b8000000 rw-p 00000000 00:00 0 7f12b8000000-7f1338000000 rw-p 00000000 00:00 0 7f1338000000-7f1338070000 rw-p 00000000 00:00 0 7f1338070000-7f133c000000 ---p 00000000 00:00 0 7f133c01d000-7f133c01e000 ---p 00000000 00:00 0 7f133c01e000-7f133c068000 rw-p 00000000 00:00 0 7f133c068000-7f133c069000 ---p 00000000 00:00 0 7f133c069000-7f133c0b3000 rw-p 00000000 00:00 0 7f133c0b3000-7f133c0b4000 ---p 00000000 00:00 0 7f133c0b4000-7f133c0fe000 rw-p 00000000 00:00 0 7f133c0fe000-7f133c0ff000 ---p 00000000 00:00 0 7f133c0ff000-7f134c000000 rw-p 00000000 00:00 0 7f134c000000-7f134c140000 rw-p 00000000 00:00 0 7f134c140000-7f1350000000 ---p 00000000 00:00 0 7f135002a000-7f135002b000 ---p 00000000 00:00 0 7f135002b000-7f1350075000 rw-p 00000000 00:00 0 7f1350075000-7f1350076000 ---p 00000000 00:00 0 7f1350076000-7f13500c0000 rw-p 00000000 00:00 0 7f13500c0000-7f13500c1000 ---p 00000000 00:00 0 7f13500c1000-7f135010b000 rw-p 00000000 00:00 0 7f135010b000-7f1350117000 r-xp 00000000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f1350117000-7f1350316000 ---p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f1350316000-7f1350317000 r--p 0000b000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f1350317000-7f1350318000 rw-p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f1350318000-7f135031e000 rw-p 00000000 00:00 0 7f1350326000-7f1350727000 rw-p 00000000 00:00 0 7f1350727000-7f1350728000 ---p 00000000 00:00 0 7f1350728000-7f1352db7000 rw-p 00000000 00:00 0 7f1352db7000-7f1352db8000 ---p 00000000 00:00 0 7f1352db8000-7f13535b8000 rw-p 00000000 00:00 0 7f13535b8000-7f13535bb000 r-xp 00000000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f13535bb000-7f13537ba000 ---p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f13537ba000-7f13537bb000 r--p 00002000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f13537bb000-7f13537bc000 rw-p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f13537bc000-7f13537ca000 r-xp 00000000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f13537ca000-7f13539ca000 ---p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f13539ca000-7f13539cb000 r--p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f13539cb000-7f13539cc000 rw-p 0000f000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f13539cc000-7f13539db000 r-xp 00000000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f13539db000-7f1353bda000 ---p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f1353bda000-7f1353bdb000 r--p 0000e000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f1353bdb000-7f1353bdc000 rw-p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f1353bdc000-7f1353bf3000 r-xp 00000000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f1353bf3000-7f1353df2000 ---p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f1353df2000-7f1353df3000 r--p 00016000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f1353df3000-7f1353df4000 rw-p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f1353df4000-7f1353e54000 r-xp 00000000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f1353e54000-7f1354054000 ---p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f1354054000-7f1354055000 r--p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f1354055000-7f1354056000 rw-p 00061000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f1354056000-7f135405a000 r-xp 00000000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f135405a000-7f1354259000 ---p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f1354259000-7f135425a000 r--p 00003000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f135425a000-7f135425b000 rw-p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f135425b000-7f135428c000 r-xp 00000000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f135428c000-7f135448b000 ---p 00031000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f135448b000-7f135448d000 r--p 00030000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f135448d000-7f135448e000 rw-p 00032000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f135448e000-7f1354491000 r-xp 00000000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f1354491000-7f1354690000 ---p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f1354690000-7f1354691000 r--p 00002000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f1354691000-7f1354692000 rw-p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f1354692000-7f135476b000 r-xp 00000000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f135476b000-7f135496a000 ---p 000d9000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f135496a000-7f1354978000 r--p 000d8000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f1354978000-7f135497b000 rw-p 000e6000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f135497b000-7f13549c5000 r-xp 00000000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f13549c5000-7f1354bc5000 ---p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f1354bc5000-7f1354bc6000 r--p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f1354bc6000-7f1354bc8000 rw-p 0004b000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f1354bc8000-7f1354bdd000 r-xp 00000000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f1354bdd000-7f1354ddc000 ---p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f1354ddc000-7f1354ddd000 r--p 00014000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f1354ddd000-7f1354dde000 rw-p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f1354dde000-7f1354e2c000 r-xp 00000000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f1354e2c000-7f135502c000 ---p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f135502c000-7f135502e000 r--p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f135502e000-7f135502f000 rw-p 00050000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f135502f000-7f1355045000 r-xp 00000000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f1355045000-7f1355244000 ---p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f1355244000-7f1355245000 r--p 00015000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f1355245000-7f1355246000 rw-p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f1355246000-7f1355248000 rw-p 00000000 00:00 0 7f1355248000-7f135524c000 r-xp 00000000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f135524c000-7f135544b000 ---p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f135544b000-7f135544c000 r--p 00003000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f135544c000-7f135544d000 rw-p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f135544d000-7f13554ca000 r-xp 00000000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f13554ca000-7f13556c9000 ---p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f13556c9000-7f13556ca000 r--p 0007c000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f13556ca000-7f13556cd000 rw-p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f13556cd000-7f13556ce000 rw-p 00000000 00:00 0 7f13556ce000-7f13556e2000 r-xp 00000000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f13556e2000-7f13558e1000 ---p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f13558e1000-7f13558e2000 r--p 00013000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f13558e2000-7f13558e3000 rw-p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f13558e3000-7f1355907000 r-xp 00000000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f1355907000-7f1355b06000 ---p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f1355b06000-7f1355b07000 r--p 00023000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f1355b07000-7f1355b08000 rw-p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f1355b08000-7f1355b0a000 rw-p 00000000 00:00 0 7f1355b0a000-7f1355b11000 r-xp 00000000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f1355b11000-7f1355d10000 ---p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f1355d10000-7f1355d11000 r--p 00006000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f1355d11000-7f1355d12000 rw-p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f1355d12000-7f1355d16000 r-xp 00000000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f1355d16000-7f1355f15000 ---p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f1355f15000-7f1355f16000 r--p 00003000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f1355f16000-7f1355f17000 rw-p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f1355f17000-7f1355f19000 r-xp 00000000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f1355f19000-7f1356118000 ---p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f1356118000-7f1356119000 r--p 00001000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f1356119000-7f135611a000 rw-p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f135611a000-7f13562dd000 r-xp 00000000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f13562dd000-7f13564dd000 ---p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f13564dd000-7f13564e1000 r--p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f13564e1000-7f13564e3000 rw-p 001c7000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f13564e3000-7f13564e8000 rw-p 00000000 00:00 0 7f13564e8000-7f13565e9000 r-xp 00000000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f13565e9000-7f13567e8000 ---p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f13567e8000-7f13567e9000 r--p 00100000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f13567e9000-7f13567ea000 rw-p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f13567ea000-7f13568d3000 r-xp 00000000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f13568d3000-7f1356ad2000 ---p 000e9000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f1356ad2000-7f1356ada000 r--p 000e8000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f1356ada000-7f1356adc000 rw-p 000f0000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f1356adc000-7f1356af1000 rw-p 00000000 00:00 0 7f1356af1000-7f1356af3000 r-xp 00000000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f1356af3000-7f1356cf3000 ---p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f1356cf3000-7f1356cf4000 r--p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f1356cf4000-7f1356cf5000 rw-p 00003000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f1356cf5000-7f1356f2b000 r-xp 00000000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f1356f2b000-7f135712b000 ---p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f135712b000-7f1357147000 r--p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f1357147000-7f1357154000 rw-p 00252000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f1357154000-7f1357158000 rw-p 00000000 00:00 0 7f1357158000-7f13571bf000 r-xp 00000000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f13571bf000-7f13573bf000 ---p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f13573bf000-7f13573c3000 r--p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f13573c3000-7f13573ca000 rw-p 0006b000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f13573ca000-7f13573f9000 r-xp 00000000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f13573f9000-7f13575f9000 ---p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f13575f9000-7f13575fa000 r--p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f13575fa000-7f13575fb000 rw-p 00030000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f13575fb000-7f1357603000 r-xp 00000000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f1357603000-7f1357802000 ---p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f1357802000-7f1357803000 r--p 00007000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f1357803000-7f1357804000 rw-p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f1357804000-7f1357832000 rw-p 00000000 00:00 0 7f1357832000-7f1357847000 r-xp 00000000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f1357847000-7f1357a46000 ---p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f1357a46000-7f1357a47000 r--p 00014000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f1357a47000-7f1357a48000 rw-p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f1357a48000-7f1357a49000 r-xp 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f1357a49000-7f1357c48000 ---p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f1357c48000-7f1357c49000 r--p 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f1357c49000-7f1357c4a000 rw-p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f1357c4a000-7f1357c6f000 r-xp 00000000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f1357c6f000-7f1357e6e000 ---p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f1357e6e000-7f1357e6f000 r--p 00024000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f1357e6f000-7f1357e70000 rw-p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f1357e70000-7f1357e87000 r-xp 00000000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f1357e87000-7f1358086000 ---p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f1358086000-7f1358087000 r--p 00016000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f1358087000-7f1358088000 rw-p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f1358088000-7f135808c000 rw-p 00000000 00:00 0 7f135808c000-7f13580ae000 r-xp 00000000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f13580b2000-7f13580b3000 ---p 00000000 00:00 0 7f13580b3000-7f13580fe000 rw-p 00000000 00:00 0 7f13580fe000-7f1358101000 rw-s 00000000 00:0b 135593 /[aio] (deleted) 7f1358101000-7f1358104000 rw-s 00000000 00:0b 135592 /[aio] (deleted) 7f1358104000-7f1358107000 rw-s 00000000 00:0b 135591 /[aio] (deleted) 7f1358107000-7f135810a000 rw-s 00000000 00:0b 135590 /[aio] (deleted) 7f135810a000-7f135810d000 rw-s 00000000 00:0b 135589 /[aio] (deleted) 7f135810d000-7f1358110000 rw-s 00000000 00:0b 135588 /[aio] (deleted) 7f1358110000-7f1358113000 rw-s 00000000 00:0b 135587 /[aio] (deleted) 7f1358113000-7f1358116000 rw-s 00000000 00:0b 135586 /[aio] (deleted) 7f1358116000-7f13582a4000 rw-p 00000000 00:00 0 7f13582a4000-7f13582a5000 rw-s 00000000 00:0b 135594 /[aio] (deleted) 7f13582a5000-7f13582a8000 rw-s 00000000 00:0b 135585 /[aio] (deleted) 7f13582a8000-7f13582ab000 rw-s 00000000 00:0b 135584 /[aio] (deleted) 7f13582ab000-7f13582ac000 rw-s 00000000 00:0b 135583 /[aio] (deleted) 7f13582ac000-7f13582ad000 rw-p 00000000 00:00 0 7f13582ad000-7f13582ae000 r--p 00021000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f13582ae000-7f13582af000 rw-p 00022000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f13582af000-7f13582b0000 rw-p 00000000 00:00 0 7ffccff85000-7ffccffa6000 rw-p 00000000 00:00 0 [stack] 7ffccffa9000-7ffccffab000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Fatal signal 6 while backtracing 2020-02-01 2:13:12 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 2:13:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 2:13:12 0 [Note] InnoDB: Uses event mutexes 2020-02-01 2:13:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 2:13:12 0 [Note] InnoDB: Number of pools: 1 2020-02-01 2:13:12 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 2:13:12 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 2:13:12 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 2:13:12 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 2:13:12 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137474624 2020-02-01 2:13:12 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 2:13:12 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 2:13:12 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 2:13:12 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 2:13:12 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 2:13:12 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 2:13:12 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 2:13:12 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137474677; transaction id 5906076 2020-02-01 2:13:12 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 2:13:12 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 2:13:12 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 2:13:12 0 [Note] Starting crash recovery... 2020-02-01 2:13:12 0 [Note] Crash recovery finished. 2020-02-01 2:13:12 0 [Note] Server socket created on IP: '::'. 2020-02-01 2:13:12 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 2:13:12 0 [Note] Added new Master_info '' to hash table 2020-02-01 2:13:12 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000176' at position 346 2020-02-01 2:13:12 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 2:13:12 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 2:13:12 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000176' at position 346 2020-02-01 2:13:12 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 2:13:12 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 2:13:12 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 2:13:12 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 2:13:12 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 2:13:12 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 2:13:12 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 2:13:12 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 2:13:12 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 2:13:12 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 2:13:14 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 2:13:14 2020-02-01 2:13:37 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 2:13:38 14 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 2:13:38 14 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 2:13:39 16 [ERROR] mysqld: Table './wpcomm305/comm305_options' is marked as crashed and should be repaired 2020-02-01 2:13:39 16 [Warning] Checking table: './wpcomm305/comm305_options' 2020-02-01 2:13:45 27 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 2:13:45 27 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 2:13:45 28 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 2:13:45 28 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 2:13:45 28 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 2:13:45 28 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 2:13:45 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=6 max_threads=202 thread_count=14 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f35f8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f3698079cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ed63cb6f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ed6374d3af] sigaction.c:0(__restore_rt)[0x7f36b25fe630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55ed6367b151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55ed637b9545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55ed6356fccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55ed63565d4b] /usr/sbin/mysqld(+0x6ca792)[0x55ed63570792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55ed635986da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ed6359a65d] /usr/sbin/mysqld(+0x65e3f3)[0x55ed635043f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55ed63503cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55ed63597b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55ed6359a65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55ed6359a761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55ed6359b377] /usr/sbin/mysqld(+0x58fd84)[0x55ed63435d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55ed63542b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55ed6355ab16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55ed6355acd2] /usr/sbin/mysqld(+0x6b55e7)[0x55ed6355b5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55ed6355b677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55ed6354a56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55ed6354c07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55ed63627afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55ed63627bdd] pthread_create.c:0(start_thread)[0x7f36b25f6ea5] /lib64/libc.so.6(clone+0x6d)[0x7f36b09978cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f35f800fec0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 29 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 2:13:51 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 2:13:51 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 2:13:51 0 [Note] InnoDB: Uses event mutexes 2020-02-01 2:13:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 2:13:51 0 [Note] InnoDB: Number of pools: 1 2020-02-01 2:13:51 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 2:13:51 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 2:13:52 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 2:13:52 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 2:13:52 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137477542 2020-02-01 2:13:52 0 [Note] InnoDB: Starting final batch to recover 1 pages from redo log. 2020-02-01 2:13:52 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 2:13:52 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 2:13:52 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 2:13:52 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 2:13:52 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 2:13:52 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 2:13:52 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 2:13:52 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137477992; transaction id 5906094 2020-02-01 2:13:52 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 2:13:52 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 2:13:52 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 2:13:52 0 [Note] Starting crash recovery... 2020-02-01 2:13:52 0 [Note] Crash recovery finished. 2020-02-01 2:13:52 0 [Note] Server socket created on IP: '::'. 2020-02-01 2:13:52 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 2:13:52 0 [Note] Added new Master_info '' to hash table 2020-02-01 2:13:52 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000178' at position 402414007 2020-02-01 2:13:52 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 2:13:52 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 2:13:52 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 2:13:52 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 2:13:52 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 2:13:52 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000178' at position 402414007 2020-02-01 2:13:52 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 2:13:52 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 2:13:52 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 2:13:52 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 2:13:52 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 2:13:52 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 2:13:52 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 2:13:53 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 2:13:53 2020-02-01 2:14:06 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000001, 312683) 2020-02-01 2:14:17 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 2:14:23 22 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 2:14:23 22 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 2:14:23 23 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 2:14:23 23 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 2:14:23 23 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 2:14:23 23 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 2:14:23 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9dc8000ac8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f9de600dcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5639e156ef2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5639e10053af] sigaction.c:0(__restore_rt)[0x7f9e8a4b0630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x5639e0f33151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x5639e1071545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x5639e0e27ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x5639e0e1dd4b] /usr/sbin/mysqld(+0x6ca792)[0x5639e0e28792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x5639e0e506da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5639e0e5265d] /usr/sbin/mysqld(+0x65e3f3)[0x5639e0dbc3f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x5639e0dbbcb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x5639e0e4fb24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x5639e0e5265d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x5639e0e52761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x5639e0e53377] /usr/sbin/mysqld(+0x58fd84)[0x5639e0cedd84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x5639e0dfab6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x5639e0e12b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x5639e0e12cd2] /usr/sbin/mysqld(+0x6b55e7)[0x5639e0e135e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x5639e0e13677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x5639e0e0256d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5639e0e0407c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5639e0edfafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5639e0edfbdd] pthread_create.c:0(start_thread)[0x7f9e8a4a8ea5] /lib64/libc.so.6(clone+0x6d)[0x7f9e888498cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f9dc800ff20): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 24 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 2:14:29 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 2:14:29 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 2:14:29 0 [Note] InnoDB: Uses event mutexes 2020-02-01 2:14:29 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 2:14:29 0 [Note] InnoDB: Number of pools: 1 2020-02-01 2:14:29 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 2:14:29 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 2:14:29 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 2:14:29 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 2:14:29 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137481041 2020-02-01 2:14:29 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 2:14:29 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 2:14:29 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 2:14:29 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 2:14:29 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 2:14:29 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 2:14:29 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137481050; transaction id 5906105 2020-02-01 2:14:29 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 2:14:29 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 2:14:29 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 2:14:29 0 [Note] Starting crash recovery... 2020-02-01 2:14:29 0 [Note] Crash recovery finished. 2020-02-01 2:14:30 0 [Note] Server socket created on IP: '::'. 2020-02-01 2:14:30 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 2:14:30 0 [Note] Added new Master_info '' to hash table 2020-02-01 2:14:30 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000178' at position 402414007 2020-02-01 2:14:30 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 2:14:30 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 2:14:30 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000178' at position 402414007 2020-02-01 2:14:30 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 2:14:30 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 2:14:30 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 2:14:30 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 2:14:30 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 2:14:31 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 2:14:31 2020-02-01 2:14:58 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 2:14:58 15 [ERROR] mysqld: Table './wpms/wp_options' is marked as crashed and should be repaired 2020-02-01 2:14:58 15 [Warning] Checking table: './wpms/wp_options' 2020-02-01 2:14:59 18 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 2:14:59 18 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 2:15:04 34 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 2:15:04 34 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 2:15:04 34 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 2:15:04 34 [Warning] Checking table: './ojsdata/controlled_vocabs' 2020-02-01 2:15:04 34 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 2:15:04 34 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 2:15:04 34 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 2:15:04 34 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 2:15:04 34 [ERROR] mysqld: Table './shockeripe/wp_options' is marked as crashed and should be repaired 2020-02-01 2:15:04 34 [Warning] Checking table: './shockeripe/wp_options' 2020-02-01 2:15:04 34 [ERROR] mysqld: Table './wpcomm303/wp_options' is marked as crashed and should be repaired 2020-02-01 2:15:04 34 [Warning] Checking table: './wpcomm303/wp_options' 2020-02-01 2:15:04 34 [ERROR] mysqld: Table './wpcomm306/wp_options' is marked as crashed and should be repaired 2020-02-01 2:15:04 34 [Warning] Checking table: './wpcomm306/wp_options' 2020-02-01 2:15:04 34 [ERROR] mysqld: Table './wpstart/wp_options' is marked as crashed and should be repaired 2020-02-01 2:15:04 34 [Warning] Checking table: './wpstart/wp_options' 200201 2:15:06 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=7 max_threads=202 thread_count=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f1724000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f168a1d5cf0 thread_stack 0x49000 *** buffer overflow detected ***: /usr/sbin/mysqld terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7f172cae0567] /lib64/libc.so.6(+0x1166e2)[0x7f172cade6e2] /lib64/libc.so.6(+0x1184c7)[0x7f172cae04c7] /usr/sbin/mysqld(my_addr_resolve+0xda)[0x55945b49194a] /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x55945b47a0c2] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55945af103af] /lib64/libpthread.so.0(+0xf630)[0x7f172e72d630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55945ae3e151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55945af7c545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55945ad32ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55945ad28d4b] /usr/sbin/mysqld(+0x6ca792)[0x55945ad33792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55945ad5b6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55945ad5d65d] /usr/sbin/mysqld(+0x65e3f3)[0x55945acc73f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55945acc6cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55945ad5ab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55945ad5d65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55945ad5d761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55945ad5e377] /usr/sbin/mysqld(+0x58fd84)[0x55945abf8d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55945ad05b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55945ad1db16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55945ad1dcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55945ad1e5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55945ad1e677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55945ad0d56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55945ad0f07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55945adeaafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55945adeabdd] /lib64/libpthread.so.0(+0x7ea5)[0x7f172e725ea5] /lib64/libc.so.6(clone+0x6d)[0x7f172cac68cd] ======= Memory map: ======== 55945a669000-55945bab7000 r-xp 00000000 fd:00 2101454 /usr/sbin/mysqld 55945bcb6000-55945bde2000 r--p 0144d000 fd:00 2101454 /usr/sbin/mysqld 55945bde2000-55945be9c000 rw-p 01579000 fd:00 2101454 /usr/sbin/mysqld 55945be9c000-55945c74b000 rw-p 00000000 00:00 0 55945d34f000-5594618a0000 rw-p 00000000 00:00 0 [heap] 7f1640000000-7f164003b000 rw-p 00000000 00:00 0 7f164003b000-7f1644000000 ---p 00000000 00:00 0 7f1648000000-7f1648021000 rw-p 00000000 00:00 0 7f1648021000-7f164c000000 ---p 00000000 00:00 0 7f164efb5000-7f164f41d000 r-xp 00000000 fd:00 2343798 /usr/lib64/mysql/plugin/ha_mroonga.so 7f164f41d000-7f164f61c000 ---p 00468000 fd:00 2343798 /usr/lib64/mysql/plugin/ha_mroonga.so 7f164f61c000-7f164f7a3000 r--p 00467000 fd:00 2343798 /usr/lib64/mysql/plugin/ha_mroonga.so 7f164f7a3000-7f164f7f8000 rw-p 005ee000 fd:00 2343798 /usr/lib64/mysql/plugin/ha_mroonga.so 7f164f7f8000-7f164f7ff000 rw-p 00000000 00:00 0 7f164f7ff000-7f164f800000 ---p 00000000 00:00 0 7f164f800000-7f1650000000 rw-p 00000000 00:00 0 7f1650000000-7f165002b000 rw-p 00000000 00:00 0 7f165002b000-7f1654000000 ---p 00000000 00:00 0 7f1654000000-7f1654021000 rw-p 00000000 00:00 0 7f1654021000-7f1658000000 ---p 00000000 00:00 0 7f1658000000-7f165812e000 rw-p 00000000 00:00 0 7f165812e000-7f165c000000 ---p 00000000 00:00 0 7f165c000000-7f165c021000 rw-p 00000000 00:00 0 7f165c021000-7f1660000000 ---p 00000000 00:00 0 7f1660000000-7f1660021000 rw-p 00000000 00:00 0 7f1660021000-7f1664000000 ---p 00000000 00:00 0 7f1664000000-7f16645ed000 rw-p 00000000 00:00 0 7f16645ed000-7f1668000000 ---p 00000000 00:00 0 7f16685e4000-7f16685f6000 r-xp 00000000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f16685f6000-7f16687f5000 ---p 00012000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f16687f5000-7f16687f8000 r--p 00011000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f16687f8000-7f16687f9000 rw-p 00014000 fd:00 2793608 /usr/lib64/libnss_myhostname.so.2 7f16687f9000-7f16687fa000 ---p 00000000 00:00 0 7f16687fa000-7f1668ffa000 rw-p 00000000 00:00 0 7f1668ffa000-7f1668ffb000 ---p 00000000 00:00 0 7f1668ffb000-7f16697fb000 rw-p 00000000 00:00 0 7f16697fb000-7f16697fc000 ---p 00000000 00:00 0 7f16697fc000-7f1669ffc000 rw-p 00000000 00:00 0 7f1669ffc000-7f1669ffd000 ---p 00000000 00:00 0 7f1669ffd000-7f166a7fd000 rw-p 00000000 00:00 0 7f166a7fd000-7f166a7fe000 ---p 00000000 00:00 0 7f166a7fe000-7f166affe000 rw-p 00000000 00:00 0 7f166affe000-7f166afff000 ---p 00000000 00:00 0 7f166afff000-7f166b7ff000 rw-p 00000000 00:00 0 7f166b7ff000-7f166b800000 ---p 00000000 00:00 0 7f166b800000-7f166c000000 rw-p 00000000 00:00 0 7f166c000000-7f166c6db000 rw-p 00000000 00:00 0 7f166c6db000-7f1670000000 ---p 00000000 00:00 0 7f1670000000-7f1673fec000 rw-p 00000000 00:00 0 7f1673fec000-7f1674000000 ---p 00000000 00:00 0 7f1674000000-7f16743f7000 rw-p 00000000 00:00 0 7f16743f7000-7f1678000000 ---p 00000000 00:00 0 7f1678000000-7f1678021000 rw-p 00000000 00:00 0 7f1678021000-7f167c000000 ---p 00000000 00:00 0 7f167c000000-7f167c021000 rw-p 00000000 00:00 0 7f167c021000-7f1680000000 ---p 00000000 00:00 0 7f16800b0000-7f16800b6000 r-xp 00000000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f16800b6000-7f16802b5000 ---p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f16802b5000-7f16802b6000 r--p 00005000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f16802b6000-7f16802b7000 rw-p 00006000 fd:00 2101230 /usr/lib64/libnss_dns-2.17.so 7f16802b7000-7f16802b8000 ---p 00000000 00:00 0 7f16802b8000-7f1681968000 rw-p 00000000 00:00 0 7f1681968000-7f1681969000 ---p 00000000 00:00 0 7f1681969000-7f1682169000 rw-p 00000000 00:00 0 7f1682169000-7f168216a000 ---p 00000000 00:00 0 7f168216a000-7f168296a000 rw-p 00000000 00:00 0 7f168296a000-7f168296b000 ---p 00000000 00:00 0 7f168296b000-7f168316b000 rw-p 00000000 00:00 0 7f168316b000-7f168316c000 ---p 00000000 00:00 0 7f168316c000-7f168396c000 rw-p 00000000 00:00 0 7f168396c000-7f168396d000 ---p 00000000 00:00 0 7f168396d000-7f168416d000 rw-p 00000000 00:00 0 7f168416d000-7f168416e000 ---p 00000000 00:00 0 7f168416e000-7f168496e000 rw-p 00000000 00:00 0 7f168496e000-7f168496f000 ---p 00000000 00:00 0 7f168496f000-7f168516f000 rw-p 00000000 00:00 0 7f168516f000-7f1685170000 ---p 00000000 00:00 0 7f1685170000-7f1685970000 rw-p 00000000 00:00 0 7f1685970000-7f1685971000 ---p 00000000 00:00 0 7f1685971000-7f1686171000 rw-p 00000000 00:00 0 7f1686171000-7f1686172000 ---p 00000000 00:00 0 7f1686172000-7f1686972000 rw-p 00000000 00:00 0 7f1686972000-7f1686973000 ---p 00000000 00:00 0 7f1686973000-7f1687173000 rw-p 00000000 00:00 0 7f1687173000-7f1687174000 ---p 00000000 00:00 0 7f1687174000-7f1687974000 rw-p 00000000 00:00 0 7f1687974000-7f1687975000 ---p 00000000 00:00 0 7f1687975000-7f1688175000 rw-p 00000000 00:00 0 7f1688175000-7f1688176000 ---p 00000000 00:00 0 7f1688176000-7f1688976000 rw-p 00000000 00:00 0 7f1688976000-7f1688977000 ---p 00000000 00:00 0 7f1688977000-7f168a0ad000 rw-p 00000000 00:00 0 7f168a141000-7f168a142000 ---p 00000000 00:00 0 7f168a142000-7f168a18c000 rw-p 00000000 00:00 0 7f168a18c000-7f168a18d000 ---p 00000000 00:00 0 7f168a18d000-7f168a1d7000 rw-p 00000000 00:00 0 7f168a1d7000-7f168a1d8000 ---p 00000000 00:00 0 7f168a1d8000-7f168a222000 rw-p 00000000 00:00 0 7f168a222000-7f168a223000 ---p 00000000 00:00 0 7f168a223000-7f168a26d000 rw-p 00000000 00:00 0 7f168a26d000-7f168a26e000 ---p 00000000 00:00 0 7f168a26e000-7f168aa6e000 rw-p 00000000 00:00 0 7f168aa6e000-7f168aa6f000 ---p 00000000 00:00 0 7f168aa6f000-7f168b26f000 rw-p 00000000 00:00 0 7f168b26f000-7f168b270000 ---p 00000000 00:00 0 7f168b270000-7f168ba70000 rw-p 00000000 00:00 0 7f168ba70000-7f168ba71000 ---p 00000000 00:00 0 7f168ba71000-7f168c271000 rw-p 00000000 00:00 0 7f168c271000-7f168e271000 rw-p 00000000 00:00 0 7f168e271000-7f1690000000 rw-p 00000000 00:00 0 7f1690000000-7f1710000000 rw-p 00000000 00:00 0 7f1710000000-7f17103f7000 rw-p 00000000 00:00 0 7f17103f7000-7f1714000000 ---p 00000000 00:00 0 7f1714030000-7f1714031000 ---p 00000000 00:00 0 7f1714031000-7f171407b000 rw-p 00000000 00:00 0 7f171407b000-7f171407c000 ---p 00000000 00:00 0 7f171407c000-7f17140c6000 rw-p 00000000 00:00 0 7f17140c6000-7f17140c7000 ---p 00000000 00:00 0 7f17140c7000-7f1724000000 rw-p 00000000 00:00 0 7f1724000000-7f17243f7000 rw-p 00000000 00:00 0 7f17243f7000-7f1728000000 ---p 00000000 00:00 0 7f1728032000-7f1728033000 ---p 00000000 00:00 0 7f1728033000-7f172807d000 rw-p 00000000 00:00 0 7f172807d000-7f172807e000 ---p 00000000 00:00 0 7f172807e000-7f17280c8000 rw-p 00000000 00:00 0 7f17280c8000-7f17280c9000 ---p 00000000 00:00 0 7f17280c9000-7f1728113000 rw-p 00000000 00:00 0 7f1728113000-7f172811f000 r-xp 00000000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f172811f000-7f172831e000 ---p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f172831e000-7f172831f000 r--p 0000b000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f172831f000-7f1728320000 rw-p 0000c000 fd:00 2101435 /usr/lib64/libnss_files-2.17.so 7f1728320000-7f1728326000 rw-p 00000000 00:00 0 7f172832e000-7f172872f000 rw-p 00000000 00:00 0 7f172872f000-7f1728730000 ---p 00000000 00:00 0 7f1728730000-7f1729665000 rw-p 00000000 00:00 0 7f1729665000-7f1729666000 ---p 00000000 00:00 0 7f1729666000-7f1729e66000 rw-p 00000000 00:00 0 7f1729e66000-7f1729e69000 r-xp 00000000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f1729e69000-7f172a068000 ---p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f172a068000-7f172a069000 r--p 00002000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f172a069000-7f172a06a000 rw-p 00003000 fd:00 2181832 /usr/lib64/libkeyutils.so.1.5 7f172a06a000-7f172a078000 r-xp 00000000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f172a078000-7f172a278000 ---p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f172a278000-7f172a279000 r--p 0000e000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f172a279000-7f172a27a000 rw-p 0000f000 fd:00 2101514 /usr/lib64/libkrb5support.so.0.1 7f172a27a000-7f172a289000 r-xp 00000000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f172a289000-7f172a488000 ---p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f172a488000-7f172a489000 r--p 0000e000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f172a489000-7f172a48a000 rw-p 0000f000 fd:00 2162174 /usr/lib64/libbz2.so.1.0.6 7f172a48a000-7f172a4a1000 r-xp 00000000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f172a4a1000-7f172a6a0000 ---p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f172a6a0000-7f172a6a1000 r--p 00016000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f172a6a1000-7f172a6a2000 rw-p 00017000 fd:00 2104626 /usr/lib64/libelf-0.176.so 7f172a6a2000-7f172a702000 r-xp 00000000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f172a702000-7f172a902000 ---p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f172a902000-7f172a903000 r--p 00060000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f172a903000-7f172a904000 rw-p 00061000 fd:00 2127081 /usr/lib64/libpcre.so.1.2.0 7f172a904000-7f172a908000 r-xp 00000000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f172a908000-7f172ab07000 ---p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f172ab07000-7f172ab08000 r--p 00003000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f172ab08000-7f172ab09000 rw-p 00004000 fd:00 2162118 /usr/lib64/libattr.so.1.1.0 7f172ab09000-7f172ab3a000 r-xp 00000000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f172ab3a000-7f172ad39000 ---p 00031000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f172ad39000-7f172ad3b000 r--p 00030000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f172ad3b000-7f172ad3c000 rw-p 00032000 fd:00 2101470 /usr/lib64/libk5crypto.so.3.1 7f172ad3c000-7f172ad3f000 r-xp 00000000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f172ad3f000-7f172af3e000 ---p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f172af3e000-7f172af3f000 r--p 00002000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f172af3f000-7f172af40000 rw-p 00003000 fd:00 2101221 /usr/lib64/libcom_err.so.2.1 7f172af40000-7f172b019000 r-xp 00000000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f172b019000-7f172b218000 ---p 000d9000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f172b218000-7f172b226000 r--p 000d8000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f172b226000-7f172b229000 rw-p 000e6000 fd:00 2101512 /usr/lib64/libkrb5.so.3.3 7f172b229000-7f172b273000 r-xp 00000000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f172b273000-7f172b473000 ---p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f172b473000-7f172b474000 r--p 0004a000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f172b474000-7f172b476000 rw-p 0004b000 fd:00 2101223 /usr/lib64/libgssapi_krb5.so.2.2 7f172b476000-7f172b48b000 r-xp 00000000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f172b48b000-7f172b68a000 ---p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f172b68a000-7f172b68b000 r--p 00014000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f172b68b000-7f172b68c000 rw-p 00015000 fd:00 2227353 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f172b68c000-7f172b6da000 r-xp 00000000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f172b6da000-7f172b8da000 ---p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f172b8da000-7f172b8dc000 r--p 0004e000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f172b8dc000-7f172b8dd000 rw-p 00050000 fd:00 2127032 /usr/lib64/libdw-0.176.so 7f172b8dd000-7f172b8f3000 r-xp 00000000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f172b8f3000-7f172baf2000 ---p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f172baf2000-7f172baf3000 r--p 00015000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f172baf3000-7f172baf4000 rw-p 00016000 fd:00 2101449 /usr/lib64/libresolv-2.17.so 7f172baf4000-7f172baf6000 rw-p 00000000 00:00 0 7f172baf6000-7f172bafa000 r-xp 00000000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f172bafa000-7f172bcf9000 ---p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f172bcf9000-7f172bcfa000 r--p 00003000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f172bcfa000-7f172bcfb000 rw-p 00004000 fd:00 2162270 /usr/lib64/libgpg-error.so.0.10.0 7f172bcfb000-7f172bd78000 r-xp 00000000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f172bd78000-7f172bf77000 ---p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f172bf77000-7f172bf78000 r--p 0007c000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f172bf78000-7f172bf7b000 rw-p 0007d000 fd:00 2162284 /usr/lib64/libgcrypt.so.11.8.2 7f172bf7b000-7f172bf7c000 rw-p 00000000 00:00 0 7f172bf7c000-7f172bf90000 r-xp 00000000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f172bf90000-7f172c18f000 ---p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f172c18f000-7f172c190000 r--p 00013000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f172c190000-7f172c191000 rw-p 00014000 fd:00 2147379 /usr/lib64/liblz4.so.1.7.5 7f172c191000-7f172c1b5000 r-xp 00000000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f172c1b5000-7f172c3b4000 ---p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f172c3b4000-7f172c3b5000 r--p 00023000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f172c3b5000-7f172c3b6000 rw-p 00024000 fd:00 2101234 /usr/lib64/libselinux.so.1 7f172c3b6000-7f172c3b8000 rw-p 00000000 00:00 0 7f172c3b8000-7f172c3bf000 r-xp 00000000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f172c3bf000-7f172c5be000 ---p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f172c5be000-7f172c5bf000 r--p 00006000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f172c5bf000-7f172c5c0000 rw-p 00007000 fd:00 2101451 /usr/lib64/librt-2.17.so 7f172c5c0000-7f172c5c4000 r-xp 00000000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f172c5c4000-7f172c7c3000 ---p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f172c7c3000-7f172c7c4000 r--p 00003000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f172c7c4000-7f172c7c5000 rw-p 00004000 fd:00 2101236 /usr/lib64/libcap.so.2.22 7f172c7c5000-7f172c7c7000 r-xp 00000000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f172c7c7000-7f172c9c6000 ---p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f172c9c6000-7f172c9c7000 r--p 00001000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f172c9c7000-7f172c9c8000 rw-p 00002000 fd:00 2227367 /usr/lib64/libfreebl3.so 7f172c9c8000-7f172cb8b000 r-xp 00000000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f172cb8b000-7f172cd8b000 ---p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f172cd8b000-7f172cd8f000 r--p 001c3000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f172cd8f000-7f172cd91000 rw-p 001c7000 fd:00 2097970 /usr/lib64/libc-2.17.so 7f172cd91000-7f172cd96000 rw-p 00000000 00:00 0 7f172cd96000-7f172ce97000 r-xp 00000000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f172ce97000-7f172d096000 ---p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f172d096000-7f172d097000 r--p 00100000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f172d097000-7f172d098000 rw-p 00101000 fd:00 2097978 /usr/lib64/libm-2.17.so 7f172d098000-7f172d181000 r-xp 00000000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f172d181000-7f172d380000 ---p 000e9000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f172d380000-7f172d388000 r--p 000e8000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f172d388000-7f172d38a000 rw-p 000f0000 fd:00 2104629 /usr/lib64/libstdc++.so.6.0.19 7f172d38a000-7f172d39f000 rw-p 00000000 00:00 0 7f172d39f000-7f172d3a1000 r-xp 00000000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f172d3a1000-7f172d5a1000 ---p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f172d5a1000-7f172d5a2000 r--p 00002000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f172d5a2000-7f172d5a3000 rw-p 00003000 fd:00 2097976 /usr/lib64/libdl-2.17.so 7f172d5a3000-7f172d7d9000 r-xp 00000000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f172d7d9000-7f172d9d9000 ---p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f172d9d9000-7f172d9f5000 r--p 00236000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f172d9f5000-7f172da02000 rw-p 00252000 fd:00 2101258 /usr/lib64/libcrypto.so.1.0.2k 7f172da02000-7f172da06000 rw-p 00000000 00:00 0 7f172da06000-7f172da6d000 r-xp 00000000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f172da6d000-7f172dc6d000 ---p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f172dc6d000-7f172dc71000 r--p 00067000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f172dc71000-7f172dc78000 rw-p 0006b000 fd:00 2101267 /usr/lib64/libssl.so.1.0.2k 7f172dc78000-7f172dca7000 r-xp 00000000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f172dca7000-7f172dea7000 ---p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f172dea7000-7f172dea8000 r--p 0002f000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f172dea8000-7f172dea9000 rw-p 00030000 fd:00 2170532 /usr/lib64/libsystemd.so.0.6.0 7f172dea9000-7f172deb1000 r-xp 00000000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f172deb1000-7f172e0b0000 ---p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f172e0b0000-7f172e0b1000 r--p 00007000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f172e0b1000-7f172e0b2000 rw-p 00008000 fd:00 2097974 /usr/lib64/libcrypt-2.17.so 7f172e0b2000-7f172e0e0000 rw-p 00000000 00:00 0 7f172e0e0000-7f172e0f5000 r-xp 00000000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f172e0f5000-7f172e2f4000 ---p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f172e2f4000-7f172e2f5000 r--p 00014000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f172e2f5000-7f172e2f6000 rw-p 00015000 fd:00 2101237 /usr/lib64/libz.so.1.2.7 7f172e2f6000-7f172e2f7000 r-xp 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f172e2f7000-7f172e4f6000 ---p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f172e4f6000-7f172e4f7000 r--p 00000000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f172e4f7000-7f172e4f8000 rw-p 00001000 fd:00 2343831 /usr/lib64/libaio.so.1.0.1 7f172e4f8000-7f172e51d000 r-xp 00000000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f172e51d000-7f172e71c000 ---p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f172e71c000-7f172e71d000 r--p 00024000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f172e71d000-7f172e71e000 rw-p 00025000 fd:00 2162139 /usr/lib64/liblzma.so.5.2.2 7f172e71e000-7f172e735000 r-xp 00000000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f172e735000-7f172e934000 ---p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f172e934000-7f172e935000 r--p 00016000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f172e935000-7f172e936000 rw-p 00017000 fd:00 2101447 /usr/lib64/libpthread-2.17.so 7f172e936000-7f172e93a000 rw-p 00000000 00:00 0 7f172e93a000-7f172e95c000 r-xp 00000000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f172e960000-7f172e961000 rw-p 00000000 00:00 0 7f172e961000-7f172e962000 ---p 00000000 00:00 0 7f172e962000-7f172e9ac000 rw-p 00000000 00:00 0 7f172e9ac000-7f172e9af000 rw-s 00000000 00:0b 151651 /[aio] (deleted) 7f172e9af000-7f172e9b2000 rw-s 00000000 00:0b 151650 /[aio] (deleted) 7f172e9b2000-7f172e9b5000 rw-s 00000000 00:0b 151649 /[aio] (deleted) 7f172e9b5000-7f172e9b8000 rw-s 00000000 00:0b 151648 /[aio] (deleted) 7f172e9b8000-7f172e9bb000 rw-s 00000000 00:0b 151647 /[aio] (deleted) 7f172e9bb000-7f172e9be000 rw-s 00000000 00:0b 151646 /[aio] (deleted) 7f172e9be000-7f172e9c1000 rw-s 00000000 00:0b 151645 /[aio] (deleted) 7f172e9c1000-7f172e9c4000 rw-s 00000000 00:0b 151644 /[aio] (deleted) 7f172e9c4000-7f172eb52000 rw-p 00000000 00:00 0 7f172eb52000-7f172eb53000 rw-s 00000000 00:0b 151652 /[aio] (deleted) 7f172eb53000-7f172eb56000 rw-s 00000000 00:0b 151643 /[aio] (deleted) 7f172eb56000-7f172eb59000 rw-s 00000000 00:0b 151642 /[aio] (deleted) 7f172eb59000-7f172eb5a000 rw-s 00000000 00:0b 151641 /[aio] (deleted) 7f172eb5a000-7f172eb5b000 rw-p 00000000 00:00 0 7f172eb5b000-7f172eb5c000 r--p 00021000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f172eb5c000-7f172eb5d000 rw-p 00022000 fd:00 2227369 /usr/lib64/ld-2.17.so 7f172eb5d000-7f172eb5e000 rw-p 00000000 00:00 0 7ffdd285d000-7ffdd287e000 rw-p 00000000 00:00 0 [stack] 7ffdd29b9000-7ffdd29bb000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Fatal signal 6 while backtracing 2020-02-01 2:15:14 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 2:15:14 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 2:15:14 0 [Note] InnoDB: Uses event mutexes 2020-02-01 2:15:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 2:15:14 0 [Note] InnoDB: Number of pools: 1 2020-02-01 2:15:14 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 2:15:14 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 2:15:14 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 2:15:14 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 2:15:14 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137585017 2020-02-01 2:15:14 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 2:15:14 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 2:15:14 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 2:15:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 2:15:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 2:15:14 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 2:15:14 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 2:15:14 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137585054; transaction id 5906479 2020-02-01 2:15:14 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 2:15:14 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 2:15:14 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 2:15:14 0 [Note] Starting crash recovery... 2020-02-01 2:15:14 0 [Note] Crash recovery finished. 2020-02-01 2:15:14 0 [Note] Server socket created on IP: '::'. 2020-02-01 2:15:14 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 2:15:14 0 [Note] Added new Master_info '' to hash table 2020-02-01 2:15:14 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 2:15:14 2020-02-01 2:15:14 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000178' at position 402414007 2020-02-01 2:15:14 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 2:15:14 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 2:15:14 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 2:15:14 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 2:15:14 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 2:15:14 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 2:15:14 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 2:15:14 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000178' at position 402414007 2020-02-01 2:15:24 12 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000003, 338201) 2020-02-01 2:15:43 14 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 2:15:48 17 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 2:15:48 17 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 2:15:50 20 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 2:15:50 20 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 2:15:50 21 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 2:15:50 21 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 2:15:50 21 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 2:15:50 21 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 2:15:50 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f81b8000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f81c61d5cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55901f319f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55901edb03af] sigaction.c:0(__restore_rt)[0x7f826a693630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55901ecde151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55901ee1c545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55901ebd2ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55901ebc8d4b] /usr/sbin/mysqld(+0x6ca792)[0x55901ebd3792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55901ebfb6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55901ebfd65d] /usr/sbin/mysqld(+0x65e3f3)[0x55901eb673f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55901eb66cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55901ebfab24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55901ebfd65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55901ebfd761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55901ebfe377] /usr/sbin/mysqld(+0x58fd84)[0x55901ea98d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55901eba5b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55901ebbdb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55901ebbdcd2] /usr/sbin/mysqld(+0x6b55e7)[0x55901ebbe5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55901ebbe677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55901ebad56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55901ebaf07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55901ec8aafa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55901ec8abdd] pthread_create.c:0(start_thread)[0x7f826a68bea5] /lib64/libc.so.6(clone+0x6d)[0x7f8268a2c8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f81b800fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 22 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 2:15:56 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 2:15:56 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 2:15:56 0 [Note] InnoDB: Uses event mutexes 2020-02-01 2:15:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 2:15:56 0 [Note] InnoDB: Number of pools: 1 2020-02-01 2:15:56 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 2:15:56 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 2:15:56 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 2:15:56 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 2:15:56 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137588752 2020-02-01 2:15:56 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 2:15:56 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 2:15:56 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 2:15:56 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 2:15:56 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 2:15:56 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 2:15:56 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137588761; transaction id 5906490 2020-02-01 2:15:56 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 2:15:56 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 2:15:56 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 2:15:56 0 [Note] Starting crash recovery... 2020-02-01 2:15:56 0 [Note] Crash recovery finished. 2020-02-01 2:15:56 0 [Note] Server socket created on IP: '::'. 2020-02-01 2:15:56 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 2:15:56 0 [Note] Added new Master_info '' to hash table 2020-02-01 2:15:56 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000178' at position 402414007 2020-02-01 2:15:56 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000010' at position 22535, relay log './dbsvc-528-relay-bin.000079' position: 15163 2020-02-01 2:15:56 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 2:15:56 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000178' at position 402414007 2020-02-01 2:15:56 11 [ERROR] mysqld: Table './wpwsutv/wp_options' is marked as crashed and should be repaired 2020-02-01 2:15:56 11 [Warning] Checking table: './wpwsutv/wp_options' 2020-02-01 2:15:56 11 [Warning] wpwsutv.wp_options: 1 client is using or hasn't closed the table properly 2020-02-01 2:15:56 11 [Note] wpwsutv.wp_options: Table is fixed 2020-02-01 2:15:56 10 [ERROR] Error reading packet from server: Could not find first log file name in binary log index file (server_errno=1236) 2020-02-01 2:15:56 10 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file', Internal MariaDB error code: 1236 2020-02-01 2:15:56 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000178', position 402414007 2020-02-01 2:15:56 11 [ERROR] mysqld: Can't find record in 'wp_options' 2020-02-01 2:15:56 11 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wpwsutv.wp_options; Table './wpwsutv/wp_options' is marked as crashed and should be repaired, Error_code: 145; Can't find record in 'wp_options', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log dbsvc-527-bin.000010, end_log_pos 22857, Gtid 0-1-2496631, Internal MariaDB error code: 1032 2020-02-01 2:15:56 11 [Warning] Slave: Table './wpwsutv/wp_options' is marked as crashed and should be repaired Error_code: 145 2020-02-01 2:15:56 11 [Warning] Slave: Can't find record in 'wp_options' Error_code: 1032 2020-02-01 2:15:56 11 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'dbsvc-527-bin.000010' position 22535 2020-02-01 2:15:56 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000010' at position 22535 2020-02-01 2:15:56 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 2:15:56 2020-02-01 2:16:23 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 2:16:24 14 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2020-02-01 2:16:24 14 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='dbsvc-527.wichita.edu', master_port='3306', master_log_file='dbsvc-527-bin.000178', master_log_pos='402414007'. New state master_host='dbsvc-527.wichita.edu', master_port='3306', master_log_file='dbsvc-527-bin.000001', master_log_pos='332'. 2020-02-01 2:16:25 14 [Note] Master connection name: '' Master_info_file: 'master.info' Relay_info_file: 'relay-log.info' 2020-02-01 2:16:25 14 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='dbsvc-527.wichita.edu', master_port='3306', master_log_file='dbsvc-527-bin.000001', master_log_pos='332'. New state master_host='dbsvc-527.wichita.edu', master_port='3306', master_log_file='dbsvc-527-bin.000001', master_log_pos='332'. 2020-02-01 2:16:34 22 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 2:16:34 22 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 2:16:34 23 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 2:16:34 23 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 2:16:34 23 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 2:16:34 23 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 2:16:34 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=202 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2a34000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2a57ffacf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56220abc7f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56220a65e3af] sigaction.c:0(__restore_rt)[0x7f2afcacf630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x56220a58c151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x56220a6ca545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x56220a480ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x56220a476d4b] /usr/sbin/mysqld(+0x6ca792)[0x56220a481792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x56220a4a96da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56220a4ab65d] /usr/sbin/mysqld(+0x65e3f3)[0x56220a4153f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x56220a414cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x56220a4a8b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x56220a4ab65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x56220a4ab761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56220a4ac377] /usr/sbin/mysqld(+0x58fd84)[0x56220a346d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x56220a453b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x56220a46bb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x56220a46bcd2] /usr/sbin/mysqld(+0x6b55e7)[0x56220a46c5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x56220a46c677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x56220a45b56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56220a45d07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x56220a538afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56220a538bdd] pthread_create.c:0(start_thread)[0x7f2afcac7ea5] /lib64/libc.so.6(clone+0x6d)[0x7f2afae688cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f2a34010ad0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 24 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 2:16:40 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 2:16:40 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 2:16:40 0 [Note] InnoDB: Uses event mutexes 2020-02-01 2:16:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 2:16:40 0 [Note] InnoDB: Number of pools: 1 2020-02-01 2:16:40 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 2:16:40 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 2:16:40 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 2:16:40 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 2:16:40 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137589100 2020-02-01 2:16:40 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 2:16:40 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 2:16:40 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 2:16:40 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 2:16:40 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 2:16:40 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 2:16:40 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 2:16:41 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137589109; transaction id 5906495 2020-02-01 2:16:41 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 2:16:41 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 2:16:41 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 2:16:41 0 [Note] Starting crash recovery... 2020-02-01 2:16:41 0 [Note] Crash recovery finished. 2020-02-01 2:16:41 0 [Note] Server socket created on IP: '::'. 2020-02-01 2:16:41 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 2:16:41 0 [Note] Added new Master_info '' to hash table 2020-02-01 2:16:41 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000001' at position 332 2020-02-01 2:16:41 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000001' at position 332, relay log './dbsvc-528-relay-bin.000001' position: 4 2020-02-01 2:16:41 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 2:16:41 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000001' at position 332 2020-02-01 2:16:41 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 2:16:41 2020-02-01 2:16:50 13 [Note] Start binlog_dump to slave_server(1), pos(dbsvc-528-bin.000005, 25710) 2020-02-01 2:16:54 11 [Note] Error reading relay log event: slave SQL thread was killed 2020-02-01 2:16:54 11 [Note] Slave SQL thread exiting, replication stopped in log 'dbsvc-527-bin.000001' at position 332 2020-02-01 2:16:54 10 [Note] Slave I/O thread exiting, read up to log 'dbsvc-527-bin.000001', position 332 2020-02-01 2:16:59 12 [Note] Deleted Master_info file '/data/mysql/master.info'. 2020-02-01 2:16:59 12 [Note] Deleted Master_info file '/data/mysql/relay-log.info'. 2020-02-01 2:17:07 19 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log '' at position 4 2020-02-01 2:17:07 20 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './dbsvc-528-relay-bin.000001' position: 4 2020-02-01 2:17:07 19 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'FIRST' at position 4 2020-02-01 2:17:08 21 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 2:17:09 22 [ERROR] mysqld: Table './ksarn/wp_options' is marked as crashed and should be repaired 2020-02-01 2:17:09 22 [Warning] Checking table: './ksarn/wp_options' 2020-02-01 2:17:10 24 [ERROR] mysqld: Table './idtgroup/wp_options' is marked as crashed and should be repaired 2020-02-01 2:17:10 24 [Warning] Checking table: './idtgroup/wp_options' 2020-02-01 2:17:17 33 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 2:17:17 33 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 2:17:17 34 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 2:17:17 34 [Warning] Checking table: './ojsdata/scheduled_tasks' 2020-02-01 2:17:17 35 [ERROR] mysqld: Table './ojsdata/controlled_vocabs' is marked as crashed and should be repaired 2020-02-01 2:17:17 35 [Warning] Checking table: './ojsdata/controlled_vocabs' 200201 2:17:17 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=9 max_threads=202 thread_count=18 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7ff6f00083d8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff733ececf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c768f98f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c768a2f3af] sigaction.c:0(__restore_rt)[0x7ff7d850e630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x55c76895d151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x55c768a9b545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x55c768851ccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x55c768847d4b] /usr/sbin/mysqld(+0x6ca792)[0x55c768852792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x55c76887a6da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c76887c65d] /usr/sbin/mysqld(+0x65e3f3)[0x55c7687e63f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55c7687e5cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x55c768879b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x55c76887c65d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x55c76887c761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55c76887d377] /usr/sbin/mysqld(+0x58fd84)[0x55c768717d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x55c768824b6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55c76883cb16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55c76883ccd2] /usr/sbin/mysqld(+0x6b55e7)[0x55c76883d5e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55c76883d677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x55c76882c56d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55c76882e07c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55c768909afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c768909bdd] pthread_create.c:0(start_thread)[0x7ff7d8506ea5] /lib64/libc.so.6(clone+0x6d)[0x7ff7d68a78cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff6f0017830): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 38 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 2:17:23 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 2:17:23 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 2:17:23 0 [Note] InnoDB: Uses event mutexes 2020-02-01 2:17:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 2:17:23 0 [Note] InnoDB: Number of pools: 1 2020-02-01 2:17:23 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 2:17:23 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 2:17:23 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 2:17:23 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 2:17:23 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137589616 2020-02-01 2:17:23 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 2:17:23 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 2:17:23 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 2:17:23 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 2:17:23 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 2:17:23 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 2:17:23 0 [Note] InnoDB: Waiting for purge to start 2020-02-01 2:17:23 0 [Note] InnoDB: 10.4.10 started; log sequence number 5137589625; transaction id 5906505 2020-02-01 2:17:23 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2020-02-01 2:17:23 0 [Note] Plugin 'FEEDBACK' is disabled. 2020-02-01 2:17:23 0 [Note] Recovering after a crash using dbsvc-528-bin 2020-02-01 2:17:23 0 [Note] Starting crash recovery... 2020-02-01 2:17:23 0 [Note] Crash recovery finished. 2020-02-01 2:17:23 0 [Note] Server socket created on IP: '::'. 2020-02-01 2:17:23 0 [Note] Reading of all Master_info entries succeeded 2020-02-01 2:17:23 0 [Note] Added new Master_info '' to hash table 2020-02-01 2:17:23 10 [Note] Slave I/O thread: Start asynchronous replication to master 'replication@dbsvc-527.wichita.edu:3306' in log 'dbsvc-527-bin.000001' at position 332 2020-02-01 2:17:23 11 [Note] Slave SQL thread initialized, starting replication in log 'dbsvc-527-bin.000001' at position 332, relay log './dbsvc-528-relay-bin.000002' position: 635 2020-02-01 2:17:23 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.4.10-MariaDB-log' socket: '/data/mysql/mysql.sock' port: 3306 MariaDB Server 2020-02-01 2:17:23 0 [Note] InnoDB: Buffer pool(s) load completed at 200201 2:17:23 2020-02-01 2:17:24 10 [Note] Slave I/O thread: connected to master 'replication@dbsvc-527.wichita.edu:3306',replication started in log 'dbsvc-527-bin.000001' at position 332 2020-02-01 2:17:48 12 [Warning] IP address '10.0.59.19' could not be resolved: Name or service not known 2020-02-01 2:17:52 14 [ERROR] mysqld: Table './ojsdata/sessions' is marked as crashed and should be repaired 2020-02-01 2:17:52 14 [Warning] Checking table: './ojsdata/sessions' 2020-02-01 2:17:52 15 [ERROR] mysqld: Table './ojsdata/scheduled_tasks' is marked as crashed and should be repaired 2020-02-01 2:17:52 15 [Warning] Checking table: './ojsdata/scheduled_tasks' 200201 2:17:52 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=3 max_threads=202 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2370000a88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2418a47cf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560b5ede3f2e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560b5e87a3af] sigaction.c:0(__restore_rt)[0x7f2434cca630] /usr/sbin/mysqld(_ZNK26Type_handler_string_result13Item_const_eqEPK10Item_constS2_b+0x41)[0x560b5e7a8151] /usr/sbin/mysqld(_ZNK9Item_func2eqEPK4Itemb+0xf5)[0x560b5e8e6545] /usr/sbin/mysqld(_ZN17Item_func_between14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x16f)[0x560b5e69cccf] /usr/sbin/mysqld(_ZN13Item_cond_and14add_key_fieldsEP4JOINPP9KEY_FIELDPjyPP14SARGABLE_PARAM+0x5b)[0x560b5e692d4b] /usr/sbin/mysqld(+0x6ca792)[0x560b5e69d792] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x17ba)[0x560b5e6c56da] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560b5e6c765d] /usr/sbin/mysqld(+0x65e3f3)[0x560b5e6313f3] /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x560b5e630cb4] /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0xc04)[0x560b5e6c4b24] /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x9d)[0x560b5e6c765d] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xe1)[0x560b5e6c7761] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x560b5e6c8377] /usr/sbin/mysqld(+0x58fd84)[0x560b5e562d84] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x449a)[0x560b5e66fb6a] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x560b5e687b16] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x560b5e687cd2] /usr/sbin/mysqld(+0x6b55e7)[0x560b5e6885e7] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x560b5e688677] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x17dd)[0x560b5e67756d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x560b5e67907c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x560b5e754afa] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560b5e754bdd] pthread_create.c:0(start_thread)[0x7f2434cc2ea5] /lib64/libc.so.6(clone+0x6d)[0x7f24330638cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f237000fee0): select statistics.*, COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0) AS submission_declined_total, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / statistics.submission_received * 100 AS submission_rejection_rate, statistics.submission_received / years.count AS avg_submission_received, statistics.submission_accepted / years.count AS avg_submission_accepted, statistics.submission_published / years.count AS avg_submission_published, statistics.submission_declined_initial / years.count AS avg_submission_declined_initial, statistics.submission_declined / years.count AS avg_submission_declined, statistics.submission_declined_other / years.count AS avg_submission_declined_other, (COALESCE(statistics.submission_declined_initial, 0) + COALESCE(statistics.submission_declined, 0) + COALESCE(statistics.submission_declined_other, 0)) / years.count AS avg_submission_declined_total from (select COUNT(active_s.submission_id) AS active_total, COUNT(CASE WHEN active_s.stage_id = 1 THEN 0 END) AS active_submission, COUNT(CASE WHEN active_s.stage_id = 2 THEN 0 END) AS active_internal_review, COUNT(CASE WHEN active_s.stage_id = 3 THEN 0 END) AS active_external_review, COUNT(CASE WHEN active_s.stage_id = 4 THEN 0 END) AS active_copy_editing, COUNT(CASE WHEN active_s.stage_id = 5 THEN 0 END) AS active_production, COUNT(ranged_s.submission_id) AS submission_received, COUNT(CASE WHEN ranged_ed.decision IN (1, 7) THEN 0 END) AS submission_accepted, COUNT(CASE WHEN s.status = 3 AND EXISTS ( SELECT 0 FROM published_submissions ps WHERE ps.submission_id = ranged_s.submission_id ) THEN 0 END) AS submission_published, COUNT(CASE WHEN ranged_ed.decision = 9 THEN 0 END) AS submission_declined_initial, COUNT(CASE WHEN ranged_ed.decision = 4 THEN 0 END) AS submission_declined, COUNT( CASE WHEN ranged_s.status = 4 AND (ed.decision NOT IN (4, 9) OR ed.decision IS NULL) THEN 0 END ) AS submission_declined_other, AVG( CASE WHEN ranged_ed.decision IN (7, 1) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_accept, AVG( CASE WHEN ranged_ed.decision IN (4, 9) THEN DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified)) END ) AS submission_days_to_reject, AVG(DATEDIFF(first_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_first_decision, AVG(DATEDIFF(ranged_ed.date_decided,COALESCE(s.date_submitted, s.last_modified))) AS submission_days_to_decide, COUNT(CASE WHEN ranged_s_ed.decision IN (1, 7) THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_acceptance_rate, COUNT(CASE WHEN ranged_s_ed.decision = 9 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_initial_rate, COUNT(CASE WHEN ranged_s_ed.decision = 4 THEN 0 END) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_rate, COUNT( CASE WHEN ranged_s.status = 4 AND ( ranged_s_ed.decision NOT IN (4, 9) OR ranged_s_ed.decision IS NULL ) THEN 0 END ) / COUNT(ranged_s.submission_id) * 100 AS submission_declined_other_rate from `submissions` as `s` left join `submissions` as `active_s` on `active_s`.`submission_id` = `s`.`submission_id` and `active_s`.`status` = 1 left join `submissions` as `ranged_s` on `ranged_s`.`submission_id` = `s`.`submission_id` and ranged_s.date_submitted BETWEEN COALESCE(NULL, ranged_s.date_submitted) AND COALESCE(NULL, ranged_s.date_submitted) left join `edit_decisions` as `ed` on `ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_s_ed` on `ranged_s_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = ranged_s.submission_id order by `ed`.`stage_id` desc, `ed`.`round` desc, `ed`.`date_decided` desc limit 1) left join `edit_decisions` as `ranged_ed` on `ranged_ed`.`edit_decision_id` = `ed`.`edit_decision_id` and ranged_ed.date_decided BETWEEN COALESCE(NULL, ranged_ed.date_decided) AND COALESCE(NULL, ranged_ed.date_decided) left join `edit_decisions` as `first_ed` on `ranged_ed`.`edit_decision_id` is not null and `first_ed`.`edit_decision_id` = (select `ed`.`edit_decision_id` from `edit_decisions` as `ed` where `ed`.`submission_id` = s.submission_id order by `ed`.`stage_id` asc, `ed`.`round` asc, `ed`.`date_decided` asc limit 1) where `s`.`submission_progress` = 0 and `s`.`context_id` = 1) AS statistics inner join (select EXTRACT(YEAR FROM COALESCE(NULL, CURRENT_TIMESTAMP)) - EXTRACT(YEAR FROM COALESCE( NULL, (select `s`.`date_submitted` from `submissions` as `s` where `s`.`date_submitted` is not null order by `s`.`date_submitted` asc limit 1), CURRENT_TIMESTAMP ) ) + 1 AS count) AS years on 1 = 1 Connection ID (thread ID): 16 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15066 15066 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15066 15066 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us 2020-02-01 2:17:58 0 [Note] InnoDB: Using Linux native AIO 2020-02-01 2:17:58 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-02-01 2:17:58 0 [Note] InnoDB: Uses event mutexes 2020-02-01 2:17:58 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-02-01 2:17:58 0 [Note] InnoDB: Number of pools: 1 2020-02-01 2:17:58 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-02-01 2:17:58 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 2020-02-01 2:17:58 0 [Note] InnoDB: Completed initialization of buffer pool 2020-02-01 2:17:58 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-02-01 2:17:58 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5137589634 2020-02-01 2:17:58 0 [Note] InnoDB: Last binlog file './dbsvc-528-bin.000179', position 21719 2020-02-01 2:17:58 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-02-01 2:17:58 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-02-01 2:17:58 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-02-01 2:17:58 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-02-01 2:17:58 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-02-01 2:17:59 0 [Note] InnoDB: Waiting for purge to start