[MCOL-676] Internal error: Lost connection to DMLProc after getting a new client Created: 2017-04-22  Updated: 2017-05-08  Resolved: 2017-05-08

Status: Closed
Project: MariaDB ColumnStore
Component/s: DMLProc
Affects Version/s: 1.0.8
Fix Version/s: Icebox

Type: Bug Priority: Critical
Reporter: rambo Assignee: Andrew Hutchings (Inactive)
Resolution: Duplicate Votes: 0
Labels: need_feedback
Environment:

Linux localhost 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Attachments: File columnstoreSupportReport.columnstore-1.tar.gz     File columnstoreSupportReport.tgz    
Issue Links:
Relates
relates to MCOL-679 Crash on prepared statement when no p... Closed

 Description   

QLSTATE[HY000]: General error: 1815 Internal error: Lost connection to DMLProc after getting a new client [2:InetStreamSocket::connect: connect() error: Connection refused to: InetStreamSocket: sd: 141 inet: 127.0.0.1 port: 8614 then InetStream2017-04-22 5:14:48
Raw SQL : INSERT INTO vstats_servers (iid,ip, ccons, cobj, CPU, cused, hit, loads, mem, reqs, rpc, rx, time, tpr, tx, uptime) VALUES (1579, '1.193.216.198', 1310, 44391947, 1.73, 100, 2634, '0.04,0.09,0.08', 68.44, 4127, 2.17, 1031830481, 1492809000, 12.12, 1425673601, 13340311),(1578, '1.193.216.199', 1364, 40337393, 2.56, 100, 1253, '0.22,0.26,0.20', 41.82, 10764, 0.99, 2843915931, 1492809000, 19.68, 3021637715, 127258),(1978, '1.193.216.205', 1142, 6475118, 2.51, 8.61, 1801, '0.21,0.17,0.11', 43.84, 10642, 1, 2148243334, 1492809000, 38.45, 2356498731, 204962),(1821, '1.82.208.118', 446, 41576235, 3.05, 94.4, 673, '0.32,0.18,0.17', 43.47, 3098, 0.87, 1433701665, 1492809000, 506.49, 1810809385, 417709),(1558, '1.82.208.122', 3608, 40979912, 1.49, 100, 2814, '0.17,0.16,0.11', 79.88, 4342, 2.71, 1961956444, 1492809000, 7.51, 2515680837, 11888960),(1957, '1.82.208.135', 678, 24607181, 2.26, 37.62, 525, '0.19,0.11,0.09', 45.17, 2053, 0.99, 1074080256, 1492809000, 193.45, 1540871724, 824505),(844, '106.39.254.22', 70, 21732133, 5.49, 100, 1491, '0.08,0.06,0.06', 45.52, 2528, 1.01, 1640883830, 1492809000, 62.75, 1899859285, 135408),(1505, '111.11.4.231', 351, 43638512, 1.65, 100, 2217, '0.17,0.16,0.16', 41.7, 4581, 1.29, 2891579019, 1492809000, 75.81, 3260308686, 926409),(789, '111.11.4.233', 566, 26160373, 4.95, 46.92, 665, '0.13,0.18,0.18', 44.38, 3594, 1.1, 4200830392, 1492809000, 345.16, 4522258531, 411100),(1824, '111.161.66.142', 69, 65084945, 2.41, 66.47, 98, '0.11,0.13,0.09', 43.74, 574, 1.09, 1311257805, 1492809000, 47, 1342277140, 653719),(1569, '111.161.66.146', 144, 42270649, 1.21, 100, 583, '0.04,0.11,0.09', 58.96, 1560, 1.22, 1918610780, 1492809000, 206.96, 2064064190, 9137914),(1832, '111.20.248.170', 215, 34002817, 0.89, 70.66, 825, '0.08,0.04,0.00', 54.97, 1115, 1.37, 782793779, 1492809000, 64.63, 872710960, 4992019),(1953, '111.20.248.171', 10230, 34715255, 3.08, 38.89, 2122, '0.29,0.30,0.27', 66.61, 12211, 1.02, 2512548837, 1492809000, 138.46, 3065521396, 989986),(1982, '111.47.231.132', 113, 1930486, 1.49, 3.89, 540, '0.05,0.06,0.05', 40.4, 984, 1.03, 931654706, 1492809000, 676.44, 1051245360, 122270),(1954, '112.17.39.77', 176, 27916191, 4.32, 61.87, 438, '0.36,0.26,0.21', 51.17, 2300, 1.1, 4396091304, 1492809000, 375.23, 4737349210, 314412),(1739, '112.21.165.172', 1094, 40359323, 3.51, 100, 1998, '1.57,1.10,0.98', 43.46, 7602, 1.04, 5889199726, 1492809000, 314.32, 6286720219, 142054),(1734, '112.21.165.245', 418, 37294944, 2.4, 100, 13839, '0.76,0.68,0.51', 91.24, 18771, 2.26, 7986314496, 1492809000, 36.79, 9619668406, 5559104),(1880, '112.29.252.229', 67, 15783670, 1.23, 32.2, 173, '0.04,0.08,0.07', 40.41, 1256, 1.09, 1339121919, 1492809000, 87.11, 1438654013, 1858461),(1923, '112.29.252.231', 1468, 43911144, 3.07, 72.22, 4135, '0.22,0.23,0.19', 50.62, 13997, 0.97, 2839653806, 1492809000, 105.12, 3250025810, 1874618),(1661, '112.35.35.200', 821, 86441285, 5.19, 100, 4937, '1.41,1.05,1.03', 47.36, 12577, 1.18, 14142927289, 1492809000, 213.49, 14757443318, 1247415),(1666, '112.35.35.201', 593, 89559533, 4.34, 100, 334, '0.31,0.24,0.23', 53.4, 3983, 1.05, 3809088896, 1492809000, 78.36, 4514427241, 1247455),(1673, '112.35.35.202', 579, 28378782, 3.36, 36.89, 391, '0.05,0.08,0.09', 59.8, 2282, 1.22, 2484413809, 1492809000, 288.51, 2593519495, 1245301),(512, '112.5.251.87', 154, 9190343, 2.89, 100, 506, '0.07,0.07,0.06', 68.07, 782, 0.85, 233737893, 1492809000, 33.49, 1126844256, 9431304),(511, '112.5.251.88', 138, 8453814, 4.11, 100, 299, '0.11,0.14,0.15', 52.73, 839, 0.25, 1529211074, 1492809000, 709.88, 1890910602, 128449),(1363, '112.65.226.213', 154, 42928747, 1.29, 100, 668, '0.11,0.07,0.01', 45.52, 1126, 0.79, 742023891, 1492809000, 60.82, 857945462, 8593079),(685, '112.67.250.22', 1, 27305912, 1.46, 100, 444, '0.02,0.01,0.00', 44.22, 510, 1, 111087519, 1492809000, 2.33, 200773944, 743054),(1948, '112.67.250.24', 944, 19286905, 1.82, 22.23, 68, '0.03,0.02,0.00', 44.77, 3618, 0.95, 396511700, 1492809000, 18.87, 410956567, 1249742),(1706, '113.200.251.4', 27, 21430585, 2.81, 100, 81, '0.22,0.38,0.21', 54.65, 486, 1.01, 1045411820, 1492809000, 5.13, 1065282454, 1122108),(1932, '113.200.251.5', 627, 21697903, 2.19, 33.77, 1161, '0.09,0.07,0.03', 45.75, 4058, 1.07, 1344054711, 1492809000, 43.72, 1530744558, 1354151),(1985, '113.207.31.83', 137, 1948266, 2.13, 3.84, 487, '0.06,0.10,0.05', 39.67, 1839, 1.04, 389239679, 1492809000, 130.12, 457784245, 51286),(1701, '113.207.31.86', 170, 56967925, 1.12, 66.2, 1400, '0.08,0.04,0.00', 58.62, 1968, 1.22, 902329409, 1492809000, 22.1, 1044938655, 11283359),(1657, '113.207.31.95', 95, 44506244, 2.26, 100, 655, '0.18,0.14,0.10', 40.38, 1780, 1.08, 966028280, 1492809000, 494.37, 1071970618, 127813),(1503, '113.215.0.13', 28, 26084483, 3.3, 80.22, 246, '0.04,0.06,0.07', 40.46, 694, 1.11, 770683327, 1492809000, 26.01, 831387922, 264936),(479, '113.215.0.7', 380, 14667514, 3.22, 100, 562, '0.78,1.56,1.59', 48.42, 2727, 0.83, 2151898015, 1492809000, 439.81, 2278277233, 452337),(1512, '113.6.248.7', 183, 43952355, 2.22, 100, 2854, '0.41,0.40,0.31', 42.17, 5403, 1.49, 4603301418, 1492809000, 67.07, 5041270091, 1017312),(74, '113.6.248.8', 425, 20985171, 5.57, 42.99, 379, '0.13,0.20,0.18', 42.25, 3286, 0.96, 6516632441, 1492809000, 435.05, 6953588082, 241920),(1771, '114.80.216.164', 144, 20510363, 2.21, 100, 606, '0.40,0.38,0.50', 51.67, 1866, 1.9, 387848703, 1492809000, 32.6, 2200179668, 1897320),(1447, '114.80.216.165', 681, 37606155, 1.2, 100, 1088, '0.23,0.10,0.06', 54.67, 1776, 1.03, 53862749, 1492809000, 42.16, 1065896589, 9098200),(1764, '114.80.223.147', 88, 43588119, 1.47, 100, 5325, '0.21,0.28,0.27', 37.48, 6586, 1.56, 2023097581, 1492809000, 41.7, 2764431407, 85474),(1964, '116.114.19.242', 138, 12031234, 2.81, 24.67, 235, '0.04,0.13,0.09', 44.62, 1035, 1.05, 2684779850, 1492809000, 468.03, 2839273378, 649670),(692, '116.114.19.247', 261, 12523341, 2.73, 100, 1466, '0.40,0.22,0.12', 64.3, 2371, 1.48, 1386275524, 1492809000, 56.62, 1742006076, 1908665),(1743, '117.159.48.15', 54, 58986483, 1.29, 66.54, 2341, '0.26,0.20,0.16', 38.99, 3439, 1.38, 1790227311, 1492809000, 21.65, 2080337763, 26660),(1834, '117.159.48.4', 3361, 41761056, 3.36, 100, 470, '0.34,0.31,0.22', 76.18, 1626, 1.22, 5194119127, 1492809000, 760.06, 5417850399, 1001639),(1714, '117.169.88.36', 1742, 85157424, 3.18, 100, 350, '0.23,0.19,0.12', 60.94, 895, 1.04, 3125365425, 1492809000, 479.88, 3239427031, 7492249),(1713, '117.169.88.38', 173, 29903510, 1.24, 33, 638, '0.00,0.02,0.00', 48, 1038, 1, 592742941, 1492809000, 52.76, 934909943, 6442438),(1351, '117.177.243.12', 1173, 30442265, 1.67, 55.84, 4249, '0.08,0.09,0.09', 49.39, 5471, 1.1, 1472992700, 1492809000, 124.13, 1643497241, 2196841),(1700, '117.177.243.13', 939, 36641210, 2.51, 100, 1807, '0.34,0.29,0.28', 42.36, 9658, 1.11, 3002084723, 1492809000, 169.68, 3398241876, 239082),(1955, '117.177.243.21', 1157, 27248573, 2.44, 37.46, 1135, '0.19,0.17,0.18', 44.73, 4805, 1.06, 4550532600, 1492809000, 105.71, 4900665379, 466687),(1448, '117.25.158.134', 64, 40040453, 1.27, 100, 3567, '0.07,0.04,0.06', 40.13, 4367, 1.67, 982094634, 1492809000, 65.61, 1465325966, 969681),(1458, '117.25.158.142', 276, 41017734, 3.2, 100, 1983, '0.10,0.23,0.27', 45.89, 3050, 0.99, 2682041544, 1492809000, 208.11, 3652321149, 474299),(669, '118.180.52.14', 89, 59122787, 2.61, 100, 119, '1.31,1.39,1.63', 63.39, 2185, 1.02, 362297079, 1492809000, 14.59, 385734787, 972268),(716, '118.180.52.8', 11, 29310175, 1.93, 100, 516, '0.05,0.03,0.00', 40.28, 897, 1.1, 140013889, 1492809000, 31.5, 172739397, 925692),(1636, '119.39.74.100', 887, 32455894, 1.21, 39.19, 437, '0.00,0.01,0.00', 48.88, 762, 1.13, 340164243, 1492809000, 25, 363101403, 12383277),(1947, '119.39.74.103', 102, 15594514, 1.88, 32.17, 372, '0.09,0.14,0.10', 44.48, 1075, 1.08, 1444218380, 1492809000, 720.26, 1530313156, 845353),(1271, '119.84.87.181', 1743, 43882290, 1.09, 100, 2021, '0.02,0.04,0.04', 61.69, 2735, 2.09, 401181816, 1492809000, 1.35, 894018982, 15171921),(1851, '119.84.87.241', 1529, 38258364, 2.19, 100, 2233, '0.19,0.20,0.18', 62.86, 2944, 1.48, 5350055500, 1492809000, 32.53, 5721793645, 4029344),(431, '120.192.90.168', 325, 24701669, 3.36, 100, 439, '0.95,0.86,0.83', 40.39, 2077, 0.91, 2377668845, 1492809000, 149.46, 2788670762, 128632),(432, '120.192.90.170', 112, 14356561, 2.39, 100, 1564, '1.43,1.10,1.05', 69.5, 2905, 1.28, 1721526607, 1492809000, 73.7, 1911483806, 1347642),(435, '120.210.209.173', 197, 21590754, 2.94, 100, 4660, '0.15,0.21,0.14', 75.93, 7134, 0, 4565720276, 1492809000, 0, 5213816162, 3313732),(436, '120.210.209.174', 2604, 22920207, 7.53, 100, 3566, '0.27,0.19,0.11', 54.82, 13616, 0.98, 2691968495, 1492809000, 96.37, 3096645251, 549007),(1787, '120.221.74.117', 171, 38724859, 2.22, 96.14, 297, '0.60,0.74,0.76', 78, 888, 2.63, 900896298, 1492809000, 95.46, 1137342731, 3869031),(1946, '120.221.74.119', 164, 12139522, 1.98, 24.84, 217, '0.10,0.05,0.01', 41.72, 1286, 0.93, 1274941871, 1492809000, 344.22, 1341425159, 56021),(1692, '120.52.28.212', 338, 65129960, 3.41, 100, 497, '0.55,0.30,0.21', 43.5, 2966, 1.12, 3453455921, 1492809000, 345.56, 3910802853, 128651),(1568, '120.52.28.220', 1935, 33259030, 1.67, 100, 2845, '0.09,0.16,0.18', 85.27, 4705, 1.07, 2509053238, 1492809000, 166.2, 2832534774, 13027322),(1492, '121.14.157.37', 2667, 43615305, 1.86, 100, 2980, '0.32,0.20,0.15', 84.98, 4097, 1.03, 1936800289, 1492809000, 52.37, 2269750408, 13534155),(1508, '121.14.157.44', 2286, 43908102, 2.64, 100, 1740, '0.02,0.07,0.07', 56.55, 8646, 0.55, 1211376764, 1492809000, 39.59, 1494870718, 2575246),(1807, '121.14.157.47', 1853, 40776227, 3.19, 100, 3746, '0.26,0.18,0.16', 54.57, 4674, 1.31, 2607982982, 1492809000, 65.31, 2832540214, 6018282),(216, '122.72.28.11', 31, 14421956, 2.01, 100, 326, '0.02,0.04,0.00', 56.9, 675, 1.11, 303169918, 1492809000, 21.5, 488009146, 1782166),(214, '122.72.28.9', 120, 11001065, 1.86, 100, 269, '0.08,0.09,0.09', 42.3, 484, 0.96, 565040743, 1492809000, 337.81, 628188615, 127566),(468, '122.72.36.155', 231, 9164362, 2.83, 100, 381, '0.27,0.09,0.08', 56.09, 1950, 0.9, 364045812, 1492809000, 39.11, 427404332, 1407600),(1651, '122.72.36.158', 5013, 37743340, 1, 97.9, 144, '0.10,0.04,0.00', 41.36, 236, 0.93, 155477176, 1492809000, 7.92, 214042078, 3755581),(176, '122.72.52.116', 1991, 7307492, 6.36, 100, 4364, '3.70,3.36,2.95', 54.71, 16125, 1.03, 6254474335, 1492809000, 244.16, 7273081952, 715095),(174, '122.72.52.117', 130, 12700342, 2.61, 100, 3638, '0.78,0.90,0.82', 50, 6877, 1.17, 3062248402, 1492809000, 48.41, 3392636833, 216444),(1969, '123.129.249.134', 5755, 18218571, 3.81, 23.61, 3858, '0.47,0.30,0.20', 47.35, 21203, 1.03, 3084849075, 1492809000, 63.21, 3527119846, 396405),(1971, '123.129.249.135', 4897, 13647224, 3.55, 18.06, 2524, '0.12,0.13,0.09', 46.42, 17543, 1, 2798931871, 1492809000, 20.07, 3076686751, 309248),(1926, '123.129.249.142', 438, 65033135, 2.79, 64.38, 5212, '0.28,0.29,0.21', 54.83, 7434, 2.36, 3839497987, 1492809000, 29.63, 4586427991, 1857438),(1823, '123.129.249.151', 1983, 40350983, 3.33, 100, 3083, '0.31,0.25,0.18', 48.53, 17933, 0.99, 3896488833, 1492809000, 48.31, 4249650190, 463390),(1681, '123.172.124.38', 217, 38062815, 1.07, 45.5, 572, '0.15,0.07,0.01', 50.47, 1411, 1.1, 834567673, 1492809000, 68.38, 936086444, 10026739),(1685, '123.172.124.41', 235, 45030234, 4.53, 100, 229, '3.52,3.68,2.21', 45.6, 1815, 1.11, 4755405645, 1492809000, 464.46, 4986148988, 127629),(361, '123.71.254.6', 110, 9252814, 3.56, 100, 768, '0.45,0.36,0.20', 60.38, 1439, 1.45, 785265078, 1492809000, 103.01, 1159482876, 126399),(690, '124.207.119.50', 711, 7833999, 2.99, 100, 319, '2.20,1.37,1.00', 58.67, 2194, 1.17, 7596598861, 1492809000, 156.92, 7696962260, 190794),(1751, '124.232.148.100', 494, 34292975, 2.95, 100, 768, '0.24,0.25,0.19', 72.16, 1292, 1.01, 2056163030, 1492809000, 632.64, 2219631224, 2614488),(1959, '124.232.148.98', 114, 14036838, 2.23, 23.34, 821, '0.30,0.23,0.19', 59.37, 1347, 1.06, 1723325194, 1492809000, 815.35, 1866158651, 390502),(1180, '125.62.37.19', 442, 19572275, 0.74, 100, 24, '0.68,0.92,1.00', 89.77, 35, 0.96, 21464934, 1492809000, 12.64, 23430756, 4825555),(1607, '150.138.166.227', 1499, 44725815, 1.28, 100, 2038, '0.11,0.07,0.01', 64.6, 2738, 1.31, 1490607703, 1492809000, 60.3, 1703977343, 12936807),(1974, '150.138.166.231', 2594, 10748211, 3.28, 14.52, 1536, '0.35,0.23,0.19', 45.73, 11001, 0.97, 2272432883, 1492809000, 62.9, 2461261185, 300992),(1248, '150.138.166.253', 2201, 43961552, 2.52, 100, 1625, '0.11,0.15,0.09', 46.91, 12022, 0.98, 2291463725, 1492809000, 32.78, 2518059743, 541294),(1644, '171.107.82.106', 126, 36979385, 1.22, 100, 1714, '0.05,0.06,0.06', 55.59, 2251, 1.09, 610155722, 1492809000, 331.92, 919035003, 10863824),(1285, '171.107.82.116', 119, 32714957, 2.37, 100, 940, '0.12,0.18,0.21', 40.86, 2638, 1.13, 3073770090, 1492809000, 144.47, 3411926299, 128126),(1258, '175.6.128.237', 261, 44580058, 1.1, 100, 3416, '0.12,0.13,0.09', 80.15, 4211, 1.8, 766437910, 1492809000, 73.19, 1306560352, 14533738),(1153, '180.88.63.22', 208, 5144331, 1.2, 31.76, 2372, '0.86,1.19,1.18', 48.53, 3094, 1.66, 1272774294, 1492809000, 51.16, 1542136633, 284485),(1972, '180.88.63.23', 401, 6283756, 1.92, 11.48, 332, '0.03,0.09,0.08', 43.39, 2723, 1.2, 1397061170, 1492809000, 27.52, 1546166831, 278985),(1921, '182.242.137.70', 107, 33884299, 1.49, 83.68, 651, '0.01,0.09,0.10', 41.37, 1471, 0.81, 1358471895, 1492809000, 456.2, 1618359053, 230653),(171, '183.203.21.72', 864, 43657742, 2.86, 100, 2009, '0.21,0.21,0.16', 42.98, 8198, 0.95, 4030967062, 1492809000, 79.66, 4367501604, 28782),(1444, '183.203.21.74', 34, 36396395, 0.94, 77.86, 1267, '0.21,0.19,0.12', 37.32, 1964, 1.31, 809333497, 1492809000, 361.82, 1032601645, 23838),(1931, '183.214.145.228', 168, 5356179, 1.06, 6.78, 141, '0.02,0.01,0.00', 35.8, 173, 1.57, 101771711, 1492809000, 55.34, 150025825, 1412012),(1930, '183.214.145.231', 909, 6637792, 1.59, 18.02, 320, '0.09,0.09,0.04', 43.33, 413, 1.05, 505503780, 1492809000, 2216.25, 555001963, 1412008),(1689, '183.232.7.131', 263, 61118813, 1.25, 73.5, 1912, '0.02,0.06,0.07', 53.89, 2513, 1.14, 1110339936, 1492809000, 158.19, 1385523514, 5136008),(956, '183.232.7.134', 462, 39637290, 1.19, 100, 2096, '0.14,0.24,0.20', 64.32, 3095, 0, 1592677143, 1492809000, 0, 1878885624, 5846095),(1977, '183.232.7.141', 552, 8165651, 2.4, 14.71, 1345, '0.29,0.19,0.18', 43.4, 5760, 1.02, 1894966231, 1492809000, 211.3, 2346915210, 234327),(957, '183.232.7.143', 821, 43197292, 2.97, 100, 1019, '0.46,0.35,0.26', 60.22, 4464, 1.03, 2606323450, 1492809000, 157.72, 2951504552, 1362851),(1949, '183.6.204.11', 148, 44418534, 2.78, 27.58, 8321, '0.35,0.39,0.30', 42.44, 10702, 1.44, 4305529851, 1492809000, 31.04, 5038573723, 823133),(1916, '183.6.204.18', 1131, 87478055, 3.07, 100, 1315, '0.18,0.16,0.14', 45.98, 9843, 0.39, 2812621723, 1492809000, 82.7, 2982218564, 289470),(1980, '183.6.204.194', 1605, 9020270, 3.05, 13.24, 1756, '0.21,0.18,0.18', 44.38, 14448, 0.69, 3025891257, 1492809000, 42.71, 3247772867, 141808),(1981, '183.6.204.196', 1134, 8770490, 3.88, 12.84, 3597, '0.20,0.20,0.13', 44.7, 14646, 0.63, 2692974869, 1492809000, 26.97, 3038938292, 141953),(1962, '183.6.204.2', 1317, 38764160, 2.06, 50.51, 1328, '0.14,0.12,0.13', 46, 9623, 0.7, 1380512060, 1492809000, 79.85, 1624974080, 648011),(1649, '183.6.204.4', 128, 79780387, 5.37, 100, 7927, '0.20,0.22,0.18', 38.98, 10998, 1.13, 6089787488, 1492809000, 46.29, 6730781425, 346426),(138, '183.6.204.5', 2664, 36703163, 2.93, 100, 1348, '0.38,0.40,0.39', 62.51, 10244, 0.46, 2623585069, 1492809000, 91.94, 2809843121, 1927334),(1744, '202.100.73.68', 659, 42949919, 1.06, 47.97, 3358, '0.06,0.02,0.00', 51.38, 4295, 1.24, 657640088, 1492809000, 60.26, 988861791, 10064892),(1940, '202.100.73.71', 373, 34683911, 3.51, 52.74, 200, '0.20,0.24,0.15', 50.02, 1792, 1.1, 2129343703, 1492809000, 686.95, 2598505386, 1251347),(1942, '202.111.173.114', 2487, 16876203, 7.07, 34.56, 929, '0.35,0.27,0.16', 51.63, 19680, 1.03, 5576512909, 1492809000, 76.19, 6006685947, 1332811),(1722, '202.111.173.90', 233, 21939939, 1.86, 40.6, 8143, '0.17,0.23,0.18', 45.01, 10482, 1.5, 2605109663, 1492809000, 190.25, 3329311799, 2055422),(382, '202.116.160.90', 30, 11373121, 1.38, 100, 36, '0.10,0.05,0.01', 57.8, 147, 1, 7377949, 1492809000, 58.95, 10023152, 127831),(383, '202.116.160.92', 4321, 530146, 0.3, 100, 298, '0.00,0.00,0.00', 72.17, 355, 1, 19287180, 1492809000, 0.11, 27743429, 9209730),(1352, '211.140.157.165', 102, 44050285, 1.25, 100, 2939, '0.32,0.39,0.32', 42.72, 4579, 1.16, 3964169174, 1492809000, 212.9, 4250679152, 626608),(1924, '211.140.157.173', 96, 32824033, 2.24, 99.41, 356, '0.14,0.17,0.12', 44.43, 1248, 0.97, 1282480465, 1492809000, 181.35, 1421573832, 673157),(1124, '211.161.80.25', 141, 14979020, 1.03, 100, 2253, '0.89,2.19,2.31', 45.21, 3137, 1.55, 607637164, 1492809000, 59.93, 815523805, 386215),(1125, '211.161.80.26', 36, 11767876, 1.41, 100, 255, '0.89,2.19,2.31', 47.8, 578, 1.02, 361766200, 1492809000, 393.22, 394148214, 57088),(1115, '211.162.213.2', 229, 14676830, 18.19, 100, 737, '4.07,4.29,4.31', 54.9, 1109, 1.8, 572451610, 1492809000, 82.47, 654869191, 3078054),(1989, '211.162.213.8', 128, 573141, 1.5, 9.64, 261, '4.07,4.29,4.31', 41.1, 963, 1.06, 1304944428, 1492809000, 65.28, 1374283651, 27269),(1897, '211.91.140.155', 133, 18348052, 1.78, 56.89, 295, '0.02,0.04,0.02', 48.79, 584, 1.12, 688181152, 1492809000, 160.16, 814241538, 2555878),(1896, '211.91.140.156', 24, 742579, 1.1, 1.41, 602, '0.17,0.08,0.03', 35.59, 829, 9, 251136538, 1492809000, 4.85, 277036878, 74057),(875, '218.11.176.40', 2628, 36554821, 4.03, 100, 2382, '0.18,0.31,0.26', 52.28, 13980, 0.77, 4677150648, 1492809000, 225.12, 5071285498, 517327),(1726, '218.11.176.44', 346, 63700664, 1.66, 71.67, 9751, '0.22,0.22,0.18', 75.01, 12875, 1.66, 3580367748, 1492809000, 38.81, 4407437482, 7575192),(1975, '218.11.176.52', 1693, 14030084, 3.34, 17.8, 4047, '0.26,0.25,0.20', 47.03, 14889, 0.96, 3849638566, 1492809000, 65.73, 4529474372, 296315),(1797, '218.199.110.53', 4, 7073905, 1.85, 100, 133, '0.10,0.06,0.01', 36.99, 189, 1.13, 98860386, 1492809000, 0.22, 103022389, 74900),(1486, '218.6.154.175', 66, 44200783, 1.65, 100, 5919, '0.14,0.16,0.11', 39.45, 6991, 1.23, 1138890124, 1492809000, 75.3, 1975475551, 397005),(1796, '218.6.154.177', 127, 61081576, 2.16, 100, 1744, '0.25,0.22,0.22', 43.73, 3569, 1.11, 4258222145, 1492809000, 130.78, 4650018645, 128673),(1956, '218.6.154.184', 117, 17697071, 2.06, 34.99, 622, '0.47,0.19,0.12', 43.9, 1259, 1.03, 1066386936, 1492809000, 426.17, 1209940756, 487240),(1283, '218.60.40.249', 346, 24712381, 6.56, 22.36, 3199, '0.35,0.36,0.30', 47.56, 8827, 1.31, 7665252766, 1492809000, 118.21, 8105970715, 1429075),(1008, '218.60.54.231', 85, 37018113, 2.89, 92.56, 2008, '0.16,0.12,0.10', 41.19, 3815, 1.57, 2788160491, 1492809000, 78.52, 3007397872, 675532),(1850, '218.60.54.235', 484, 34552793, 3.56, 100, 243, '0.44,0.33,0.20', 56.8, 2529, 0.95, 2445419641, 1492809000, 213.52, 2588897934, 3233750),(1560, '219.138.39.138', 62, 41688560, 2.21, 100, 6184, '0.12,0.19,0.15', 37.79, 8152, 3.37, 2703192377, 1492809000, 8.09, 3087638241, 316576),(1827, '219.138.39.139', 380, 57337217, 3.95, 100, 2160, '0.55,0.48,0.45', 59.81, 4486, 0.87, 7024030656, 1492809000, 363.32, 7546903265, 316482),(1436, '219.238.7.149', 128, 44575623, 1.47, 100, 2881, '0.13,0.19,0.21', 29.86, 4194, 1.13, 2214088216, 1492809000, 73.47, 2557176638, 109328),(1289, '219.83.160.13', 61, 80067077, 4.22, 100, 0, '2.64,2.58,2.37', 52.01, 27247, 1.05, 2949290509, 1492809000, 15.79, 2949428828, 752064),(1370, '219.83.160.17', 61, 87979778, 1.65, 100, 0, '0.71,0.88,0.93', 40.58, 13662, 1.1, 1089415355, 1492809000, 12.18, 1089481084, 746786),(1367, '219.83.160.18', 49, 88257254, 2.21, 100, 0, '0.16,0.21,0.18', 41.93, 17135, 1.11, 1512662488, 1492809000, 15.43, 1512749033, 746311),(1371, '219.83.160.19', 61, 87938964, 2.21, 100, 1, '0.89,0.95,0.95', 41.22, 14788, 1.09, 1251397044, 1492809000, 13.2, 1252158794, 758156),(778, '219.83.160.2', 31, 80066025, 21.69, 100, 4, '2.10,1.93,1.84', 35.3, 24721, 1, 3101260589, 1492809000, 14.64, 3101389484, 747232),(1804, '219.83.160.34', 89, 82874403, 2.4, 92.91, 0, '0.06,0.06,0.01', 41.58, 10184, 1.13, 694610579, 1492809000, 13.03, 694661614, 747813),(761, '220.165.12.67', 1510, 27855859, 2.56, 100, 1122, '0.08,0.14,0.09', 73.05, 1623, 1.45, 1050714301, 1492809000, 105.87, 1363103041, 2557647),(1320, '221.12.140.201', 5, 3272, 0.69, 0.06, 0, '2.25,2.29,2.42', 2.46, 0, 1, 0, 1492809000, 0.05, 0, 398100),(1943, '221.12.140.204', 0, 0, 0.67, 0, 0, '0.00,0.00,0.00', 2.02, 0, 1, 0, 1492809000, 0.14, 0, 1278541),(1534, '221.12.140.205', 144, 43523316, 1.56, 100, 3564, '0.13,0.34,0.29', 42.15, 5857, 1.37, 4152324752, 1492809000, 52.45, 4452634761, 631090),(1944, '221.12.140.211', 2, 0, 0.6, 0, 0, '0.00,0.00,0.00', 2.12, 0, 1, 0, 1492809000, 0.04, 0, 1277825),(1223, '221.131.105.12', 62, 34912682, 1.29, 93.7, 25, '0.34,0.21,0.18', 43.01, 2698, 1.24, 2925988824, 1492809000, 43.52, 2982106130, 759084),(1334, '221.131.105.8', 1757, 41704702, 2.38, 100, 577, '0.37,0.33,0.28', 64.15, 1506, 1.21, 4069485054, 1492809000, 1164.99, 4526749328, 2796735),(974, '221.14.149.232', 175, 43730583, 1.53, 99, 3044, '0.15,0.13,0.09', 56.45, 4145, 2.97, 1607541518, 1492809000, 110.76, 2070128722, 4889578),(975, '221.14.149.233', 2458, 40348895, 3.05, 100, 1847, '0.17,0.22,0.18', 49.28, 12101, 1.01, 2602667608, 1492809000, 65.17, 2943600924, 680077),(1986, '221.14.149.242', 2068, 2623672, 3.31, 3.57, 1494, '0.28,0.16,0.10', 42.04, 11276, 0.98, 2101253696, 1492809000, 94.16, 2272131042, 48999),(94, '221.204.173.203', 43, 40419394, 2.02, 100, 2352, '0.20,0.15,0.10', 38.16, 3423, 1.52, 977451280, 1492809000, 116.57, 1234178937, 539719),(1655, '221.204.173.216', 122, 84562357, 3.18, 100, 599, '0.16,0.18,0.16', 42.69, 3047, 0.85, 7952736079, 1492809000, 198.22, 8271530260, 128650),(1862, '221.228.69.34', 122, 132793445, 4.22, 100, 7397, '1.10,0.77,0.70', 48, 29694, 1.03, 5312625889, 1492809000, 11.45, 6645172187, 55352),(1575, '221.228.69.35', 100, 87021243, 3.56, 100, 8954, '0.36,0.45,0.47', 47.77, 35225, 1.05, 6406333611, 1492809000, 7.21, 7622300353, 55307),(926, '221.228.69.37', 55, 87591396, 6.8, 100, 3634, '0.50,0.40,0.37', 48.68, 35743, 1.01, 766440465, 1492809000, 18.82, 7351502462, 788313),(1936, '221.228.69.41', 48, 33280862, 1.98, 12.34, 0, '0.23,0.15,0.10', 37.34, 595, 1.15, 507211796, 1492809000, 17.28, 507216128, 749406),(1574, '221.228.69.44', 41, 88610838, 2.61, 100, 4329, '0.56,0.44,0.38', 47.15, 19640, 1, 3979328481, 1492809000, 11.46, 4540027235, 56299),(1979, '221.228.69.45', 3, 8972941, 0.83, 8.73, 0, '0.21,0.11,0.06', 39.12, 378, 1.03, 149520, 1492809000, 4.7, 165701089, 203836),(1504, '221.228.69.47', 248, 15360582, 1.1, 100, 0, '1.08,1.05,1.00', 58.14, 225, 2.01, 23225, 1492809000, 8.51, 37625, 127911),(1640, '221.228.69.48', 62, 87132761, 4.54, 100, 1693, '0.22,0.32,0.29', 45.3, 28938, 0.96, 3212685742, 1492809000, 15.2, 4000154817, 745786),(1925, '221.228.69.49', 128, 130273504, 4.91, 100, 8942, '0.71,0.83,0.81', 48.27, 36450, 1.06, 6272047715, 1492809000, 8.3, 7749863414, 55788),(1935, '221.228.69.50', 60, 33242205, 2.13, 12.33, 0, '0.10,0.14,0.09', 38.34, 650, 1.12, 563486130, 1492809000, 10.98, 563490576, 748920),(1934, '221.228.69.51', 37, 33286285, 1.49, 12.31, 0, '0.08,0.12,0.09', 37.11, 439, 1.15, 362668268, 1492809000, 12.06, 362671820, 748474),(1433, '221.228.69.61', 1, 71598034, 0.73, 100, 0, '0.00,0.01,0.00', 39.15, 400, 1.03, 157735, 1492809000, 7.52, 159367895, 201444),(366, '222.140.154.140', 1784, 36986606, 2.7, 100, 2490, '0.16,0.19,0.18', 43.19, 13018, 0.96, 2317610364, 1492809000, 202.42, 2636870279, 150947),(1573, '222.140.154.146', 658, 69445690, 1.46, 91.55, 2309, '0.28,0.25,0.20', 73.72, 3574, 1.41, 2395045226, 1492809000, 73.37, 2743989479, 13689931),(1976, '222.140.154.148', 2358, 12695828, 3.13, 18.89, 1900, '0.11,0.14,0.10', 45.82, 11427, 0.69, 2248510761, 1492809000, 91.16, 2457664458, 212679),(1958, '222.222.12.197', 1260, 36196268, 3.89, 45.74, 3735, '0.40,0.43,0.36', 49.37, 12583, 0.95, 4371074559, 1492809000, 48.81, 4759427797, 751463),(903, '222.222.12.58', 2214, 40299574, 4.22, 100, 2798, '0.24,0.21,0.22', 48.57, 12149, 1.13, 3382249606, 1492809000, 136.01, 3838637049, 805002),(1564, '222.222.12.60', 1881, 41989386, 1.48, 100, 6351, '0.50,0.51,0.39', 86.81, 11039, 1.47, 4847375208, 1492809000, 87.31, 5984816306, 10498058),(1741, '222.37.3.181', 200, 23642141, 0.91, 50.94, 2238, '0.11,0.07,0.07', 32.69, 3025, 1.02, 893776146, 1492809000, 11.8, 1029349157, 1343961),(615, '222.42.5.9', 107, 10975993, 1.7, 100, 588, '5.56,2.23,1.54', 34.38, 1080, 1.33, 1007250343, 1492809000, 323.38, 1085908649, 128609),(412, '222.46.123.151', 11, 18490012, 1.02, 100, 235, '0.27,0.37,0.47', 37.67, 368, 0.82, 369587989, 1492809000, 19.44, 416171942, 932101),(1929, '36.250.226.111', 84, 12712491, 2.01, 25.74, 275, '0.04,0.09,0.03', 45.79, 707, 0.87, 695723744, 1492809000, 38.2, 736200648, 1423951),(1600, '36.250.226.116', 27, 31488219, 1.1, 88.36, 405, '0.00,0.02,0.00', 48.44, 880, 1, 335596000, 1492809000, 43.2, 454625072, 13090103),(1789, '42.123.114.36', 533, 31513576, 1.28, 33.4, 4241, '0.03,0.04,0.06', 46.48, 4706, 1.82, 333950725, 1492809000, 22.77, 558659631, 6930524),(1833, '42.123.114.39', 811, 20407263, 2.73, 100, 1731, '0.39,0.24,0.13', 45.98, 11478, 1.37, 2946807187, 1492809000, 53.3, 3178967031, 999565),(1945, '42.81.59.132', 249, 26506592, 2.31, 58.79, 450, '0.35,0.29,0.20', 47.51, 1677, 1.17, 2307208065, 1492809000, 361.6, 2975307388, 1261241),(1613, '42.81.59.144', 754, 38238116, 1.04, 100, 333, '0.05,0.07,0.02', 50.95, 800, 1.1, 746194074, 1492809000, 15.45, 835353835, 12844185),(1803, '58.222.48.13', 946, 18728415, 3.46, 41.11, 3025, '0.39,0.27,0.20', 46.08, 11145, 1.09, 3970444108, 1492809000, 248.48, 4401709593, 305733),(1382, '58.222.48.30', 943, 44419968, 3.91, 98.76, 3842, '0.58,0.49,0.40', 49.29, 14147, 1.08, 4619836368, 1492809000, 78.32, 5259958580, 677207),(1963, '58.222.48.31', 1569, 34009587, 3.09, 44.58, 3578, '0.30,0.25,0.19', 50.76, 11912, 1.05, 3857246658, 1492809000, 90.74, 4457132418, 665196),(1407, '58.222.48.8', 1477, 18274864, 4.69, 43.21, 2862, '0.39,0.24,0.19', 48.44, 9744, 0.98, 4194715733, 1492809000, 131.49, 4628863231, 144450),(1408, '58.222.48.9', 131, 43984206, 2.21, 100, 345, '0.45,0.33,0.23', 39.06, 6506, 1.18, 5156573161, 1492809000, 206.66, 5507367947, 76143),(1970, '58.255.173.42', 778, 12631483, 2.61, 17.64, 2370, '0.23,0.20,0.12', 44.21, 8718, 1.01, 2122175610, 1492809000, 171.04, 2692720748, 310191),(1619, '58.255.173.45', 2120, 43873237, 2.77, 100, 1096, '0.20,0.22,0.18', 42.54, 7581, 1.03, 2665648160, 1492809000, 247.01, 2795885971, 168587),(1563, '58.255.173.50', 186, 32731522, 1.48, 100, 2692, '0.06,0.15,0.16', 83.55, 3875, 1.4, 3075412268, 1492809000, 95.18, 3413352380, 8930235),(1561, '59.151.124.131', 3, 36634, 1.85, 1.07, 0, '0.05,0.06,0.01', 31.15, 0, 1, 0, 1492809000, 0.1, 0, 226114),(515, '59.151.124.133', 3, 431400, 2.93, 10.53, 0, '0.08,0.06,0.01', 50.32, 24, 1, 8538, 1492809000, 2.3, 9030, 226203),(1928, '59.46.13.252', 659, 25311110, 4.83, 42.64, 1089, '0.43,0.30,0.27', 49.32, 4159, 1.03, 5712953074, 1492809000, 383.35, 6189329118, 1428866),(439, '59.49.43.58', 525, 17729888, 2.01, 100, 524, '0.02,0.05,0.01', 62.4, 788, 1.43, 557445743, 1492809000, 35.3, 796190700, 2028297),(1260, '59.53.95.154', 582, 26985275, 1.47, 54.22, 1783, '0.12,0.17,0.11', 51.09, 2571, 2.68, 680058095, 1492809000, 1.53, 958578907, 3678071),(1839, '59.53.95.169', 1033, 67413028, 3.18, 94.65, 1456, '0.21,0.23,0.18', 69.65, 2429, 1.06, 4483346595, 1492809000, 452.41, 5017009987, 4369376),(1732, '60.174.242.11', 975, 44027964, 2.95, 100, 339, '0.28,0.15,0.10', 64.35, 4106, 1.16, 2179503285, 1492809000, 84.51, 2363996358, 3215885),(6, '60.174.242.41', 316, 24669275, 4.45, 100, 370, '1.70,1.56,1.30', 37.47, 3617, 0.8, 2409907934, 1492809000, 225.13, 3297347015, 128624),(1728, '60.174.243.86', 1619, 64971685, 1.24, 74.03, 5003, '0.19,0.20,0.13', 73.5, 6433, 1.28, 2486244462, 1492809000, 155.39, 2953895711, 10424278),(1847, '60.191.139.101', 524, 35254522, 2.4, 100, 296, '0.09,0.10,0.09', 68.21, 847, 0.96, 820609788, 1492809000, 203.18, 935474121, 3006260),(1490, '60.191.139.99', 144, 42901605, 1.67, 100, 2523, '0.04,0.09,0.08', 88.86, 4747, 1.37, 3650394359, 1492809000, 288.28, 3920392013, 13503575),(1368, '60.191.30.228', 30, 87001036, 2.95, 92.57, 2510, '0.24,0.22,0.18', 60.24, 26998, 1.01, 219110393, 1492809000, 23.96, 5010746628, 1885381),(1369, '60.191.30.229', 35, 87230770, 4.8, 100, 2784, '0.26,0.21,0.18', 37.37, 38245, 1.01, 3448253143, 1492809000, 17.93, 4556627321, 83046),(1879, '60.191.30.242', 226, 32920322, 3.76, 100, 486, '0.44,0.48,0.39', 50.92, 2060, 0.9, 4356610926, 1492809000, 99.28, 4805536929, 311349),(1984, '60.191.30.250', 119, 20204159, 5.25, 16.85, 4332, '0.46,0.54,0.66', 62.38, 22319, 1.05, 5908743040, 1492809000, 16.02, 6740925843, 56216),(1506, '60.217.25.212', 509, 42915963, 1.47, 100, 4601, '0.22,0.20,0.12', 61.58, 5767, 1.31, 3448533080, 1492809000, 488.1, 3770350158, 14515035),(1973, '60.217.25.213', 2429, 8202852, 2.61, 10.54, 1051, '0.07,0.04,0.00', 45.32, 10892, 0.99, 1411683551, 1492809000, 33.89, 1518648172, 302015),(1663, '60.217.25.218', 676, 87912437, 2.33, 100, 792, '0.22,0.09,0.04', 47.3, 8144, 0.99, 1148113873, 1492809000, 15.88, 1297239475, 725885),(1937, '61.160.196.48', 39, 80644858, 5.2, 100, 1699, '0.26,0.39,0.43', 43.96, 18799, 1, 282486839, 1492809000, 20.68, 4472548436, 1342523),(1565, '61.160.245.197', 78, 86728760, 3.89, 100, 9190, '0.28,0.36,0.39', 47.59, 35376, 1.03, 5090865844, 1492809000, 7.27, 6175328861, 55486),(1571, '61.160.249.24', 132, 88649742, 5.38, 100, 21090, '0.64,0.69,0.69', 50.01, 69452, 1.08, 9323703245, 1492809000, 7.64, 12321578803, 57889),(1635, '61.160.250.203', 75, 89122417, 4.61, 100, 17354, '0.56,0.66,0.64', 49.06, 60237, 0, 7962052158, 1492809000, 0, 10002111539, 57451),(1566, '61.160.250.71', 72, 87474588, 4.91, 100, 7947, '0.40,0.47,0.47', 47.29, 35308, 1.01, 5522558464, 1492809000, 15.32, 6462837103, 57227),(1175, '61.160.251.121', 40, 87106013, 4.41, 100, 2100, '0.19,0.26,0.26', 57.07, 24199, 1, 450353718, 1492809000, 18.67, 5850614794, 1339589),(865, '61.160.251.79', 14, 0, 3.1, 0, 0, '3.77,3.58,2.86', 12.02, 2177, 1.02, 0, 1492809000, 0.13, 217294077, 83394);

vstats_servers table create info:
CREATE TABLE `vstats_servers` (
`id` int(11) NOT NULL DEFAULT '0' COMMENT 'id',
`iid` int(11) NOT NULL DEFAULT '0' COMMENT '主机IP表vstats_servers_ip主键',
`ip` varchar(32) NOT NULL DEFAULT '' COMMENT '节点IP地址',
`ccons` bigint(20) NOT NULL DEFAULT '0' COMMENT '连接数',
`cobj` bigint(20) NOT NULL DEFAULT '0' COMMENT '对象数',
`cpu` decimal(10,2) NOT NULL DEFAULT '0.00' COMMENT 'CPU占用比',
`cused` decimal(10,2) NOT NULL DEFAULT '0.00' COMMENT '缓存占用比',
`hit` bigint(20) NOT NULL DEFAULT '0' COMMENT '命中数',
`loads` varchar(255) NOT NULL DEFAULT '' COMMENT '负载',
`mem` decimal(10,2) NOT NULL DEFAULT '0.00' COMMENT '内存占用比',
`reqs` bigint(20) NOT NULL DEFAULT '0' COMMENT '请求数',
`rx` bigint(20) NOT NULL DEFAULT '0' COMMENT '流入带宽',
`tx` bigint(20) NOT NULL DEFAULT '0' COMMENT '流出带宽',
`rpc` decimal(10,2) NOT NULL DEFAULT '0.00' COMMENT 'RPC',
`tpr` decimal(10,2) NOT NULL DEFAULT '0.00' COMMENT '响应时间',
`uptime` bigint(20) NOT NULL DEFAULT '0' COMMENT '在线时间',
`time` bigint(20) NOT NULL DEFAULT '0' COMMENT '时间'
) ENGINE=Columnstore DEFAULT CHARSET=utf8 COMMENT='autoincrement=id, 1'

 Comments   
Comment by Andrew Hutchings (Inactive) [ 2017-04-22 ]

Hi,
Are you streaming lots of DML before this occurs?

Comment by rambo [ 2017-04-22 ]

I have a batch insert loop.

Comment by Andrew Hutchings (Inactive) [ 2017-04-22 ]

OK, this is likely caused by MCOL-529. Basically every insert requires a short-lived socket connection which ends up in the TIME_WAIT state. If this happens too much in a short amount of time you exhaust the maximum port count.

There will be a fix for this in the next release, but in the mean time there are two workarounds for this:

1. Increase the port count:

/sbin/sysctl net.ipv4.ip_local_port_range="1024 65000"

2. Reduce the TIME_WAIT:

/sbin/sysctl net.ipv4.tcp_fin_timeout=30
 
/sbin/sysctl net.ipv4.tcp_tw_recycle=1
 
/sbin/sysctl net.ipv4.tcp_tw_reuse=1

The first one will make it take longer to hit it, so it depends on how many inserts you do and how long it takes you to hit it now. The second should have a larger effect.

You will also likely need to restart your ColumnStore cluster as sometimes transactions and locks get out-of-sync when this happens.

Comment by rambo [ 2017-04-22 ]

The configuration has already been changed。

  1. # cat /etc/sysctl.conf
    kernel.shmall = 268435456
    net.ipv4.ip_local_port_range = 1024 65000
    net.ipv4.tcp_max_tw_buckets = 5000
    net.ipv4.tcp_fin_timeout = 30
    net.ipv4.tcp_keepalive_time = 300
    net.ipv4.tcp_syncookies = 1
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_mem = 786432 1048576 1572864
    net.core.wmem_max = 873200
    net.core.rmem_max = 873200
    net.ipv4.tcp_wmem = 8192 436600 873200
    net.ipv4.tcp_rmem = 32768 436600 873200
    net.core.somaxconn = 256
    net.core.netdev_max_backlog = 1000
    net.ipv4.tcp_max_syn_backlog = 2048
    net.ipv4.tcp_retries2 = 5
    net.ipv4.tcp_keepalive_intvl = 30
    net.ipv4.tcp_keepalive_probes = 3
    net.ipv4.conf.lo.arp_ignore = 0
    net.ipv4.conf.lo.arp_announce = 0
    net.ipv4.conf.all.arp_ignore = 0
    net.ipv4.conf.all.arp_announce = 0
    net.ipv4.tcp_timestamps = 0
    net.ipv4.tcp_orphan_retries = 3
    vm.max_map_count = 655300
Comment by Andrew Hutchings (Inactive) [ 2017-04-23 ]

Do you mean you made these changes prior to this issue? If so can you please run the ColumnStore support tool so that we can get some more information? https://mariadb.com/kb/en/mariadb/system-troubleshooting-mariadb-columnstore/#mariadb-columnstore-support-tool

Alternatively if you have just made this change and it has resolved this for you please let us know.

Comment by rambo [ 2017-04-23 ]

Yes, I made these changes prior to this issue.

I run it's a single server.

/usr/local/mariadb/columnstore/bin/columnstoreSupport -a
columnstoreSupportReport.tgz

Comment by Andrew Hutchings (Inactive) [ 2017-04-24 ]

For some reason this didn't collect the logs from /usr/local/mariadb/columnstore. Can you please create a tar.gz of these and attach them?

One of the missing parts of the puzzle is you are using prepared statements. It appears you are getting mysqld crashes from these. I'll look into that in the mean time.

Comment by rambo [ 2017-04-24 ]

The logs(330MB) is very large to upload, so please download from here:

https://static.verycloud.cn/download/columnstoreSupportReport.columnstore-1.tar.gz

Comment by Andrew Hutchings (Inactive) [ 2017-04-24 ]

I have opened MCOL-679 for the mysqld crashes in your logs.

Comment by Andrew Hutchings (Inactive) [ 2017-04-24 ]

Beyond MCOL-679 the only semi-related problem I can spot in the logs in WriteEngineServ stopped responding to connections at 04:10 on the 22nd and was restarted. This could be related as a connection refused to that process would also trigger this.

How often are you getting this error?

Are you able to install the "sysstat" package via yum and run the following when it happens?:

sudo sar -n SOCK 1

This will help us make sure it isn't due to port exhaustion.

Comment by rambo [ 2017-04-25 ]

Linux 3.10.0-514.10.2.el7.x86_64 (xxxx) 04/25/2017 x86_64 (56 CPU)

09:27:55 AM totsck tcpsck udpsck rawsck ip-frag tcp-tw
09:27:56 AM 1320 66 1 0 0 950
09:27:57 AM 1320 66 1 0 0 950
09:27:58 AM 1320 66 1 0 0 950
09:27:59 AM 1320 66 1 0 0 953
09:28:00 AM 1320 66 1 0 0 953
09:28:01 AM 1320 66 1 0 0 953
09:28:02 AM 1488 231 1 0 0 1130
09:28:03 AM 1637 381 1 0 0 959
09:28:04 AM 1531 277 1 0 0 1116
09:28:05 AM 1531 277 1 0 0 1127
09:28:06 AM 1531 277 1 0 0 1127
09:28:07 AM 1531 277 1 0 0 1127
09:28:08 AM 1531 277 1 0 0 1128
09:28:09 AM 1531 277 1 0 0 1131
09:28:10 AM 1531 277 1 0 0 1131
09:28:11 AM 1531 277 1 0 0 1139
09:28:12 AM 1537 282 1 0 0 1146
09:28:13 AM 1537 282 1 0 0 1158
09:28:14 AM 1537 282 1 0 0 1169
09:28:15 AM 1537 282 1 0 0 1177
09:28:16 AM 1537 282 1 0 0 1188
09:28:17 AM 1537 282 1 0 0 1198
09:28:18 AM 1537 282 1 0 0 947
09:28:19 AM 1537 282 1 0 0 960
09:28:20 AM 1537 282 1 0 0 980
09:28:21 AM 1537 282 1 0 0 991
09:28:22 AM 1537 282 1 0 0 1001
09:28:23 AM 1537 282 1 0 0 1011
09:28:24 AM 1537 282 1 0 0 1020
09:28:25 AM 1537 282 1 0 0 1031
09:28:26 AM 1537 282 1 0 0 1041
09:28:27 AM 1537 282 1 0 0 1050
09:28:28 AM 1326 71 1 0 0 1273
09:28:29 AM 1326 71 1 0 0 1285
09:28:30 AM 1326 71 1 0 0 1295
09:28:31 AM 1326 71 1 0 0 1305
09:28:32 AM 1326 71 1 0 0 1315
09:28:33 AM 1326 71 1 0 0 1083
09:28:34 AM 1326 71 1 0 0 1091

Comment by Andrew Hutchings (Inactive) [ 2017-04-25 ]

That is higher than I was expecting, but nowhere near danger levels.

Do the inserts ever happen in parallel or do you serialise them?

Comment by rambo [ 2017-04-25 ]

yeah，it happens when parallels insert.

Comment by Andrew Hutchings (Inactive) [ 2017-04-25 ]

Ah! Now there is the problem! ColumnStore is not designed to do that using INSERT queries, there is all sorts of locking involved to prevent it. If you have a multi node setup you can parallel insert into the PM nodes using cpimport (which is much faster than INSERT queries anyway): https://mariadb.com/kb/en/mariadb/columnstore-bulk-data-loading/

Comment by rambo [ 2017-05-03 ]

columnstoreSupportReport.columnstore-1.tar.gz
SQLSTATE[HY000]: General error: 1815 Internal error: CAL0001: Insert Failed: IDB-2009: Unable to perform the batch insert operation because DMLProc batchinsert with PID 59290 is currently holding the table lock for session 2446. 2017-05-03 22:41:47
Raw SQL : INSERT INTO vstats_servers (iid,ip, ccons, cobj, CPU, cused, hit, loads, mem, reqs, rpc, rx, time, tpr, tx, uptime) VALUES (1631, '1.193.216.204', 475, 36317, 0.61, 0.05, 1735, '2.61,1.83,1.70', 25.14, 3390, 3.38, 8187192, 1493822400, 70.17, 83481030, 29314),(1308, '1.82.208.120', 9307, 6858776, 0.71, 7.84, 2985, '2.69,2.47,2.53', 63.94, 4954, 3.64, 5397772, 1493822400, 25.03, 317611257, 134631),(238, '103.245.81.201', 4721, 56070498, 8.19, 63.14, 2694, '1.99,2.18,2.07', 60.84, 14042, 6.41, 111596145, 1493822400, 13.2, 133885810, 78614),(1272, '112.21.165.170', 12825, 57177185, 1.64, 76.05, 4305, '0.51,0.51,0.53', 14.35, 21454, 0, 142340369, 1493822400, 0, 238182901, 78614),(592, '112.245.16.151', 16706, 14490957, 0.51, 100, 3218, '3.10,3.26,3.22', 94.33, 5324, 1.42, 9835534, 1493822400, 10.74, 135721631, 387896),(1502, '113.215.0.12', 656, 6934448, 0.5, 58.13, 700, '1.15,0.91,0.80', 46.29, 959, 1.19, 1398464, 1493822400, 16.6, 36978063, 109216),(75, '113.6.248.5', 2520, 28516564, 0.61, 65.35, 2698, '2.52,2.50,2.71', 53.74, 3994, 1.35, 3211027, 1493822400, 7.84, 117135667, 49772),(1455, '117.25.158.145', 7060, 24221181, 0.51, 34.3, 3740, '1.43,1.68,1.73', 43.59, 5528, 3.15, 5655095, 1493822400, 17.66, 165979713, 86677),(1677, '119.39.74.105', 6824, 12593114, 0.51, 6.25, 2151, '3.04,2.58,2.28', 77, 3523, 2.87, 2338584, 1493822400, 5.01, 79731599, 258805),(80, '119.84.87.242', 16670, 14879908, 0.51, 100, 2374, '1.30,1.62,1.77', 81.6, 3895, 1.52, 4366769, 1493822400, 14.66, 173253148, 344126),(1496, '121.14.157.42', 8502, 16908136, 0.71, 22.42, 4637, '1.46,1.67,1.86', 46.5, 7392, 1.94, 9400180, 1493822400, 4.08, 157211285, 86990),(164, '123.129.249.140', 442, 14670971, 0.47, 100, 2801, '2.29,1.91,1.76', 25.13, 4335, 2.48, 3198980, 1493822400, 5.54, 131359415, 49979),(1688, '123.172.124.40', 609, 11545075, 0.51, 2.71, 1898, '9.51,10.91,10.80', 35.02, 2966, 1.67, 4213131, 1493822400, 3.57, 73647239, 41201),(1244, '150.138.166.249', 10757, 29669367, 0.51, 60.98, 3152, '5.44,5.41,5.78', 62.95, 4588, 1.18, 4430656, 1493822400, 331.19, 178195849, 111936),(13, '171.107.82.109', 2488, 22719717, 0.61, 63.57, 2387, '1.56,1.93,2.04', 84.32, 4025, 1.32, 3992148, 1493822400, 18, 105245201, 87855),(725, '175.6.128.230', 21327, 11638048, 0.71, 6.16, 2240, '2.88,2.85,2.87', 83.88, 3633, 4, 4282552, 1493822400, 10.78, 102365095, 301824),(141, '183.6.204.13', 24617, 14813198, 0.66, 100, 4903, '4.70,4.65,4.90', 85.63, 9247, 1.54, 26119199, 1493822400, 57.91, 218391244, 260635),(89, '202.111.173.104', 3150, 14748612, 0.47, 100, 2804, '1.92,1.87,1.92', 55.42, 3906, 1.77, 4486236, 1493822400, 9.7, 113483357, 111283),(1206, '211.91.140.153', 8, 19689304, 0.51, 23.97, 1414, '4.82,3.50,3.19', 90.06, 2365, 1.48, 2541754, 1493822400, 2.06, 59464936, 349230),(386, '218.6.154.180', 7810, 5704658, 0.51, 22.52, 5002, '2.74,3.13,3.11', 39.99, 7359, 2.39, 5667071, 1493822400, 78.42, 265443723, 87091),(127, '219.138.39.144', 1229, 27695933, 0.61, 57.46, 2774, '2.34,2.25,2.19', 38.51, 4122, 1.55, 3277095, 1493822400, 3.6, 92495397, 56375),(978, '221.14.149.236', 12430, 6795695, 0.71, 3.4, 3018, '3.30,3.62,3.50', 76.11, 4306, 3.05, 4465198, 1493822400, 2.53, 151447189, 263799),(97, '221.204.173.202', 24027, 28246078, 0.72, 100, 9269, '4.17,4.22,4.34', 88.33, 12741, 2.28, 18935519, 1493822400, 13.55, 450455491, 143074),(368, '222.140.154.139', 15227, 14117756, 0.56, 100, 2347, '1.02,1.21,1.25', 68.81, 3845, 1.29, 2578987, 1493822400, 2.59, 163231324, 189786),(901, '222.222.12.57', 11223, 22414188, 0.72, 15.85, 6070, '3.28,3.20,3.25', 89.65, 8882, 1.45, 13669101, 1493822400, 3.82, 264112503, 49982),(724, '36.250.226.106', 561, 14145747, 0.61, 52.05, 2785, '1.22,1.23,1.17', 26.29, 4241, 1.49, 11431870, 1493822400, 4.05, 81359476, 49748),(1416, '58.222.48.17', 27616, 29718550, 0.61, 93.96, 6292, '1.56,1.83,2.02', 95.95, 9361, 1.97, 25779033, 1493822400, 2.28, 288683023, 134787),(204, '58.255.173.40', 11893, 16116851, 0.71, 17.34, 7757, '2.16,2.03,1.82', 73.58, 11933, 2.2, 14087069, 1493822400, 9.79, 214986945, 56397),(723, '59.53.95.149', 6018, 14795269, 0.56, 100, 2413, '0.83,1.32,1.50', 74.53, 3694, 2.47, 3386156, 1493822400, 31.43, 109835708, 86991),(4, '60.174.242.40', 20938, 12326132, 0.71, 47.27, 4190, '2.60,2.25,1.98', 87.56, 5952, 2.21, 26607982, 1493822400, 13.52, 214818753, 292028),(26, '60.191.139.121', 12720, 29521649, 5.52, 100, 1512, '0.50,0.54,0.59', 95.43, 11356, 5.18, 46977064, 1493822400, 30.46, 184063102, 364751),(1358, '60.217.25.204', 13889, 7731309, 0.61, 7.84, 2875, '1.62,1.32,1.27', 93.46, 4353, 2.61, 3683304, 1493822400, 13.15, 122836898, 370054);

Comment by Andrew Hutchings (Inactive) [ 2017-05-03 ]

This error is telling us that MariaDB session ID 2446 has a table lock on vstats_servers. Unfortunately there are no logs before 22:40 in your upload so it is impossible to tell if it is because the session is still open and uncommitted or something else happened.

Comment by rambo [ 2017-05-03 ]

Please download from here.

columnstoreSupportReport.columnstore-1.tar.gz

Comment by Andrew Hutchings (Inactive) [ 2017-05-04 ]

Unfortunately that still doesn't have any log data before 22:40 on the 3rd May.
You can view the table lock to see when it happened using the "viewtablelock" tool which will be in the /usr/local/mariadb/columnstore/bin directory. Assuming the cause isn't a current connection you can use the "cleartablelock" tool to clear it.

Comment by rambo [ 2017-05-04 ]

There is 1 table lock

Table LockID Process PID Session Txn CreationTime State DBRoots
billing.vstats_servers 375966 DMLProc batchinsert 59290 2446 181069 Wed May 3 17:11:06 2017 LOADING 1

Rolling back and clearing table lock for table billing.vstats_servers; table lock 375966

Rollback error: Unable to grab lock; Lock not found or still in use.
Table lock 375966 for table billing.vstats_servers is not cleared.

Comment by Andrew Hutchings (Inactive) [ 2017-05-04 ]

That is the one. You logs don't go far back enough to give details on it.

Are you 100% sure there is no active connection to the MariaDB server holding that lock open?

The only other way to clear it would be to tell mcsadmin to restart ColumnStore.

Comment by rambo [ 2017-05-04 ]

It is not often appear, usually run a few days later, will produce this error, this time on the need to restart the service via mcsadmin, this is the latest log.

columnstoreSupportReport.columnstore-2.tar.gz

Comment by Andrew Hutchings (Inactive) [ 2017-05-05 ]

The latest logs show us that the lock is still there and your version buffer is now full, likely due to the table lock on a DML query.

Comment by rambo [ 2017-05-05 ]

Is there any other solution besides restarting?

Comment by Andrew Hutchings (Inactive) [ 2017-05-05 ]

Until we know what caused the lock to remain we don't have any other solutions. If it is a current MariaDB connection that is held open then killing that connection should also release it.

Comment by rambo [ 2017-05-05 ]

@Andrew Hutchings I came from china, Do you have IM? may help to comminute quickly.

Recently I am frequently getting this problem, tks.

Comment by Andrew Hutchings (Inactive) [ 2017-05-05 ]

I am in the UK and the rest of the team are in the US. I am on Freenode IRC in the #mariadb under the name LinuxJedi. It may be better to use the MariaDB ColumnStore Google Group as others will be able to respond whilst I am not available: https://groups.google.com/d/forum/mariadb-columnstore

To be able to diagnose this further we will need data from before and during it happening. At the moment it is my belief that the mysqld crashes you were observing in MCOL-679 were the cause due to transactions happening in another session at the time.

Comment by rambo [ 2017-05-06 ]

OK, tks。
If this problem is repeated again, I will reopen a issue.

Comment by David Thompson (Inactive) [ 2017-05-08 ]

The issue is addressed in MCOL-679.

Generated at Thu Feb 08 02:22:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.