[MCOL-509] deploy buildbot for build and regression test automation Created: 2017-01-17  Updated: 2017-07-07  Resolved: 2017-07-07

Status: Closed
Project: MariaDB ColumnStore
Component/s: Build
Affects Version/s: None
Fix Version/s: 1.1.0

Type: Task Priority: Major
Reporter: David Thompson (Inactive) Assignee: David Hill (Inactive)
Resolution: Fixed Votes: 0
Labels: technicaldebt

Issue Links:
Blocks
blocks MCOL-510 create build for SUSE 12 Closed
Sprint: 2017-2, 2017-3, 2017-4, 2017-5, 2017-6, 2017-7, 2017-8, 2017-9, 2017-10, 2017-11, 2017-12, 2017-13, 2017-14

 Description   

buildbot is the corporate standard for build automation within mariadb. We should migrate the builds to an instance of this and trigger builds from github. This should perform both build and regression tests for pull requests as well as merges.



 Comments   
Comment by David Hill (Inactive) [ 2017-02-15 ]

I have the make installs working plus I committed a script that is used to start CS up so regression test can be done.
Working on regression testing next

Comment by David Hill (Inactive) [ 2017-02-16 ]

First step toward getting the regression test working on buildbot, I got test000 passing...
So will work on a small subset of the regression test cases that can be run bu BuildBot

Updating Columnstore.xml settings.
22restartsystem Fri Feb 17 02:21:55 2017
23
24 System being restarted now ............
25 Successful restart of System
26
27Running test000.sh.
28000 Create/Load Uncompressed Tables: Passed
29Running test001.sh.

Comment by David Hill (Inactive) [ 2017-02-24 ]

I have everything working for centos6 ,7 and suse12. successfuly build, startsystem and run regression test.
Ubuntu and Debian are having issues for some reason starting up. So those 2 are still work in progress

Comment by David Hill (Inactive) [ 2017-03-01 ]

Woo Hoo...

With Andrews change to get the system started on ubuntu and debian and an additional change I had to make to get the regression test script to work on those 2, I have all the buildbot successfully running inclduing test a cutdown version of the regression test, test000 and test001.

still to do...

1. Tie it into build packages for repo extractions
2. doing a run based on a time set where it will run the complete regression test suite.

Comment by David Hill (Inactive) [ 2017-07-07 ]

completed. now have all OS's successfully build, running regression test successfully and pushing packages up to the master-buildbot node. This happens on each github checking. test000.sh and test001.sh are running each time here. There are a few failures in the test runs, but should be resolved in the 1.1.0

also have a nightly centos 7 build that runs the full regression test and that is working great.

For this one and the github checkin runs, I out a copt of the regression test report. This is from the nightly

SoftwareVersion = 1.1.0
18000 Create/Load Uncompressed Tables: Passed
19001 Working Folder Test: Failed
20002 Concurrency Test: Passed
21005 Working DML Test: Passed
22006 Count while loading: Passed (loads=100, rowsPerLoad=25000, queries=3076)
23007 Count while updating: Passed (updates=1000, rowsPerUpdate=50000, queries=47020)
24008 Count while importing: Passed (imports=200, rowsPerImport=500000, queries=305)
25009 Count while deleting: Passed (imports=50, rowsPerImport=75000, queries=2253)
26010 Drop Partition Test: Passed (18 counts all matched!)
27011 cpimport Features Test: Passed
28012 Varbinary Test: Passed
29013 BLOB Test: Passed
30100 Create/Load Compressed Tables: Passed
31101 Working Folder UM Join Comp On: Passed
32102 Concurrency Test tpch1c: Passed
33105 Working DML Test dmlc: Passed
34106 Count while loading dmlc: Passed (loads=100, rowsPerLoad=25000, queries=6324)
35107 Count while updating dmlc: Passed (updates=1000, rowsPerUpdate=50000, queries=53430)
36108 Count while importing dmlc: Passed (imports=200, rowsPerImport=500000, queries=357)
37109 Count while deleting dmlc: Passed (imports=50, rowsPerImport=75000, queries=1472)
38110 Drop Partition Test dmlc: Passed (18 counts all matched!)
39112 Varbinary Test: Passed
40200 Monitor TotalUmMemory: Passed
41201 Version Buffer Test: Passed
42202 Wide Table Tests: Passed
43203 EM Min/Max Valid L.D.I. Test: Passed (200 loads)
44204 EM Min/Max Valid cpimport Test: Passed (200 imports)
45210 VSS Flush Block in Block Cache: Passed
46211 Concurrent Transactions Test: Passed (seconds=1800, thr=15, rowFactor=10000, batches=4905, rows=345510000)
47212 Concurrent Transactions Test: Passed (seconds=60, thr=5, tables=5, batches=108, rows=5184)
48295 DML schema EM Validation: Passed
49296 DMLC schema EM Validation: Passed
50297 Miscellaneous Tests: Passed
51299 Japanese Language Test: Passed (44 scripts all matched)
52-----------------------------------------------------------------------------------------------------------
53+----------------------------------------------------------------------+
54| runId | version | rel | buildDtm | run_time | maxPP | maxEM | maxDMP | maxImp | maxCntlr | maxAll |
55+----------------------------------------------------------------------+
56| 1 | | | | 03:00:58 | 26.8 | 36.9 | 1.5 | 2.4 | 0.0 | 59.0 |
57+----------------------------------------------------------------------+
58+----------------------------------------------------------------------------------+
59| Run | Start | Run ID | Version | Rel | Run Time | Max ExeMgr | Avg ExeMgr | Tests Run |
60+----------------------------------------------------------------------------------+
61| Current | 2017-07-07 05:18:29 | 1 | | | 03:00:58 | 36.9 | 5.40156 | 34 |
62+----------------------------------------------------------------------------------+
63+------------------------------------------------------------+
64| test | run_time | maxPP | maxEM | maxDMP | maxImp | maxCntlr | maxWrkr | maxAll |
65+------------------------------------------------------------+
66| test000 | 00:17:10 | 11.2 | 0.1 | 0.1 | 0.8 | 0.0 | 0.0 | 11.4 |
67| test001 | 00:19:45 | 22.2 | 16.4 | 0.1 | 0.0 | 0.0 | 0.0 | 36.9 |
68| test002 | 00:01:12 | 24.5 | 13.0 | 0.1 | 0.0 | 0.0 | 0.0 | 37.4 |
69| test005 | 00:03:29 | 21.7 | 9.1 | 0.1 | 0.0 | 0.0 | 0.0 | 30.6 |
70| test006 | 00:03:01 | 21.3 | 8.6 | 0.1 | 0.0 | 0.0 | 0.0 | 30.0 |
71| test007 | 00:02:13 | 21.3 | 8.6 | 0.1 | 0.0 | 0.0 | 0.0 | 30.1 |
72| test008 | 00:09:43 | 21.3 | 8.6 | 0.1 | 0.4 | 0.0 | 0.0 | 30.4 |
73| test009 | 00:03:36 | 21.3 | 8.7 | 0.2 | 0.3 | 0.0 | 0.0 | 30.3 |
74| test010 | 00:02:23 | 21.3 | 8.7 | 0.1 | 0.0 | 0.0 | 0.0 | 30.2 |
75| test011 | 00:00:44 | 21.3 | 8.8 | 0.1 | 0.2 | 0.0 | 0.0 | 30.4 |
76| test012 | 00:00:46 | 21.3 | 9.6 | 0.2 | 0.2 | 0.0 | 0.0 | 31.2 |
77| test013 | 00:00:10 | 21.3 | 9.6 | 0.2 | 0.0 | 0.0 | 0.0 | 31.2 |
78| test100 | 00:02:47 | 21.7 | 9.7 | 0.2 | 2.4 | 0.0 | 0.0 | 33.7 |
79| test101 | 00:04:28 | 24.1 | 10.4 | 0.2 | 0.0 | 0.0 | 0.0 | 34.8 |
80| test102 | 00:01:09 | 26.8 | 12.8 | 0.2 | 0.0 | 0.0 | 0.0 | 39.6 |
81| test105 | 00:03:59 | 24.3 | 8.7 | 0.2 | 0.1 | 0.0 | 0.0 | 33.2 |
82| test106 | 00:05:20 | 24.2 | 8.4 | 0.2 | 0.0 | 0.0 | 0.0 | 32.9 |
83| test107 | 00:02:37 | 24.2 | 8.4 | 0.2 | 0.0 | 0.0 | 0.0 | 32.9 |
84| test108 | 00:09:48 | 24.3 | 8.5 | 0.2 | 0.6 | 0.0 | 0.0 | 33.6 |
85| test109 | 00:02:51 | 24.2 | 8.6 | 0.2 | 0.4 | 0.0 | 0.0 | 33.4 |
86| test110 | 00:02:17 | 24.2 | 8.6 | 0.2 | 0.0 | 0.0 | 0.0 | 33.1 |
87| test112 | 00:00:46 | 24.2 | 9.3 | 0.3 | 0.4 | 0.0 | 0.0 | 33.9 |
88| test200 | 00:20:25 | 26.5 | 36.9 | 0.3 | 0.6 | 0.0 | 0.1 | 59.0 |
89| test201 | 00:07:52 | 21.7 | 0.1 | 0.3 | 0.3 | 0.0 | 1.0 | 23.2 |
90| test202 | 00:01:16 | 23.6 | 2.2 | 0.3 | 2.4 | 0.0 | 1.0 | 26.9 |
91| test203 | 00:04:53 | 23.6 | 0.4 | 0.3 | 0.0 | 0.0 | 1.0 | 25.4 |
92| test204 | 00:04:59 | 23.6 | 0.4 | 0.3 | 0.2 | 0.0 | 1.0 | 25.6 |
93| test210 | 00:07:37 | 23.6 | 0.7 | 0.3 | 0.0 | 0.0 | 1.0 | 25.7 |
94| test211 | 00:30:44 | 23.7 | 1.1 | 1.5 | 0.2 | 0.0 | 1.0 | 27.4 |
95| test212 | 00:01:15 | 23.7 | 1.1 | 1.5 | 0.0 | 0.0 | 1.0 | 27.4 |
96| test297 | 00:00:02 | 23.7 | 1.1 | 1.5 | 0.0 | 0.0 | 1.0 | 27.4 |
97| test299 | 00:01:39 | 23.7 | 1.1 | 1.5 | 0.1 | 0.0 | 1.0 | 27.4 |
98+------------------------------------------------------------+
99-----------------------------------------------------------------------------------------------------------
100TEST100 : df -h output:
101Filesystem Size Used Avail Use% Mounted on
102/dev/xvda1 20G 2.3G 18G 12% /
103devtmpfs 15G 0 15G 0% /dev
104tmpfs 15G 11M 15G 1% /dev/shm
105tmpfs 15G 17M 15G 1% /run
106tmpfs 15G 0 15G 0% /sys/fs/cgroup
107/dev/xvdb 100G 24G 77G 24% /data
108tmpfs 3.0G 0 3.0G 0% /run/user/1001
109-----------------------------------------------------------------------------------------------------------
110Working Folder Test scripts that failed:
111Compare failed - working_tpch1_compareLogOnly/functionsAndExpressions/bug3784.sql
112Compare failed - working_tpch1_compareLogOnly/view/mts_view.80.sql
113Compare failed - working_tpch1_compareLogOnly/view/mts_view.sql
114-----------------------------------------------------------------------------------------------------------
115Working DML Test Details:
116Details from queryTester:
117Total Local Passed = 70
118Total Local Failed = 0
119Total Ref Passed = 0
120Total Fef Failed = 0
121Total Compare Passed = 0
122Total Compare Failed = 0
123Total Selects = 1139
124Total Inserts = 436
125Total Updates = 139
126Total Deletes = 26
127Total Creates = 306
128Total Drops = 408
129-----------------------------------------------------------------------------------------------------------
130Working Folder UM Join Test Details:
131Details from queryTester:
132Total Local Passed = 706
133Total Local Failed = 0
134Total Ref Passed = 0
135Total Fef Failed = 0
136Total Compare Passed = 0
137Total Compare Failed = 0
138Total Selects = 26954
139Total Inserts = 1754
140Total Updates = 128
141Total Deletes = 45
142Total Creates = 1006
143Total Drops = 1472
144Changed this Columnstore.xml setting:
145 <PmMaxMemorySmallSide>64M</PmMaxMemorySmallSide><!-- divide by 48 to get element count -->
146TEST101 : df -h output:
147Filesystem Size Used Avail Use% Mounted on
148/dev/xvda1 20G 2.3G 18G 12% /
149devtmpfs 15G 0 15G 0% /dev
150tmpfs 15G 14M 15G 1% /dev/shm
151tmpfs 15G 153M 15G 2% /run
152tmpfs 15G 0 15G 0% /sys/fs/cgroup
153/dev/xvdb 100G 43G 58G 43% /data
154tmpfs 3.0G 0 3.0G 0% /run/user/1001
155tmpfs 3.0G 0 3.0G 0% /run/user/0
156-----------------------------------------------------------------------------------------------------------
157TEST200 : df -h output:
158Filesystem Size Used Avail Use% Mounted on
159/dev/xvda1 20G 2.4G 18G 12% /
160devtmpfs 15G 0 15G 0% /dev
161tmpfs 15G 15M 15G 1% /dev/shm
162tmpfs 15G 225M 15G 2% /run
163tmpfs 15G 0 15G 0% /sys/fs/cgroup
164/dev/xvdb 100G 52G 49G 52% /data
165tmpfs 3.0G 0 3.0G 0% /run/user/1001
166-----------------------------------------------------------------------------------------------------------
167TEST299 : df -h output:
168Filesystem Size Used Avail Use% Mounted on
169/dev/xvda1 20G 2.4G 18G 12% /
170devtmpfs 15G 0 15G 0% /dev
171tmpfs 15G 15M 15G 1% /dev/shm
172tmpfs 15G 481M 15G 4% /run
173tmpfs 15G 0 15G 0% /sys/fs/cgroup
174/dev/xvdb 100G 60G 41G 60% /data
175tmpfs 3.0G 0 3.0G 0% /run/user/1001
176-----------------------------------------------------------------------------------------------------------
177Tests completed!
178Archive directory ./archive/2017-07-07@05:17:45
179program finished with exit code 0
180elapsedTime=0.004101

Generated at Thu Feb 08 02:21:37 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.