[MDEV-10255] Test timeouts on p8-trusty-bintar and p8-trusty-bintar-debug Created: 2016-06-19  Updated: 2016-07-16  Resolved: 2016-07-16

Status: Closed
Project: MariaDB Server
Component/s: Tests
Affects Version/s: 5.5, 10.0, 10.1
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Daniel Bartholomew
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Blocks
blocks MDEV-7069 Fix buildbot failures in main server ... Stalled

 Description   

Various tests fail with timeouts on p8 buliders.
http://buildbot.askmonty.org/buildbot/builders/p8-trusty-bintar/builds/1071

rpl.rpl_mixed_drop_create_temp_table 'innodb_plugin,mix' w1 [ fail ]  timeout after 900 seconds
 
        Test ended at 2016-06-17 17:14:16
 
 
 
Test case timeout after 900 seconds
 
 
 
== /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/var/1/log/rpl_mixed_drop_create_temp_table.log == 
Log_name	Pos	Event_type	Server_id	End_log_pos	Info
 
master-bin.000001	#	Query	#	#	BEGIN
 
master-bin.000001	#	Query	#	#	use `test`; INSERT INTO tt_xx_1() VALUES (1)
 
master-bin.000001	#	Xid	#	#	COMMIT /* XID */
 
master-bin.000001	#	Query	#	#	use `test`; DROP TABLE IF EXISTS `xx_1` /* generated by server */
 
-e-e-e-e-e-e-e-e-e-e-e- >> B T Drop-If-Xe << -e-e-e-e-e-e-e-e-e-e-e-
 
 
 
SET @commands= 'B T Drop-TXe';
 
BEGIN;
 
INSERT INTO tt_xx_1() VALUES (1);
 
DROP TABLE tt_2, xx_1;
 
ERROR 42S02: Unknown table 'xx_1'
 
-b-b-b-b-b-b-b-b-b-b-b- >> B T Drop-TXe << -b-b-b-b-b-b-b-b-b-b-b-
 
Log_name	Pos	Event_type	Server_id	End_log_pos	Info
 
master-bin.000001	#	Query	#	#	BEGIN
 
master-bin.000001	#	Query	#	#	use `test`; INSERT INTO tt_xx_1() VALUES (1)
 
master-bin.000001	#	Xid	#	#	COMMIT /* XID */
 
master-bin.000001	#	Query	#	#	use `test`; DROP TABLE `tt_2`,`xx_1` /* generated by server */
 
-e-e-e-e-e-e-e-e-e-e-e- >> B T Drop-TXe << -e-e-e-e-e-e-e-e-e-e-e-
 
 
 
 
 
 == /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/var/1/tmp/analyze-timeout-mysqld.1.err ==
 
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/var/tmp/1/mysqld.1.sock' (111)
 
 
 
 == /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/var/1/tmp/analyze-timeout-mysqld.2.err ==
 
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/var/tmp/1/mysqld.2.sock' (111)


parts.partition_alter1_1_innodb 'innodb_plugin' w1 [ fail ]  timeout after 900 seconds
 
        Test ended at 2016-06-17 20:12:50
 
 
 
Test case timeout after 900 seconds
 
 
 
== /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/var/1/log/partition_alter1_1_innodb.log == 
CAST(f_int1 AS CHAR), 'just inserted' FROM t0_template
 
WHERE f_int1 BETWEEN @max_row_div2 - 1 AND @max_row_div2 + 1
 
ORDER BY f_int1;
 
DROP TRIGGER trg_3;
 
	
# check trigger-12 success: 	1
 
DELETE FROM t1
 
WHERE f_int1 <> CAST(f_char1 AS SIGNED INT)
 
AND f_int2 <> CAST(f_char1 AS SIGNED INT)
 
AND f_charbig = '####updated per insert trigger####';
 
ANALYZE  TABLE t1;
 
Table	Op	Msg_type	Msg_text
 
test.t1	analyze	status	OK
 
CHECK    TABLE t1 EXTENDED;
 
Table	Op	Msg_type	Msg_text
 
test.t1	check	status	OK
 
CHECKSUM TABLE t1 EXTENDED;
 
Table	Checksum
 
test.t1	<some_value>
 
OPTIMIZE TABLE t1;
 
 
 
 == /var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/var/1/tmp/analyze-timeout-mysqld.1.err ==
 
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/var/lib/buildbot/maria-slave/p8-trusty-bintar/build/mysql-test/var/tmp/1/mysqld.1.sock' (111)

etc.

Another flavor:
http://buildbot.askmonty.org/buildbot/builders/p8-trusty-bintar/builds/1069/steps/test/logs/stdio

innodb.innodb_autoinc_lock_mode_zero 'innodb_plugin' w3 [ fail ]
 
        Test ended at 2016-06-17 04:28:09
 
 
 
CURRENT_TEST: innodb.innodb_autoinc_lock_mode_zero
 
 
 
 
 
Failed to start mysqld.1
 
mysqltest failed but provided no output



 Comments   
Comment by Elena Stepanova [ 2016-06-26 ]

Many varieties, e.g. http://buildbot.askmonty.org/buildbot/builders/p8-trusty-bintar/builds/9/steps/test/logs/stdio

Failing test(s): multi_source.gtid rpl.rpl_mdev6020 rpl.rpl_row_drop_create_temp_table rpl.rpl_row_img_blobs rpl.rpl_row_img_eng_min

They can fail directly with "test timeout" or indirectly because some part of the test failed due to a timeout.

Comment by Sergey Vojtovich [ 2016-07-07 ]

P8 builders (not only trusty) seem to have rather slow disks, according to hdparm: Timing buffered disk reads: 44 MB in 3.34 seconds = 13.17 MB/sec

Since we run tests with --parallel=4 it may happen so that 2 (or more) IO hungry tests may load disks heavily.

Some options:

  • use --mem, but it may not be suitable for big tests and we'll have to make sure /dev/shm is clean after every test run
  • reduce parallel, but it may not give desired effect (unless reduced down to 1) because 2 heavy ddl_innodb tests may run concurrently
  • remove xtra-big tests, but I don't completely like reducing coverage this way
  • increase timeouts
Comment by Elena Stepanova [ 2016-07-07 ]

It seems that the situation might have been resolved already. Around July 3-4 both builders were switched from p8-trusty-bb slave to power8-vlp04. Since then there have been 11 builds on both builders together, and not a single timeout. Before that, p8-trusty-bintar-debug was failing pretty much all the time, and p8-trusty-bintar all the time, so 11 successful builds in a row is a statistically significant number.

So, I won't change the buildbot config just yet, lets observe it for some more time and see how it goes.

p8-trusty-bb is still on the list, but as of the time of this comment, it's offline, hopefully it stays this way.

Comment by Elena Stepanova [ 2016-07-16 ]

There have been no timeouts for 2 weeks, since the tests were switched to the new slave power8-vlp04, so I think it's safe to assume the problem was fixed by the upgrade.

Generated at Thu Feb 08 07:40:50 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.