LevelDB: ER_KEY_NOT_FOUND (Can't find record)

The provided concurrent test creates several LevelDB tables, populates them and starts running parallel REPLACE or DELETE. During the test flow 1-2 threads report an error like

REPLACE INTO ... failed: 1032 Can't find record in 'C'

DELETE FROM `B`... failed: 1032 Can't find record in 'B'

revision-id: psergey@askmonty.org-20121224102848-849oxv5xi5c2oyqn

revno: 4480

branch-nick: mysql-5.6-leveldb

Notes on running the test:

• don't forget to restart server with a clean datadir between test runs, since DROP doesn't work well for LevelDB;
• start server with disable-debug, it is known to increase chances of hitting the error; you can try to remove it later if you get the failure;
• the command line assumes you have an already running server; if it's not so, replace gentest.pl with runall.pl and dsn=... with basedir=..;
• if you run gentest.pl, it will report an error as above, but it might not stop until the end of the test duration – you can interrupt it if you don't need it any longer; if you wait till the end, it should finish saying "Test completed with failure status STATUS_DATABASE_CORRUPTION (105)";
• if you encounter the problem quickly, you can try to reduce the number of threads;
• if you don't encounter the problem, try to modify the number of threads both ways (increase, decrease);
• I built the server with BUILD/compile-pentium-debug-max-no-ndb;
• running this test on my machine with valgrind didn't produce any valgrind warnings (although the error still happened);
• gentest.pl sometimes hangs for quite a while after printing 'Reporters: ErrorLog, Backtrace' – I don't know yet why it happens, just wait through it;
• on test start it prints some errors like "ALTER TABLE `A` DISABLE KEYS failed: 1031 Table storage engine for 'A' doesn't have this option" - it's expected, ignore them;
• the flow most certainly can be simplified further, but I don't want to risk reducing the probability of the failure on your machine, since I cannot try it in advance.

RQG grammar (leveldb-bug-2.yy)

query:

	replace | delete ;

replace:

	REPLACE INTO _table ( _field_no_pk[invariant] ) 

  SELECT _field_no_pk[invariant] FROM _table AS X where ORDER BY _field_no_pk LIMIT large_digit;

delete:

	DELETE FROM _table where_delete ORDER BY _field_no_pk LIMIT small_digit ;

where:

	WHERE X . _field < value | 	

	WHERE X . _field IN ( value , value , value , value , value ) |

	WHERE X . _field BETWEEN small_digit AND large_digit |

	WHERE X . _field BETWEEN _tinyint_unsigned AND _int_unsigned  ;

where_delete:

	|

	WHERE _field = value |

	WHERE _field IN ( value , value , value , value , value ) |

	WHERE _field BETWEEN small_digit AND large_digit ;

large_digit:

	5 | 6 | 7 | 8 ;

small_digit:

	1 | 2 | 3 | 4 ;

value:

	_digit | _tinyint_unsigned | _int_unsigned ;

RQG data template (leveldb-bug-2.zz):

$tables = {

	names => ['A','B','C','D'],

        rows => [0,1,10,100],

	pk => [ 'int' ],

	engines => [ 'LevelDB' ]

};

$fields = {

        types => [ 'int', 'smallint', 'tinyint', 'mediumint', 'bigint', 'decimal(6,2)', 'numeric(6,2)', 'float(6,2)', 'double(6,2)', 'blob(1024)', 'tinyblob', 'mediumblob', 'longblob' ],

        indexes => [ undef ]

};

RQG command line (modify port in dsn if needed):

perl gentest.pl --gendata=leveldb-bug-2.zz --threads=8 --queries=100M --duration=600 --dsn=dbi:mysql:host=127.0.0.1:port=3306:user=root:database=test --grammar=leveldb-bug-2.yy