It looks like rotation of binlogs involves some broken logic involving . After binlog rotation MYSQL_BIN_LOG::do_checkpoint_request() is called. It calls ha_commit_checkpoint_request(), and from there for each engine that has commit_checkpoint_request function binlog_checkpoint_callback() is called. There mysql_bin_log.mark_xids_active() is called and corresponding mark_xid_done() is not called from anywhere.

I don't understand why this problem is not exposed anywhere in the test suite but I was able to hit it like this:

After that I get "void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed" during shutdown after the test.