Debian 8, Linux 64 bit, ZFS

Please provide an example of the exact structure of a table with which you experience the error (SHOW CREATE TABLE ...).
Does it only happen on RENAME?
Does only SHOW TABLE STATUS help? What about, let's say, selecting from the table?
Is there a lot of activity on the server and on the machine around the time when the problem occurs?

Please also attach your cnf file(s).

The table structures affected were several Mediawiki tables. Both of them have blob columns, but that may be coincidental.

CREATE TABLE `wiki_interwiki` (

  `iw_prefix` varbinary(32) NOT NULL,

  `iw_url` blob NOT NULL,

  `iw_api` blob NOT NULL,

  `iw_wikiid` varbinary(64) NOT NULL,

  `iw_local` tinyint(1) NOT NULL,

  `iw_trans` tinyint(4) NOT NULL DEFAULT '0',

  UNIQUE KEY `iw_prefix` (`iw_prefix`)

) ENGINE=InnoDB DEFAULT CHARSET=binary DATA DIRECTORY='/var/lib/mysql/innodb_data/'

I attempted the rename first by ALTER TABLE .. RENAME then RENAME TABLE, both with the same result.

I do still have several tables on our production server that are affected, they are no longer used by our application but are still exhibiting the strange rename behavior. Neither SELECT nor DESCRIBE result in "fixing" the affected tables. SHOW CREATE TABLE does fix it. Interestingly, the table remains "stuck" again immediately after the rename operation. Here's my session output:

MariaDB [yyy]> alter table xxx_interwiki rename xxx_interwiki_test;

ERROR 1025 (HY000): Error on rename of './yyy/xxx_interwiki' to './yyy/xxx_interwiki_test' (errno: -1 "Internal error < 0 (Not system error)")

MariaDB [yyy]> show create table xxx_interwiki;

[...]

1 row in set (0.00 sec)

MariaDB [yyy]> alter table xxx_interwiki rename xxx_interwiki_test;

Query OK, 0 rows affected (0.00 sec)

MariaDB [yyy]> alter table xxx_interwiki_test rename xxx_interwiki;

ERROR 1025 (HY000): Error on rename of './yyy/xxx_interwiki_test' to './yyy/xxx_interwiki' (errno: -1 "Internal error < 0 (Not system error)")

MariaDB [yyy]> show create table xxx_interwiki_test;

[...]

1 row in set (0.00 sec)

MariaDB [yyy]> alter table xxx_interwiki_test rename xxx_interwiki;

Query OK, 0 rows affected (0.00 sec)

The server is always rather active, averaging 2.5k QPS. There is very little I/O load (our dataset fits in RAM). I'd prefer not posting the full config, however it is mostly defaults with a larger innodb_log_file_size and innodb_buffer_pool_size and skip-innodb_doublewrite, separate innodb_data_home_dir and innodb_log_group_home_dir directories (separate ZFS datasets).

I can't reproduce it so far.
I found some bugs that look alike to your problem, https://bugs.mysql.com/bug.php?id=67656 and https://bugs.mysql.com/bug.php?id=15991, but both of them use symbolic links, not DATA DIRECTORY clause.
What is innodb_flush_method variable value? Maybe you are willing to provide more information (configuration file) to the private ftp? (ftp.askmonty.org/private)

Hello,
I've uploaded the config to the private FTP as MDEV-13237-mariadb.cnf (please ignore the MDEV-13237-my.cnf file, this was not the correct version).

innodb_flush_method is O_DIRECT

I've double checked and there are no filesystem-level symlinks under the /var/lib/mysql directory.