MySQL Bug 26662 -> http://bugs.mysql.com/bug.php?id=26662
Also please note that /data is a partition on a software RAID.
The 2 underlying disks have write cache disabled:
[root@mysql mysql]# hdparm -W /dev/sda
/dev/sda:
write-caching = 0 
[root@mysql mysql]# hdparm -W /dev/sdb
/dev/sdb:
write-caching = 0
I'm not getting the warning with a similar disk configuration (no raid though, and I'm operating inside a VM, so maybe either of that makes the difference).
While I didn't investigate the queston at depth, a bit of googling shows that O_DIRECT + data=journal is known to be a problematic combination. Do the warnings go away if you turn off data journal?
Yes they do go away if I turn off data journal.
So which option is recommended?
data=journal and no O_DIRECT vs. default data journal + O_DIRECT
Anyway... the warning is not correct as I'm not on a tmpfs.
Well, the warning doesn't really say that you are on tmpfs, it just suggests it as one possible reason.
My guess (which might be wrong though) is that the attempt to set O_DIRECT fails not on InnoDB level, but deeper, and InnoDB has no way to know why exactly it failed. If that's so, all that could be done is add yet another possible reason to the warning text, but I really don't think it's wise because a warning message cannot cover all possible wrong combinations of storage options on all file systems. It can and should be added to a troubleshooting section of documentation though.
For the recommended options, it obviously depends on your priorities and workflow.
data=journal is by definition is slowest mode unless you have another botlleneck which makes the delay caused by full journaling insignificant; but it's supposed to be safer in the event of a crash. So, if you really need it, you don't have much choice but keep it and turn off O_DIRECT. If you don't, you can switch off the journal and try to play with O_DIRECT.
The effect of setting innodb_flush_method to O_DIRECT can be either positive or negative, and the only reliable advice (even the documented one: http://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_flush_method) is to benchmark your configuration with the workflow similar to real one to decide whether it works for you.
As I understand it, innodb was in this case configured to use O_DIRECT with
something like this in the my.cnf:
innodb_flush_method = O_DIRECT
And the data directory was on a file system that does not support O_DIRECT. In
this case InnoDB gives the mentioned warning, but continues anyway. Database
integrity should not be affected, but it makes sense to reconfigure innodb.
It could be nice to clarify the warning. I would suggest report it to
MySQL/Oracle developers, perferably with a suggestion on better wording. I
think I would prefer keeping the warning the same as MySQL for now.
With respect to recommendations: I would say that data-journal would not
generally be recommended - if it does what I guess it does, InnoDB already
does the same thing with its own redo log. On the other hand, if performance
is accectable, then it should not hurt either.
Based on the comments above, I'm closing it as "won't fix" (which relates to the warning text which we would like to keep in sync with the one in MySQL version for now; the rest is not a bug).
I don't have a good suggestion for a better wording for the warning; if you do, please either submit a report for MySQL, or add it here so we could file it on your behalf. Please note however that the warning should not be too verbose since it already floods the error log as it is.
On the documentation side, I have added a comment to our mysqld options page in the KB:
https://kb.askmonty.org/en/mysqld-options-full-list/#comment_481