Not only but all sort fail.
Indeed, the way result file names are retrieved does not permit position read. Therefore, DIR tables cannot be indexed or sorted.
This will be indicated in the documentation and a better error message returned.

I got an impression that all other cases are working properly, there was just a problem with varchar types, e.g. when I add size in order by, results are correct.

It must be depending on how MariaDB does sorting. In some cases it is done internally on a result set kept in memory, in some other cases it is done by storing positions, sort them, and re-reading by position.
This depends on the memory size needed and perhaps on the column type.

I have two suggestions regarding this issue.

First, I see that you have changed the error message to a more suitable "Not supported by this table type", which is good. However, it comes with the error code 1105, which is ER_UNKNOWN_ERROR, which might be not great for various tools which rely on the error code rather than the message. Maybe it's better to use ER_ILLEGAL_HA instead (Storage engine %s of the table %`s.%`s doesn't have this option)?

If on some reason you want to keep 1105, then – since you provide a custom message anyway – it makes sense to make a more comprehensive one. E.g. what is not supported? Although, I suppose it might be difficult to specify since it's a common function used in all types of queries.

Secondly, I think such limitations need to be documented, since it's really not obvious what exactly is wrong with a query, why a simple query like that does not work, but a seemingly identical query with somewhat different types works (and it's of course impossible to squeeze it all into the error message). So, I'm re-opening it for documentation purposes.