Current documentation on minor version upgrades (example) say to use mysql_upgrade as a final step and details the use of database credentials but not that it may also require the unix user running the command have write permission to the mysql data directory. This is not mentioned in mysql_upgrade's documentation either.
Running mysql_upgrade with correct db credentials but a user without write access to the datadir results in most of the process looking complete except for a one line error message relating to writing an innodb file. There may be further files to be written after that point. This is very easy to miss when doing a rolling upgrade of a whole set of servers in a galera cluster.
This was observed on Centos7 with SELinux disabled. It was a direct (and slightly problematic) upgrade from 10.1 to 10.4, which may be a factor if this doesn't reproduce in closer upgrades.
As well in this 'done but not done' state, mysql_upgrade doesn't get marked to not be run again, so running it again with correct permissions will complete normally and mark it to not re-run if attempted again. This is how I came to note one server had completed it correctly (and I do recall accidentally running the upgrade as root on one machine.)
In our case, this resulted in a sporadic crash relating to wsrep transactions:
Jul 29 14:00:06 live1 mysqld: mysqld: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.6/wsrep-lib/src/transaction.cpp:951: void wsrep::transaction::after_replay(const wsrep::transaction&): Assertion `other.state() == s_committed || other.state() == s_aborted' failed.
I'm just including this as a possible tell elsewhere. It happened only on the servers that hadn't completed mysql_upgrade properly. If we continue to experience this I'll bring more info but it might be fixed.
There are also no warnings logged about tables in that state, entailing that tables did get upgrade and just some other tasks possibly remain incomplete.