InnoDB part of fix for b89feb5b28176daa27c4a3bdc32f2c70f6a996e2 was lost in "MDEV-11487 Revert InnoDB internal temporary tables from WL#7682" (see c868acdf656213cdc081c4c965a1bcf3d22558bb).
To protect against concurrent UNINSTALL PLUGIN InnoDB must call thd_set_ha_data() whenever it wants to modify it's ha_data pointer.
Background threads most probably need special handling: we don't want to increment plugin reference counter for reference from plugin itself. That is we shouldn't use thd_set_ha_data() for background threads.