Yes , have two plugins of different types in one .so (or .dll) file!
You said that:
What do you mean, you wrote a plugin that has two types? A plugin cannot have two types. Do you mean, you have two plugins of different types in one .so (or .dll) file? This is fine, many storage engines do that.
That line works when somebody does INSTALL PLUGIN ..., it is reached of the plugin was successfully loaded. It means "if the plugin name was specified (that is, it's INSTALL PLUGIN), then we're done."
Whenever mariadb run , it will load the .so (or .dll) file , then only the first plugins type is handed, the second plugin type is pass by the code:
DBUG_RETURN(FALSE); // all done