I think the semantics is not flawless, and I expect there will be some confusion (and probably bug reports), particularly due to this combination:

STATIC	dynamic builds only	built dynamically

The problem is, comparing to the old semantics, awful as it was, we are losing one value. Previously we had 4:

• both dynamic and static are allowed (WITH=ON, WITHOUT=OFF) => build static if possible, dynamic otherwise;
• only static is allowed (WITH=ON, WITHOUT=ON) => build static if possible, none otherwise;
• only dynamic is allowed (WITH=OFF, WITHOUT=OFF) => build dynamic if possible, none otherwise;
• nothing is allowed (WITH=OFF, WITHOUT=ON) => do not build.

With the new values, we have
BUILD=NO equals WITH=OFF, WITHOUT=ON
BUILD=DYNAMIC equals WITH=OFF, WITHOUT=OFF
BUILD=STATIC equals WITH=ON, WITHOUT=OFF.

Thus, we have lost WITH=ON, WITHOUT=ON, and semantics of the last line is wrong.
BUILD=STATIC should be equal to WITH=ON, WITHOUT=ON.
And for WITH=ON, WITHOUT=OFF, we should have a 4th value, BUILD="yes", or "auto", or something, which would be the default and which would mean choosing the best possible option.

IIRC we have a similar thing for SSL (yes/system/bundled/no).

Right. I considered the case when both WITH and WITHOUT are used as "undefined
behavior", I'm a bit surprised it actually provided consistent results

I don't know whether it was intentionally implemented to "allow only static"
or it just happened to behave this way. So, I didn't try to replicate this
case in the new implementation.

But ok, I can do it your way too.

We discussed on IRC the following idea:

Currently, if user specifies eg. -DWITH_CASSANDRA_STORAGE_ENGINE=1, but some dependency is missing and Cassandra cannot be built, the cmake run will still succeed. It does give a warning about not using the option, but since the exit code is zero, this might easily be missed by following outputs from eg. build scripts.

So if the new options could be made to fail the cmake run if the requested plugin could not be built, that might be a better semantics.

The original design was based on preserving the same functionality we had with WITH_ and WITHOUT_ but in a GUI-friendly and logically consistent matter. Adding new functionality was not the goal. But If I'll list all possible preference and combinations, I'd get something like

1. do not compile the plugin at all
2. compile statically, if possible, otherwise not at all
3. compile dynamically, if possible, otherwise not at all
4. compile statically, if possible, otherwise dynamically, if possible
5. compile dynamically, if possible, otherwise statically, if possible
6. compile statically, if possible, otherwise abort
7. compile dynamically, if possible, otherwise abort
8. compile statically, if possible, otherwise dynamically, if possible, otherwise abort
9. compile dynamically, if possible, otherwise statically, if possible, otherwise abort

Those in bold were possible in 5.5 and 10.0. They are supported in my current implementation, with the values NO, STATIC, DYNAMIC, AUTO.

But I don't really know how to support all nine variants without making this option too confusing to use. And I don't know whether it makes sense to do that.

Suggestion:

BUILD_PLUGIN_X= {NO|STATIC|DYNAMIC} [, {NO|STATIC|DYNAMIC}] [, {NO|ABORT}]

Where NO is, basically, a noise word, always ignored. It's added for readability. Otherwise, all 9 variants as above can be specified as

1. NO (or the empty string)
2. STATIC,NO (or simply STATIC)
3. DYNAMIC (I will omit NO from now on)
4. STATIC,DYNAMIC
5. DYNAMIC,STATIC
6. STATIC,ABORT
7. DYNAMIC,ABORT
8. STATIC,DYNAMIC,ABORT
9. DYNAMIC,STATIC,ABORT

also, I could allow abbreviating them to the first letter. Making it simply D,A or S,D, for example.

Alternatively, we can keep the list of NO, STATIC, DYNAMIC, AUTO. And add the value of FORCE (which means the same as AUTO, but aborts if the plugin cannot be build. It'd be S,D,A in the previous comment). This FORCE matches the behavior of --plugin-XXX=FORCE as the server command-line option

per irc discussion, I've renamed FORCE to YES

Final implementation is described in the KB: https://mariadb.com/kb/en/mariadb/specifying-which-plugins-to-build/

Use PLUGIN_xxx cmake variables. They can be set on the command line with -DPLUGIN_xxx=value or in the cmake gui. Supported values are

Value	Effect
NO	the plugin will be not compiled at all
STATIC	the plugin will be compiled statically, if supported. Otherwise it will be not compiled.
DYNAMIC	the plugin will be compiled dynamically, if supported. Otherwise it will be not compiled. This is the default behavior.
AUTO	the plugin will be compiled statically, if supported. Otherwise it will be compiled dynamically.
YES	same as AUTO, but if plugin prerequisites (for example, specific libraries) are missing, it will not be skipped, it will abort cmake with an error.