[MDEV-22570] Split wsrep_provider_options in several options - Jira

Details

Type: Task
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Fix Version/s: 11.0.1
Component/s: Galera, Variables
Labels:
- Preview_11.0

Description

In some cases length 2048 for wsrep_provider_options are not sufficient and the string will be truncated in show variables output and information_schema.system_variables

set global wsrep_provider_options = "base_dir = /opt/mysql/data/; base_host = 192.168.0.1; base_port = 3607; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.causal_keepalive_period = PT1S; evs.debug_log_mask = 0x1; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.info_log_mask = 0; evs.install_timeout = PT7.5S; evs.join_retrans_period = PT1S; evs.keepalive_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.use_aggregate = true; evs.user_send_window = 2; evs.version = 1; evs.view_forget_timeout = P1D; gcache.dir = /opt/mysql/data/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = ssl://0.0.0.0:3607; gmcast.mcast_addr = ; gmcast.mcast_ttl = 1; gmcast.peer_timeout = PT3S; gmcast.segment = 1; gmcast.time_wait = PT5S; gmcast.version = 0; ist.recv_addr = 192.168.0.1; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.linger = PT20S; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 4; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; socket.ssl = YES; socket.ssl_ca = /etc/my.cnf.d/pki/ca-cert.pem; socket.ssl_cert = /etc/my.cnf.d/pki/aevcitdat010-cert.pem; socket.ssl_cipher = ; socket.ssl_compression = YES; ";

set global wsrep_provider_options = "socket.ssl_key = /etc/mysql/ssl/server-key";

UPDATE:
As it is anyhow not optimal to use a system variable for a long list of options it would be better to separate the different options. The list of new option can become quite long, they therefore should be added when the galera library is loaded. This comment is explaining a possible solution.

UPDATE after issue was closed:
The new option

plugin-wsrep-provider=ON

is required to enable this new behavior to split the Galera Cluster system variable wsrep_provider_options into separate Galera Cluster system variables, instead of managing them in one large string.

With

plugin-wsrep-provider=OFF

the old behavior is used, with the given limitation of the max length a system variable can have.

The default is

plugin-wsrep-provider=ON

Attachments

Issue Links

causes

MDEV-30318 galera error messages in mariadb log without galera enabled

Closed

relates to

MDEV-27465 FR: allow to split wsrep_provider_options across multiple option file lines

Closed

MDEV-30120 Update the wsrep_provider_options read_only value in the system_variables table.

Closed

MDEV-30121 The wsrep_provider options are missing from the mariadb --help commandline list

Closed

MDEV-30133 MariaDB startup does not validate plugin-wsrep-provider when wsrep_mode=off or wsrep_provider is not set

Closed

MDEV-30141 The wsrep provider option gmcast.isolate=0 is not in newly added variable list

Closed

MDEV-30147 Remove the deprecated variables from the wsrep_provider_options list

Closed

MDEV-30367 Add I_S.WSREP_PROVIDER_OPTIONS

Closed

MDEV-7331 information_schema.user_variables

Closed

MDEV-30140 Some wsrep provider options only validate the first character of the value when setting it dynamically.

Closed

(5 relates to)

Activity

Ascending order - Click to sort in descending order

Jan Lindström (Inactive) added a comment - 2020-12-16 12:22 - edited

So you can set wsrep_provider_options value to more than 2048 characters or does it truncate it already there ?
We could provide some I_S.WSREP_PROVIDER_OPTIONS interface where wsrep_provider_options is spitted to variable_name and variable_value pairs so that every option is it's own row.
I do not know what we can do for show variables
If wsrep_provider is a plugin we could add these as normal variables (there would be just many of them)

Jan Lindström (Inactive) added a comment - 2020-12-16 12:22 - edited So you can set wsrep_provider_options value to more than 2048 characters or does it truncate it already there ? We could provide some I_S.WSREP_PROVIDER_OPTIONS interface where wsrep_provider_options is spitted to variable_name and variable_value pairs so that every option is it's own row. I do not know what we can do for show variables If wsrep_provider is a plugin we could add these as normal variables (there would be just many of them)

Teemu Ollakka added a comment - 2021-02-12 10:44 - edited

It seems to be possible to implement wsrep_provider plugin to expose wsrep provider options as system variables. This however requires that the provider library is loaded before plugins so that the default options string can be read from the provider and plugin variables can be generated before the plugin in loaded.

I have POC implementation under work, and see that there will be following limitations which this approach will bring:

Provider library must always be loaded at server startup before plugins are initialized.
Provider cannot be reloaded while the server is running.
wsrep_provider and wsrep_provider_options will be read only.
All the variables must be string variables because there is no type information available from the provider.

Do these limitations sound reasonable from usage point of view?

I saw also the following issues with current Galera:

There may be some variables on Galera side which may not allow changing the value back to default once changed.
There are also many variables on Galera side which are no exposed by default unless set in initial provider options.

I believe these can be worked on Galera side.

Teemu Ollakka added a comment - 2021-02-12 10:44 - edited It seems to be possible to implement wsrep_provider plugin to expose wsrep provider options as system variables. This however requires that the provider library is loaded before plugins so that the default options string can be read from the provider and plugin variables can be generated before the plugin in loaded. I have POC implementation under work, and see that there will be following limitations which this approach will bring: Provider library must always be loaded at server startup before plugins are initialized. Provider cannot be reloaded while the server is running. wsrep_provider and wsrep_provider_options will be read only. All the variables must be string variables because there is no type information available from the provider. Do these limitations sound reasonable from usage point of view? I saw also the following issues with current Galera: There may be some variables on Galera side which may not allow changing the value back to default once changed. There are also many variables on Galera side which are no exposed by default unless set in initial provider options. I believe these can be worked on Galera side.

Jan Lindström (Inactive) added a comment - 2021-03-04 13:11 - edited

teemu.ollakka As now wsrep_provider is read only I do not mind that wsrep_provider_options is also read only.

Jan Lindström (Inactive) added a comment - 2021-03-04 13:11 - edited teemu.ollakka As now wsrep_provider is read only I do not mind that wsrep_provider_options is also read only.

Daniele Sciascia added a comment - 2022-09-01 12:45

ralf.gebhardt you will get around 80 - 90 options out of this.
However, these options won't show up in non-Galera setups. Also, the feature can be enabled / disabled.

Daniele Sciascia added a comment - 2022-09-01 12:45 ralf.gebhardt you will get around 80 - 90 options out of this. However, these options won't show up in non-Galera setups. Also, the feature can be enabled / disabled.

Daniele Sciascia added a comment - 2022-09-01 12:53

ralf.gebhardt would the above be acceptable?

Daniele Sciascia added a comment - 2022-09-01 12:53 ralf.gebhardt would the above be acceptable?

Daniel Black added a comment - 2022-09-06 00:10 - edited

If plugin-load-add=wsrep_plugin was used wsrep_provider_plugin_enabled as a variable becomes redundant.

Or better, plugin-load-add=galera (with plugin-dir symlink in packaging) as an implementer of the wsrep API is there (replacing wsrep_provider=x and the fake wsrep_provider plugin). Galera can expose variables in a standard MYSQL_SYSVAR way, which it almost does anyway. Then all provider system variables automaticity become galera_gcache_dir, galera_repl.max_ws_size etc.

For compatibility/legacy interface wsrep_provider_options_check / wsrep_provider_options_check break up the option list and pass each option though the plugin check/update mechanism.

With this the number of new options is reduced (eliminated?) and the facility to treat galera system variables as standard unlimited variables becomes available.

Daniel Black added a comment - 2022-09-06 00:10 - edited If plugin-load-add=wsrep_plugin was used wsrep_provider_plugin_enabled as a variable becomes redundant. Or better, plugin-load-add=galera (with plugin-dir symlink in packaging) as an implementer of the wsrep API is there (replacing wsrep_provider=x and the fake wsrep_provider plugin). Galera can expose variables in a standard MYSQL_SYSVAR way, which it almost does anyway. Then all provider system variables automaticity become galera_gcache_dir , galera_repl.max_ws_size etc. For compatibility/legacy interface wsrep_provider_options_check / wsrep_provider_options_check break up the option list and pass each option though the plugin check/update mechanism. With this the number of new options is reduced (eliminated?) and the facility to treat galera system variables as standard unlimited variables becomes available.

Jan Lindström (Inactive) added a comment - 2022-09-06 05:12

sciascid Can you see if suggested changes could be done please.

Jan Lindström (Inactive) added a comment - 2022-09-06 05:12 sciascid Can you see if suggested changes could be done please.

Daniele Sciascia added a comment - 2022-09-06 07:32

danblack thanks for the feedback! I'll look into your suggestions

Daniele Sciascia added a comment - 2022-09-06 07:32 danblack thanks for the feedback! I'll look into your suggestions

Daniele Sciascia added a comment - 2022-09-14 15:11

danblack I looked into your suggestions.

If plugin-load-add=wsrep_plugin was used wsrep_provider_plugin_enabled as a variable becomes redundant.

I think this is possible.

Or better, plugin-load-add=galera (with plugin-dir symlink in packaging) as an implementer of the wsrep API is there (replacing wsrep_provider=x and the fake wsrep_provider plugin). Galera can expose variables in a standard MYSQL_SYSVAR way, which it almost does anyway. Then all provider system variables automaticity become galera_gcache_dir, galera_repl.max_ws_size etc.

Are you suggesting we load galera library (i.e. libgalera_ssm.so) through mariadb's plugin interface?
Even if we were to do this, we would still not be able to have a list of pre-declared variables (to expose using the standard MYSQL_SYSVAR way), because the library does not have one. Internally, different modules populate their own variables at runtime.

For compatibility/legacy interface wsrep_provider_options_check / wsrep_provider_options_check break up the option list and pass each option though the plugin check/update mechanism.

Notice that in the current pull request wsrep_provider_options is still supported when the plugin is disabled. And when the plugin is enabled wsrep_provider_options becomes read only. This way a user is either using the old wsrep_provider_options or the new single options. Not both at the same. I think this is a sensible choice, do you agree?

Daniele Sciascia added a comment - 2022-09-14 15:11 danblack I looked into your suggestions. If plugin-load-add=wsrep_plugin was used wsrep_provider_plugin_enabled as a variable becomes redundant. I think this is possible. Or better, plugin-load-add=galera (with plugin-dir symlink in packaging) as an implementer of the wsrep API is there (replacing wsrep_provider=x and the fake wsrep_provider plugin). Galera can expose variables in a standard MYSQL_SYSVAR way, which it almost does anyway. Then all provider system variables automaticity become galera_gcache_dir, galera_repl.max_ws_size etc. Are you suggesting we load galera library (i.e. libgalera_ssm.so) through mariadb's plugin interface? Even if we were to do this, we would still not be able to have a list of pre-declared variables (to expose using the standard MYSQL_SYSVAR way), because the library does not have one. Internally, different modules populate their own variables at runtime. For compatibility/legacy interface wsrep_provider_options_check / wsrep_provider_options_check break up the option list and pass each option though the plugin check/update mechanism. Notice that in the current pull request wsrep_provider_options is still supported when the plugin is disabled. And when the plugin is enabled wsrep_provider_options becomes read only. This way a user is either using the old wsrep_provider_options or the new single options. Not both at the same. I think this is a sensible choice, do you agree?

Daniel Black added a comment - 2022-09-15 08:42

> Are you suggesting we load galera library (i.e. libgalera_ssm.so) through mariadb's plugin interface?

yes

> Even if we were to do this, we would still not be able to have a list of pre-declared variables (to expose using the standard MYSQL_SYSVAR way), because the library does not have one. Internally, different modules populate their own variables at runtime.

It could be made could populate the MYSQL_SYSVAR array during the plugin initialization with a restructure of the internal modules of galera.

> Notice that in the current pull request wsrep_provider_options is still supported when the plugin is disabled.

Why is this useful? How is error handling on value bounds done? No other plugin needs such a feature. A careful look at the initialization order would remove this need.

> And when the plugin is enabled wsrep_provider_options becomes read only. This way a user is either using the old wsrep_provider_options or the new single options. Not both at the same. I think this is a sensible choice, do you agree?

So rather restructuring galera to work like a mariadb/mysql plugin, the wsrep_api gained an entire configuration service, its own re-invention of variable types.

So mariadb is getting a lot of padding (debt would be the unfriend term) to avoid galera doing any change or actually behaving like a mariadb plugin.

Daniel Black added a comment - 2022-09-15 08:42 > Are you suggesting we load galera library (i.e. libgalera_ssm.so) through mariadb's plugin interface? yes > Even if we were to do this, we would still not be able to have a list of pre-declared variables (to expose using the standard MYSQL_SYSVAR way), because the library does not have one. Internally, different modules populate their own variables at runtime. It could be made could populate the MYSQL_SYSVAR array during the plugin initialization with a restructure of the internal modules of galera. > Notice that in the current pull request wsrep_provider_options is still supported when the plugin is disabled. Why is this useful? How is error handling on value bounds done? No other plugin needs such a feature. A careful look at the initialization order would remove this need. > And when the plugin is enabled wsrep_provider_options becomes read only. This way a user is either using the old wsrep_provider_options or the new single options. Not both at the same. I think this is a sensible choice, do you agree? So rather restructuring galera to work like a mariadb/mysql plugin, the wsrep_api gained an entire configuration service, its own re-invention of variable types. So mariadb is getting a lot of padding (debt would be the unfriend term) to avoid galera doing any change or actually behaving like a mariadb plugin.

Daniele Sciascia added a comment - 2022-09-15 14:17 - edited

>> Notice that in the current pull request wsrep_provider_options is still supported when the plugin is disabled.

> Why is this useful? How is error handling on value bounds done? No other plugin needs such a feature. A careful look at the initialization order would remove this need.

This comment is confusing to me. Which feature are you talking about?
You originally commented about wsrep_provider_options variable and suggested how to ensure compatibility and still support the legacy interface. And my reply was that the current pull request still supports the legacy interface, when the plugin is disabled. I don't see how this is related to initialization order.

> So rather restructuring galera to work like a mariadb/mysql plugin, the wsrep_api gained an entire configuration service, its own re-invention of variable types.
> So mariadb is getting a lot of padding (debt would be the unfriend term) to avoid galera doing any change or actually behaving like a mariadb plugin.

I'm not sure I agree with this. The "configuration service" is literally one function call.
In its current form, the feature is entirely implemented in galera, wsrep-lib, and wsrep_* files. No interference with the rest of mariadb.
And even if Galera was loaded through plugin interface, you'd still need to interact with it. So assuming that you'd still use wsrep-api, you'd need one call to somehow expose the variables of the library.

Overall, I think what this pull request is doing, is very similar to what you propose, minus the loading of galera through mariadb plugin interface.

Daniele Sciascia added a comment - 2022-09-15 14:17 - edited >> Notice that in the current pull request wsrep_provider_options is still supported when the plugin is disabled. > Why is this useful? How is error handling on value bounds done? No other plugin needs such a feature. A careful look at the initialization order would remove this need. This comment is confusing to me. Which feature are you talking about? You originally commented about wsrep_provider_options variable and suggested how to ensure compatibility and still support the legacy interface. And my reply was that the current pull request still supports the legacy interface, when the plugin is disabled. I don't see how this is related to initialization order. > So rather restructuring galera to work like a mariadb/mysql plugin, the wsrep_api gained an entire configuration service, its own re-invention of variable types. > So mariadb is getting a lot of padding (debt would be the unfriend term) to avoid galera doing any change or actually behaving like a mariadb plugin. I'm not sure I agree with this. The "configuration service" is literally one function call. In its current form, the feature is entirely implemented in galera, wsrep-lib, and wsrep_* files. No interference with the rest of mariadb. And even if Galera was loaded through plugin interface, you'd still need to interact with it. So assuming that you'd still use wsrep-api, you'd need one call to somehow expose the variables of the library. Overall, I think what this pull request is doing, is very similar to what you propose, minus the loading of galera through mariadb plugin interface.

Daniele Sciascia added a comment - 2022-09-16 08:38

The pull request has been updated to get rid of variable wsrep_provider_plugin_enabled, as per first suggestion by danblack:
> If plugin-load-add=wsrep_plugin was used wsrep_provider_plugin_enabled as a variable becomes redundant.

Since this is a built-in plugin, it appears that instead of plugin-load-add, one is supposed to use -~~plugin~~<name of the plugin>=ON, to enable the plugin.
So in our case --plugin-wsrep-provider=ON, enables the feature.

Daniele Sciascia added a comment - 2022-09-16 08:38 The pull request has been updated to get rid of variable wsrep_provider_plugin_enabled, as per first suggestion by danblack : > If plugin-load-add=wsrep_plugin was used wsrep_provider_plugin_enabled as a variable becomes redundant. Since this is a built-in plugin, it appears that instead of plugin-load-add, one is supposed to use - plugin <name of the plugin>=ON, to enable the plugin. So in our case --plugin-wsrep-provider=ON, enables the feature.

Daniel Black added a comment - 2022-10-28 07:37 - edited

edit: needed upstream galera 4.x latest to work.

I still think plugin-load-add=galera would be a good way to start the service
wsrep_provider_options_pc_linger=PT2S - I thought the "wsrep" was notionally an API, and galera was the specific bit, so now we've got an implementation detail in the "wsrep" namespace and no resemblance of proxying it through "wsrep_provider_options".
The galera code now registers centrally the config, so why not populate an MYSQL_SYSVAR array in galera's maria_declare_plugin.
If you can't validate a parameter, maybe its better if the user migrates their config to galera_pc_linger= system variables, and they can be validated on initialization.
this concept would brutally remove the wsrep_provider_options, which by this and linked bugs was never a good choice anyway.

Daniel Black added a comment - 2022-10-28 07:37 - edited edit: needed upstream galera 4.x latest to work. I still think plugin-load-add=galera would be a good way to start the service wsrep_provider_options_pc_linger=PT2S - I thought the "wsrep" was notionally an API, and galera was the specific bit, so now we've got an implementation detail in the "wsrep" namespace and no resemblance of proxying it through "wsrep_provider_options". The galera code now registers centrally the config, so why not populate an MYSQL_SYSVAR array in galera's maria_declare_plugin . If you can't validate a parameter, maybe its better if the user migrates their config to galera_pc_linger= system variables, and they can be validated on initialization. this concept would brutally remove the wsrep_provider_options , which by this and linked bugs was never a good choice anyway.

Daniele Sciascia added a comment - 2022-11-22 13:23

Elsewhere, it was suggested to me that we could just hard code variables in the plugin, instead of fetching them from Galera and populate them dynamically (as it is done in the current proposal).
There are two problems with hard coding the options in the plugin: 1) This would somehow fix the dependency between Mariadb and Galera. You may end up with a version of Galera that has some option, which is not in the MariaDB plugin (or viceversa). It is my understanding that there is no version dependency right now. 2) Even if we do fix this version dependency somehow (let's say by packaging Galera together with MariaDB), any change to Galera configuration must be reflected into the mariadb plugin. Any change in default values, ranges, addition or removal of variable must be done in two places. That's not good, and could easily turn into a maintenance mess.
I don't see many advantages with the hard coding of Galera variables approach . Other than range checking done in server side.
I could try to extend the current approach so that range checking is done in server side. Let me know if that's of interest.

Daniele Sciascia added a comment - 2022-11-22 13:23 Elsewhere, it was suggested to me that we could just hard code variables in the plugin, instead of fetching them from Galera and populate them dynamically (as it is done in the current proposal). There are two problems with hard coding the options in the plugin: 1) This would somehow fix the dependency between Mariadb and Galera. You may end up with a version of Galera that has some option, which is not in the MariaDB plugin (or viceversa). It is my understanding that there is no version dependency right now. 2) Even if we do fix this version dependency somehow (let's say by packaging Galera together with MariaDB), any change to Galera configuration must be reflected into the mariadb plugin. Any change in default values, ranges, addition or removal of variable must be done in two places. That's not good, and could easily turn into a maintenance mess. I don't see many advantages with the hard coding of Galera variables approach . Other than range checking done in server side. I could try to extend the current approach so that range checking is done in server side. Let me know if that's of interest.

Jan Lindström (Inactive) added a comment - 2022-11-28 12:21

branch : bb-10.1-~~MDEV-22570~~-wsrep-provider-options-plugin
commit : 9cf15ff3335f77031b71ca0590d5b2e9ce337f05
Requires Galera library : 26.4.13 commit e6c1388b

Jan Lindström (Inactive) added a comment - 2022-11-28 12:21 branch : bb-10.1- MDEV-22570 -wsrep-provider-options-plugin commit : 9cf15ff3335f77031b71ca0590d5b2e9ce337f05 Requires Galera library : 26.4.13 commit e6c1388b

Jan Lindström (Inactive) added a comment - 2022-11-28 12:25

Config required to use new feature :

[mysqld.1]

wsrep-on=ON

wsrep-cluster-address=gcomm://

wsrep-provider=@ENV.WSREP_PROVIDER

binlog-format=ROW

plugin-wsrep-provider=ON

Jan Lindström (Inactive) added a comment - 2022-11-28 12:25 Config required to use new feature : [mysqld.1] wsrep-on=ON wsrep-cluster-address=gcomm:// wsrep-provider=@ENV.WSREP_PROVIDER binlog-format=ROW plugin-wsrep-provider=ON

Jan Lindström (Inactive) added a comment - 2022-11-28 12:27

Test plan :

Make sure that new variables are not visible if
plugin-wsrep-provider=OFF
or default
Make sure that new variables are visible and you can change them if
plugin-wsrep-provider=ON
Defaults for parameters should be same

Jan Lindström (Inactive) added a comment - 2022-11-28 12:27 Test plan : Make sure that new variables are not visible if plugin-wsrep-provider=OFF or default Make sure that new variables are visible and you can change them if plugin-wsrep-provider=ON Defaults for parameters should be same

Jan Lindström (Inactive) added a comment - 2022-12-09 09:34

Updated branch : preview-10.11-~~MDEV-22570~~-wsrep-provider-options-plugin

Jan Lindström (Inactive) added a comment - 2022-12-09 09:34 Updated branch : preview-10.11- MDEV-22570 -wsrep-provider-options-plugin

Ramesh Sivaraman added a comment - 2023-01-18 12:50

ok to push

Ramesh Sivaraman added a comment - 2023-01-18 12:50 ok to push

Jan Lindström (Inactive) added a comment - 2023-01-24 08:47

Updated branch: bb-11.0-galera-split-wsrep-provider-options
Requires Galera library : 26.4.14 from mariadb-4.x branch commit cf9e5e253

Jan Lindström (Inactive) added a comment - 2023-01-24 08:47 Updated branch: bb-11.0-galera-split-wsrep-provider-options Requires Galera library : 26.4.14 from mariadb-4.x branch commit cf9e5e253

People

Assignee:: Julius Goryavsky

Reporter:: Richard Stracke

Votes:: 0 Vote for this issue

Watchers:: 14 Start watching this issue

Dates

Created:: 2020-05-15 05:52

Updated:: 2024-07-07 22:14

Resolved:: 2023-02-14 15:53

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server