[MXS-121] Ability to launch external scripts from Monitor Plugins Created: 2015-05-05  Updated: 2019-12-10  Resolved: 2015-05-09

Status: Closed
Project: MariaDB MaxScale
Component/s: Plugin
Affects Version/s: None
Fix Version/s: 1.2.0

Type: New Feature Priority: Major
Reporter: Dipti Joshi (Inactive) Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Blocks
blocks MXS-189 FAILOVER and SWITCHOVER admin command... Closed
Relates
relates to MXS-133 Testing MaxScale 1.2.0 Features Closed
relates to MXS-141 External scripts launch test Closed
relates to MXS-2791 Special arguments are not passed to M... Closed
relates to MXS-2792 Documentation for MaxScale's monitor ... Closed
Epic Link: MaxScale 1.2 Features

 Description   

All monitor plugin should have ability to launch external scripts upon trigger events.
Example trigger events:

  • Master down
  • New Master found (A Slave has been promoted to a master)

List of Events and Event Parameters here

The script to be launched should be invoked with this command line format
<script_name> --event <event_name> --initiator <initiating_node> --nodelist <server1> <server2> <server3>

Where event_name is the name of the event, initiator is the node whose status change resulted in this event and nodelist is the list of nodes that the plugin is monitoring

The script associated with the events should be configurable and admin user should be able to change the configured script name associated with an event during runtime without requireing MaxScale or Service to restart

Each monitor plugin should also have configurable list of events for which a script will be launched

 Comments   
Comment by markus makela [ 2015-05-05 ]

What's the difference between detect master and new master?

Comment by markus makela [ 2015-05-09 ]

Added full implementation of executable monitor scripts to all monitors.
The monitors take a new script=<path to script> parameter and the script is called with --event=<type> --node=<server> --nodelist=<servers...> command line arguments.

Comment by Dipti Joshi (Inactive) [ 2015-05-12 ]

We should not launch script for every event - only for those events that are configured to launch script for a monitor

i.e. I can configure a MySQLMon instance in MaxScale.cnf to launch script for only Master_down event - and another MySQLMon instance in MaxScle.cnf may be configured to launch script for Master_down, Slave_down as well as Slave_up events

That way we do not keep on forking a script process even when the administrator does not want to do any thing on that event (edited)

Forking script process unnecessarily will cause performance issue, especially when a node keeps on flip-flopping.

So while we want to have framework to be able to launch a script on any event - we should only launch script if the event is configured for it

Comment by Dipti Joshi (Inactive) [ 2015-05-14 ]

markus makela In the documentation, can we add an example of a script (MHA script from Guillaume) that can be launched to initiate failover using this feature ?

Comment by martin brampton (Inactive) [ 2015-05-18 ]

The link in "List of Events and Event Parameters here" points to a document that is not generally available for reading.

Comment by markus makela [ 2015-05-18 ]

Each of the new monitor documents has a list of possible events they receive: https://github.com/mariadb-corporation/MaxScale/tree/MXS-121/Documentation/monitors

Comment by Timofey Turenko [ 2015-05-20 ]

I tried to test it: after a while Maxscale node becomes unresponsive.
I tried both; when script exists (but I'm not 100% sure script is configured properly - execution rights on the file can be missing) and script does not exist

The reason is huge amount of Maxscale processes running:

17627 ? Rs 0:00 sshd: root@pts/0
17629 pts/0 Ss 0:00 -bash
17672 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17673 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17674 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17675 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17676 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17677 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17678 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17679 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17680 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17681 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17682 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17683 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17684 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17685 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17686 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17727 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17728 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17729 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17730 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17731 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17732 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17733 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17734 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17735 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17736 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17737 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17738 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17739 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17740 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17741 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17744 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17745 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17746 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17747 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17748 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17749 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17750 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17752 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17753 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17754 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17755 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17756 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17757 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17758 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17759 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17760 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17761 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17762 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17763 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17764 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17765 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17766 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17767 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17768 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17769 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17770 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17771 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17772 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17773 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17774 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17775 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17776 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17777 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17778 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17779 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17780 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17781 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17782 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17783 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17784 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17785 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17807 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17808 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17809 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17810 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17811 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17812 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17813 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17814 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17815 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17816 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17817 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17818 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17819 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17820 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17821 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17824 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17825 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17826 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17827 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17828 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17829 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17830 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17831 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17832 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17833 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17834 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17835 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17836 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17837 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17838 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17839 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17840 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17841 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17842 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17843 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17844 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17845 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17846 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17847 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17848 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17849 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17850 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17851 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17852 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17853 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17854 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17855 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17856 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17857 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17858 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17859 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17860 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17861 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17862 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17863 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17864 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17865 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17866 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17867 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17868 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17869 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17870 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17871 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17872 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17873 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17874 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17875 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17876 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17877 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17878 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17879 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17880 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17881 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17882 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17883 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17884 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17885 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17886 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17887 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17888 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17889 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17890 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17891 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17892 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17893 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17894 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17896 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17897 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17898 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17899 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale

Comment by Timofey Turenko [ 2015-05-20 ]

Still a number of processes are visible:

1557 ? Ss 0:00 /usr/sbin/sshd
17572 ? Ssl 0:05 /usr/local/mariadb-maxscale/bin/maxscale
17588 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17589 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17590 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17591 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17596 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17597 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17598 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17599 ? S 0:00 /usr/local/mariadb-maxscale/bin/maxscale
17612 ? Rs 0:00 sshd: root@pts/0

Comment by Dipti Joshi (Inactive) [ 2015-06-04 ]

markus makela Can we put link to unit test cases here ?

Comment by markus makela [ 2015-06-10 ]

Monitor script test cases: https://docs.google.com/spreadsheets/d/1W2A5H0k98xLVPpd4unB_4Dd2Xjtu8y9l447hm_6fLg0/edit#gid=0

Comment by Dipti Joshi (Inactive) [ 2015-06-10 ]

Thanks markus makela, have you tried using failover script from Guillaume as the external script ?

Comment by markus makela [ 2015-06-10 ]

Not yet

Comment by Dipti Joshi (Inactive) [ 2015-06-16 ]

markus makela, following comments apply to these four documents (a) https://github.com/mariadb-corporation/MaxScale/blob/develop/Documentation/monitors/MySQL-Monitor.md (b) https://github.com/mariadb-corporation/MaxScale/blob/develop/Documentation/monitors/Galera-Monitor.md (c) https://github.com/mariadb-corporation/MaxScale/blob/develop/Documentation/monitors/NDB-Cluster-Monitor.md (d) https://github.com/mariadb-corporation/MaxScale/blob/develop/Documentation/monitors/MM-Monitor.md

(1) "Script Events" table at the end of document has a typo in row "server_down" "A server with no assigned role has done down" - Change "done down" to "gone down"

(2) Please add an example that uses a script which will send an email to a pre-configured system admin account when a master node does down

Comment by markus makela [ 2015-06-16 ]

Here is an example which uses a shell script to send an email to an admin when a master or a slave server goes down: https://github.com/mariadb-corporation/MaxScale/blob/develop/Documentation/monitors/MySQL-Monitor.md

Comment by Dipti Joshi (Inactive) [ 2015-06-16 ]

Thanks markus makela!

Can you have the mail_to_admin.sh actually print the event name, the node that went down, and the list of the nodes that are passed to the script ? Some thing like:

"master_down detected
Down node = <master server name>
Node list = <server1> <server2>"

"slave_down detected
Down node = <slave server name>
Nodelist = <server 1> <server 2> <server 3>"

Thanks,
Dipti

Comment by Guillaume Lefranc [ 2015-06-18 ]

Hello,

I think that the options format is not practical. Today it is like this:
<script_name> --event <event_name> --initiator <initiating_node> --nodelist <server1> <server2> <server3>

The above assumes that the destination scripts understands the --event, --initiator and --nodelist options.
That's not really practical if you want to use already existing solutions which don't understand those options. For example right now if I want to interface maxscale with MHA or with MariaDB Replication Manager i have to use a wrapper script which parses those options and passes them to the destination program.

I'd rather have those options customizable in the directive and use placeholders, e.g.

script=/usr/bin/failover.sh --myprogramoption1 $event_name --myprogramoption2 $initiating_node

Thanks
Guillaume

Comment by Guillaume Lefranc [ 2015-06-18 ]

I have additional issues performing with failover with scripts, notably because the script passes node names instead of IP addresses. As node names can be absolutely any kind of string and not necessarily resolvable hostnames, I'd rather have the function pass the address= directive as argument, which makes much more sense to me.

Comment by Guillaume Lefranc [ 2015-06-19 ]

When failover script returns an error, process becomes a zombie:
root 2258 0.0 0.0 0 0 ? Z 11:10 0:00 [failover.sh] <defunct>
root 2698 0.0 0.0 0 0 ? Z 12:25 0:00 [failover.sh] <defunct>

Please read the following: http://lubutu.com/code/spawning-in-unix and apply one of the suggested techniques at your discretion

Thanks

Comment by Dipti Joshi (Inactive) [ 2015-06-19 ]

tanj can you please paste the script you launched from MaxScale for your testing ?

Comment by Guillaume Lefranc [ 2015-06-19 ]

I have another (albeit small) issue with the current implementation:
--nodelist is a list of all nodes.
However, I mostly know this list already. In the case of a master_down event, I would prefer to have the list of remaining nodes only. Why?
To perform failover, I need to elect a master among the remaining nodes.
If I want to obtain the full node list, I just need to concatenate "initiator" and "nodelist".

Thanks

Comment by Dipti Joshi (Inactive) [ 2015-08-04 ]

markus makela tturenko Can this be closed now ?

Generated at Thu Feb 08 03:56:56 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.