Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-4921

Add a interface to mysql protocol and DAEMON plugins

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Won't Fix
    • Fix Version/s: 10.1.0
    • Component/s: None
    • Labels:
      None

      Description

      Add a interface to plugins that allow a communication between plugin at server side and mysql client

      The "mysql" keyword here can be mysql/mariadb (or maybe only mariadb =] )
      I will use the php mysql* function to explain this MDEV

      THIS IS VERY IMPORTANT TO DAEMON PLUGINS!
      The mysql server will be used as a "PROXY" between mysql client api and plugin server side interface


      with this MDEV we can add, via plugins, a nosql interface using ONLY ONE mysql client connection

      the main goals here are:

      1. only one connection to mysql server (easier to firewalls and developers since we only need to configure one connection, and not a connection to mysql server and a connection to a memcached/handler socket/nosql server)
      2. connection via tcp/ip, unix socket, windows pipes, or anything else mysql protocol can use
      3. can use mysql embedded =]
      4. can use ssl security layer of mysql protocol, transparent to plugins (no external link to openssl or different ssl from mysql server) =)
      5. since we must have a mysql link, plugin communication is secured via mysql_connect() auth system using user, password, host, auth plugins and others kinds of validation
      6. we don't need to configure clients that can or can't connect to our plugin since every user that will connect here is know in mysql server user tables
      7. we don't need external links inside plugin source code to libevent, socket, or anyother communication api that allow a client-server communication to client side
      8. we can use it on linux and windows and any mysql supported operation system \o/
      9. many many others goals

      problems:

      1. maybe some one will want many connections, and we could/should have a limit to this connections
      2. others buffers / connection limits that must be considered
      3. others client permissions that must be considered
      4. this allow to a bad user create a http proxy or a security problem at server side, but since it's open source maybe DBA will read the source of plugin before allowing it to execute
      5. maybe some plugins will want "query cache", it's not the mysql server query cache, but a interface to it... when a table change contact plugin that table changed to invalidate the plugin cached too, this allow a daemon plugin query cache that is in sync with mysql daemon tables

      ------------------
      SERVER SIDE API
      we need some functions to plugins.... (i don't know what plugin can use it...)

      plugin_register_connection_name(
        "some unique connection name, like the TCP port for IP protocol, or the unix socket file", 
        plugin function to create connections,
        plugin function to kill connection,
        plugin function to receive packets,
        plugin function to query cache table / partition prune / or query cache reset
      );

      plugin_create_connection_function( THD being_used, server_unique_plugin_connection_id)

      this function will be called when client create a connection
      the server unique plugin connection id is created before calling it, it's unique in entire server (mutex must be used here)
      the function will return OK - connection created, NO - connection rejected
      maybe we could add a ERROR code and a ERROR message to client side plugin api

      plugin_drop_connection_function( THD being_used, server_unique_plugin_connection_id, type)

      this function will be called to drop communication and free resources used
      use type variable parameter to check why connection is being droped
      the plugin connection can be droped when:

      1. client side api drop a connection to plugin
      2. dba send a "KILL PLUGIN CONNECTION" (new command) to kill this plugin connection
      3. thread is killed in server (KILL command)
      4. server shutdown is being executed
      5. plugin is being removed (UNINSTALL PLUGIN command)

      plugin_new_packet_function(THD being_used, server_unique_plugin_connection id, packet, length)

      this function is called when client send a raw command to plugin
      here the packet have the right order and right checksum (tcp/ip)
      we must return a RAW packet or a QUERY return (used by mysql_query / mysql_fetch_assoc / mysql_num_rows client side api)
      how we return isn't defined... maybe three functions...

      1. one for tell what kind of return (RAW or QUERY)
      2. one to send raw packets
      3. one to send the result (like a information schema plugin return - http://dev.mysql.com/doc/internals/en/com-query-response.html)

      plugin_send_to_client(server_unique_plugin_connection_id, packet, length)

      here we send a packet to client
      that's not a client->plugin return, this is a new plugin->client message
      for example a video stream, send server informations like a thread created, or a slow query executed, or anything else that occur first at server, and client don't need to pool server every second to know if something changed...

      plugin_close_connection(server unique plugin connection id)

      this can be used by plugin to kill a connection to client side api
      maybe we could send a packet to client to tell that the connection was dropped by plugin

      plugin_query_cache_prune(parameters)

      this function will be called when:
      a table/partition change that cause a query cache invalidation
      query cache was reseted (truncated)

      ------------------
      DBA INTERFACE
      1)we must know if a thread have plugin connections associated with it or not...
      this can be done via information schema tables...

      UNIQUE PLUGIN CONNECTION ID THREAD ID PLUGIN NAME CONNECTION NAME STATE
      1 2 1 "slow query monitor" SENDING DATA TO CLIENT
      2 2 1 "memcached" OPEN
      3 2 1 "handler socket write" OPEN
      4 2 1 "handler socket read" OPEN

      STATE = OPEN / CLOSING / RECEIVED DATA FROM CLIENT / RETURNING DATA TO CLIENT / SENDING DATA TO CLIENT

      2)we can kill a plugin connection ...
      using unique plugin connection id:

      KILL PLUGIN CONNECTION <unique plugin connection id>, <unique plugin connection id>, <unique plugin connection id>

      kill one connection (or a list of connections)

      using thread id

      KILL PLUGIN CONNECTION THREAD <thread id>

      kill all connection from a thread

      using user name

      KILL PLUGIN CONNECTION USER <user>

      kill all connection from a user

      ------------------
      CLIENT SIDE API

      Today we have some commands at mysql protocol http://dev.mysql.com/doc/internals/en/text-protocol.html
      this MDEV will add 5 new commands (maybe more if we need):

      1)PLUGIN_CREATE_CONNECTION
      2)PLUGIN_DROP_CONNECTION
      3)PLUGIN_CLIENT_SEND
      4)PLUGIN_SERVER_RETURN
      5)PLUGIN_SERVER_SEND


      1) Create a new communication with plugin (PLUGIN_CREATE_CONNECTION)
      this command will create a new ID that client will use to communicate between mysql client, and the plugin at mysql server, and the plugin to client, example:

      $link = mysql_connect('localhost', 'mysql_user', 'mysql_password');
      $plugin_socket = mysql_plugin_connect($link,"plugin_connection_name");  // <---HERE

      the mysql_plugin_connect call mysql server to create a new communication with plugin
      mysql server will check if any plugin have a interface registered as "plugin_connection_name"
      if yes, it will ask plugin if we could or couldn't create a new plugin connection
      the return of this function is a unique id that represent the plugin and connection being used in entire server (like query_id), this must be mutex locked

      2) Drop communication with plugin
      based on mysql_plugin_connect(), we need a plugin close command:

      $link = mysql_connect('localhost', 'mysql_user', 'mysql_password');
      $plugin_socket = mysql_plugin_connect($link,"plugin_connection_name");
      mysql_plugin_close($link,$plugin_socket);

      here the connection with $plugin_socket will be closed at mysql server side
      plugin will be called to free all resources used in this connection

      3,4) sending packets to plugin (PLUGIN_CLIENT_SEND), and plugin returning packets to client (PLUGIN_SERVER_RETURN)

      now that we have a link between mysql client, and plugin, we need to send and receive data...
      this can be done in many ways... here i will give an example, but we can improve it (like always =] )
      the communication can (must check if it's easy) start at plugin side, or at client side

      when returning a client command (a plugin return):
      the plugin can send a query result (like a SQL command) that can be used with mysql_fetch_assoc, mysql_num_rows, operating like a mysql_query function
      the plugin can send a raw command (like a raw tcp/ip protocol) that can be used to create http proxy, nosql protocol, and anything using client/server idea

      $link = mysql_connect('localhost', 'mysql_user', 'mysql_password');
      $plugin_socket = mysql_plugin_connect($link,"plugin_connection_name");
      while(true){
        $return=mysql_plugin_send($link,$plugin_socket,"here i'm sending a raw text command to plugin");
        //$return have information about what kind of return we have, a mysql command, or a raw command
        if(mysql_plugin_return_type($return)==MYSQL_PLUGIN_RETURN_RAW){ // we have a raw packet
          $data=mysql_plugin_return_read($return); // get raw data
          mysql_plugin_return_free($return); // free return buffer
          // here we can process data (maybe create a handler socket protocol or a memcache protocol or any other nonsql protocol
        }else{  // we have a query return
          for($i=0;$i<mysql_num_rows($return);$i++){
             $data=mysql_fetch_assoc($return);
          }
          mysql_free_result($return);
        }
      }
      mysql_plugin_close($plugin_socket);

      now we have 3 new functions (they are sync and blocking in this example, but we can add more functions in future...)

      $return=mysql_plugin_send($link,$plugin_socket,"here i'm sending a raw text command to plugin");

      mysql_plugin_send(<mysql link>,<plugin socket id>, <message>)
      this function send raw data to plugin and wait a return,
      the return can be used by three others functions:

      mysql_plugin_return_type($return) 

      this function tell if we have:
      raw return (MYSQL_PLUGIN_RETURN_RAW) or
      query return ("COM_QUERY Response" like mysql_query function - http://dev.mysql.com/doc/internals/en/com-query-response.html )

      $data=mysql_plugin_return_read($return); 

      this function will read raw data from raw return buffer

      mysql_plugin_return_free($return);

      this function will clean raw read buffer

      5) PLUGIN_SERVER_SEND, plugin starting a communication to client
      i didn't think about this kind of problem yet....
      but we can use to know for example if a table have being changed
      or if some trigger happened, this is usefull in situations that we 'pool' server
      with queries to know what's happening, for example we could create a plugin that
      report when a new client connect, or when a new query start, or when a slow query
      is being executed

      the main problem here is how to implement this at client side, since api don't have a async call (or have?)...
      this is a solution like libevent, select, and many others client/server apis

      maybe we could have a two functions:

      mysql_plugin_check_new_message()

      function to know if we received a new plugin packet

      mysql_plugin_read_message() 

      to read the message

      or a async api...

      mysql_plugin_register_async_receive( function being called when new packet is received )

      this will register a function that will be called when a new packet from plugin is received

      and use the mysql_plugin_check and read functions to read

      to return to server we can use the PLUGIN_CLIENT_SEND, PLUGIN_SERVER_RETURN flow (3,4)


      that's all

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            rspadim roberto spadim
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: