Uploaded image for project: 'MariaDB Connector/Python'
  1. MariaDB Connector/Python
  2. CONPY-266

Make executemany() faster on non-MariaDB servers

    XMLWordPrintable

Details

    • New Feature
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      The MariaDB Python connector appears to use COM_STMT_BULK_EXECUTE for performing executemany() calls. This optimization is only enabled for MariaDB versions 10.2 or newer. For older versions or non-MariaDB servers (e.g. Xpand), the fallback code is to convert it into multiple execute() calls which performs very poorly.

      The MySQL Python connector has an optimization that converts executemany() calls with INSERT statements into a multi-valued INSERT statment. This is one way of improving bulk operation speed but it requires both parsing and SQL string creation.

      Instead, a pipelined approach could be taken. The MariaDB/MySQL protocol is capable of pipelined execution where multiple requests can be sent without first reading their responses. One example of this is the mysql_send_query() API function in the MariaDB Connector/C. The Python connector could be extended to support mysql_send_query() or the COM_STMT_EXECUTE APIs in Connector/C could be improved (here's a pull request that adds the support for it in Connector/C). If the latter approach is taken, the executemany() call can always use the binary protocol and the only difference would be how much overhead the non-optimized COM_STMT_EXECUTE adds compared to the optimized COM_STMT_BULK_EXECUTE.

      Attachments

        Issue Links

          Activity

            People

              georg Georg Richter
              markus makela markus makela
              Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.