Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-21708

Raise hard coded limit of 16M for 'net_buffer_length' in mysqldump

    XMLWordPrintable

Details

    Description

      I open this as task but it could be considered a bug as well in the whole.

      mysqldump has an hardcoded value of 16MB for net_buffer_length, indeed you can't use a value larger than 16MB:

      mysqldump -u UUUU -pXXXX --debug-info dbname --net_buffer_length=32M --complete-insert --single-transaction

      Warning: option 'net_buffer_length': unsigned value 33554432 adjusted to 16777216

      In the mysqldump.c source file you have:

      static struct my_option my_long_options[] =
      ....
      ....

      {"net_buffer_length", OPT_NET_BUFFER_LENGTH, "The buffer size for TCP/IP and socket communication.", &opt_net_buffer_length, &opt_net_buffer_length, 0, GET_ULONG, REQUIRED_ARG, 1024*1024L-1025, 4096, 16*1024L*1024L, MALLOC_OVERHEAD-1024, 1024, 0}

      ,

      Changing the 16*1024L*1024L to 32*1024L*1024, works as expected.
      I have the impression that the hardcoded 16M value is legacy and not up to current times.

      Also, the manual says some confusing things:

      " If you increase this variable, ensure that the server net_buffer_length system variable has a value at least this large."

      But current maximum value for net_buffer_length is 16M on mysqldump and 1M in the server.

      Also I don't understand the recommendation because net_buffer_length is just the initial in/out(to client) buffers size which can grow to max_allowed_packet.
      It should say:
      "If you increase this variable, ensure that the server max_allowed_packet system variable has a value at least this large."

      =======================================
      [mysqldump]
       
      --net-buffer-length=value
       
      The initial size of the buffer for client/server communication. When creating multiple-row INSERT statements (as with the --extended-insert or --opt option), mysqldump creates rows up to --net-buffer-length bytes long. If you increase this variable, ensure that the MySQL server net_buffer_length system variable has a value at least this large.
      =======================================
       
      =======================================
      [server]
       
      Each client thread is associated with a connection buffer and result buffer. Both begin with a size given by net_buffer_length but are dynamically enlarged up to max_allowed_packet bytes as needed. The result buffer shrinks to net_buffer_length after each SQL statement.
       
      This variable should not normally be changed, but if you have very little memory, you can set it to the expected length of statements sent by clients. If statements exceed this length, the connection buffer is automatically enlarged. The maximum value to which net_buffer_length can be set is 1MB.
      =======================================
      

      As a side note I'm wondering if the variable name in mysqldump is the most appropriate.
      While it's true that in a way or another affects the outgoing buffer of the server more than the starting buffer size as the manual states it seems more to set the exact chunk size of the extended inserts of mysqldump, not just the beginning size of the buffer but the maximum size of the chunk.
      So wondering if a name like this would be more meaningful: dump_chunk_size
      (in case I'd open a new jira task)

      Attachments

        Activity

          People

            Unassigned Unassigned
            claudio.nanni Claudio Nanni
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.