Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24313

Hang with innodb_use_native_aio=0 and innodb_write_io_threads=1

Details

    Description

      elenst reports that her tests are hanging.

      The reason seems to be the following combination of parameters:

      • innodb_write_io_threads=1
      • innodb_use_native_aio=0
      • innodb_doublewrite=1

      To add confusion, InnoDB will claim that it is using native AIO (because innodb_use_native_aio=1 by default), but a little later, it will silently change to simulated asynchronous I/O (innodb_use_native_aio=0) due to io_setup() failure.

      The hang is possible, because starting with MDEV-23855, the doublewrite buffer issues asynchronous writes. On buf_dblwr_t::flush_buffered_writes_completed(), up to 128 page writes may be submitted. But, there may already be 128 outstanding page writes. The thread would then hang here, because the maximum number of outstanding requests per thread is 256:

      mariadb-10.5.8

      		 
      #14 0x000055c699979745 in io_slots::acquire (this=0x55c69c565d50) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:103
      		 
      #15 0x000055c699977bfe in os_aio (type=..., buf=0x7f9c7847c000, offset=4128768, n=16384) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:4211
      		 
      #16 0x000055c699c14ce0 in fil_space_t::io (this=0x55c69cd565e8, type=..., offset=4128768, len=16384, buf=0x7f9c7847c000, bpage=0x7f9c78009ab0)
      		 
          at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/fil/fil0fil.cc:3431
      		 
      #17 0x000055c699b9113c in buf_dblwr_t::flush_buffered_writes_completed (this=0x55c69b62b960 <buf_dblwr>, request=...) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/buf/buf0dblwr.cc:678
      		 
      #18 0x000055c699c14fb9 in fil_aio_callback (request=...) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/fil/fil0fil.cc:3466
      		 
      #19 0x000055c699976827 in io_callback (cb=0x55c69c565f40) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:3918
      		 
      #20 0x000055c699ca75dd in tpool::simulated_aio::simulated_aio_callback (param=0x55c69c565f40) at /home/mariadb/semaphore1/10.5-enterprise/tpool/aio_simulated.cc:162

      A possible workaround could be to make innodb_write_io_threads=2 the minimum value.

      Thanks to wlad for participating in the debugging.

      Attachments

        Issue Links

          is caused by

          Task - A task that needs to be done. MDEV-16264 Implement a common work queue for InnoDB background tasks

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23855 InnoDB log checkpointing causes regression for write-heavy OLTP

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25953 Tpool - prevent potential deadlock in simulated AIO

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            marko Marko Mäkelä created issue -
            marko Marko Mäkelä made changes -
            Field Original Value New Value
            marko Marko Mäkelä made changes -
            Description [~elenst] reports that her tests are hanging.

            The reason seems to be the following combination of parameters:
            * {{innodb_read_io_threads=1}}
            * {{innodb_use_native_aio=0}}
            * {{innodb_doublewrite=1}}

            To add confusion, InnoDB will claim that it is using native AIO, but a little later, it will silently change to simulated asynchronous I/O due to {{io_setup()}} failure.

            The hang is possible, because starting with MDEV-23855, the doublewrite buffer issues asynchronous writes. On {{buf_dblwr_t::flush_buffered_writes_completed()}}, up to 128 page writes may be submitted. But, there may already be 128 outstanding page writes. The thread would then hang here, because the maximum number of outstanding requests per thread is 256:
            {noformat:title=mariadb-10.5.8}
            #14 0x000055c699979745 in io_slots::acquire (this=0x55c69c565d50) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:103
            #15 0x000055c699977bfe in os_aio (type=..., buf=0x7f9c7847c000, offset=4128768, n=16384) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:4211
            #16 0x000055c699c14ce0 in fil_space_t::io (this=0x55c69cd565e8, type=..., offset=4128768, len=16384, buf=0x7f9c7847c000, bpage=0x7f9c78009ab0)
                at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/fil/fil0fil.cc:3431
            #17 0x000055c699b9113c in buf_dblwr_t::flush_buffered_writes_completed (this=0x55c69b62b960 <buf_dblwr>, request=...) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/buf/buf0dblwr.cc:678
            #18 0x000055c699c14fb9 in fil_aio_callback (request=...) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/fil/fil0fil.cc:3466
            #19 0x000055c699976827 in io_callback (cb=0x55c69c565f40) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:3918
            #20 0x000055c699ca75dd in tpool::simulated_aio::simulated_aio_callback (param=0x55c69c565f40) at /home/mariadb/semaphore1/10.5-enterprise/tpool/aio_simulated.cc:162
            {noformat}

            A possible workaround could be to make {{innodb_read_io_threads=2}} the minimum value.

            Thanks to [~wlad] for participating in the debugging.             		[~elenst] reports that her tests are hanging.

            The reason seems to be the following combination of parameters:
            * {{innodb_write_io_threads=1}}
            * {{innodb_use_native_aio=0}}
            * {{innodb_doublewrite=1}}

            To add confusion, InnoDB will claim that it is using native AIO, but a little later, it will silently change to simulated asynchronous I/O due to {{io_setup()}} failure.

            The hang is possible, because starting with MDEV-23855, the doublewrite buffer issues asynchronous writes. On {{buf_dblwr_t::flush_buffered_writes_completed()}}, up to 128 page writes may be submitted. But, there may already be 128 outstanding page writes. The thread would then hang here, because the maximum number of outstanding requests per thread is 256:
            {noformat:title=mariadb-10.5.8}
            #14 0x000055c699979745 in io_slots::acquire (this=0x55c69c565d50) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:103
            #15 0x000055c699977bfe in os_aio (type=..., buf=0x7f9c7847c000, offset=4128768, n=16384) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:4211
            #16 0x000055c699c14ce0 in fil_space_t::io (this=0x55c69cd565e8, type=..., offset=4128768, len=16384, buf=0x7f9c7847c000, bpage=0x7f9c78009ab0)
                at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/fil/fil0fil.cc:3431
            #17 0x000055c699b9113c in buf_dblwr_t::flush_buffered_writes_completed (this=0x55c69b62b960 <buf_dblwr>, request=...) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/buf/buf0dblwr.cc:678
            #18 0x000055c699c14fb9 in fil_aio_callback (request=...) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/fil/fil0fil.cc:3466
            #19 0x000055c699976827 in io_callback (cb=0x55c69c565f40) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:3918
            #20 0x000055c699ca75dd in tpool::simulated_aio::simulated_aio_callback (param=0x55c69c565f40) at /home/mariadb/semaphore1/10.5-enterprise/tpool/aio_simulated.cc:162
            {noformat}

            A possible workaround could be to make {{innodb_write_io_threads=2}} the minimum value.

            Thanks to [~wlad] for participating in the debugging.
            Summary Hang with innodb_use_native_aio=0 and innodb_read_io_threads=1 Hang with innodb_use_native_aio=0 and innodb_write_io_threads=1
            marko Marko Mäkelä made changes -
            Description [~elenst] reports that her tests are hanging.

            The reason seems to be the following combination of parameters:
            * {{innodb_write_io_threads=1}}
            * {{innodb_use_native_aio=0}}
            * {{innodb_doublewrite=1}}

            To add confusion, InnoDB will claim that it is using native AIO, but a little later, it will silently change to simulated asynchronous I/O due to {{io_setup()}} failure.

            The hang is possible, because starting with MDEV-23855, the doublewrite buffer issues asynchronous writes. On {{buf_dblwr_t::flush_buffered_writes_completed()}}, up to 128 page writes may be submitted. But, there may already be 128 outstanding page writes. The thread would then hang here, because the maximum number of outstanding requests per thread is 256:
            {noformat:title=mariadb-10.5.8}
            #14 0x000055c699979745 in io_slots::acquire (this=0x55c69c565d50) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:103
            #15 0x000055c699977bfe in os_aio (type=..., buf=0x7f9c7847c000, offset=4128768, n=16384) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:4211
            #16 0x000055c699c14ce0 in fil_space_t::io (this=0x55c69cd565e8, type=..., offset=4128768, len=16384, buf=0x7f9c7847c000, bpage=0x7f9c78009ab0)
                at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/fil/fil0fil.cc:3431
            #17 0x000055c699b9113c in buf_dblwr_t::flush_buffered_writes_completed (this=0x55c69b62b960 <buf_dblwr>, request=...) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/buf/buf0dblwr.cc:678
            #18 0x000055c699c14fb9 in fil_aio_callback (request=...) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/fil/fil0fil.cc:3466
            #19 0x000055c699976827 in io_callback (cb=0x55c69c565f40) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:3918
            #20 0x000055c699ca75dd in tpool::simulated_aio::simulated_aio_callback (param=0x55c69c565f40) at /home/mariadb/semaphore1/10.5-enterprise/tpool/aio_simulated.cc:162
            {noformat}

            A possible workaround could be to make {{innodb_write_io_threads=2}} the minimum value.

            Thanks to [~wlad] for participating in the debugging.             		[~elenst] reports that her tests are hanging.

            The reason seems to be the following combination of parameters:
            * {{innodb_write_io_threads=1}}
            * {{innodb_use_native_aio=0}}
            * {{innodb_doublewrite=1}}

            To add confusion, InnoDB will claim that it is using native AIO (because {{innodb_use_native_aio=1}} by default), but a little later, it will *silently change to simulated asynchronous I/O ({{innodb_use_native_aio=0}})* due to {{io_setup()}} failure.

            The hang is possible, because starting with MDEV-23855, the doublewrite buffer issues asynchronous writes. On {{buf_dblwr_t::flush_buffered_writes_completed()}}, up to 128 page writes may be submitted. But, there may already be 128 outstanding page writes. The thread would then hang here, because the maximum number of outstanding requests per thread is 256:
            {noformat:title=mariadb-10.5.8}
            #14 0x000055c699979745 in io_slots::acquire (this=0x55c69c565d50) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:103
            #15 0x000055c699977bfe in os_aio (type=..., buf=0x7f9c7847c000, offset=4128768, n=16384) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:4211
            #16 0x000055c699c14ce0 in fil_space_t::io (this=0x55c69cd565e8, type=..., offset=4128768, len=16384, buf=0x7f9c7847c000, bpage=0x7f9c78009ab0)
                at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/fil/fil0fil.cc:3431
            #17 0x000055c699b9113c in buf_dblwr_t::flush_buffered_writes_completed (this=0x55c69b62b960 <buf_dblwr>, request=...) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/buf/buf0dblwr.cc:678
            #18 0x000055c699c14fb9 in fil_aio_callback (request=...) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/fil/fil0fil.cc:3466
            #19 0x000055c699976827 in io_callback (cb=0x55c69c565f40) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:3918
            #20 0x000055c699ca75dd in tpool::simulated_aio::simulated_aio_callback (param=0x55c69c565f40) at /home/mariadb/semaphore1/10.5-enterprise/tpool/aio_simulated.cc:162
            {noformat}

            A possible workaround could be to make {{innodb_write_io_threads=2}} the minimum value.

            Thanks to [~wlad] for participating in the debugging.
            marko Marko Mäkelä made changes -
            Labels hang hang regression-10.5
            Priority Major [ 3 ] Blocker [ 1 ]
            marko Marko Mäkelä made changes -
            elenst Elena Stepanova made changes -
            Attachment mysql.err [ 55221 ]
            marko Marko Mäkelä made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            marko Marko Mäkelä made changes -
            issue.field.resolutiondate 2020-12-14 15:16:22.0 2020-12-14 15:16:22.831
            marko Marko Mäkelä made changes -
            Fix Version/s 10.5.9 [ 25109 ]
            Fix Version/s 10.5 [ 23123 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            wlad Vladislav Vaintroub made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 116301 ] MariaDB v4 [ 158632 ]

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.