Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-2547

Stop MaxScale during Rest-API query cause process hung

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.21, 2.3.7
    • Fix Version/s: 2.2.22, 2.3.9
    • Component/s: REST-API
    • Labels:
      None

      Description

      Stopping MaxScale will stop all the running workers(including worker#0), and the exit of worker#0 interrupt the processing of microhttpd API query.

      In the resource_handle_request function, microhttpd thread will post a task to worker#0 and wait for the semaphore. But when worker#0 shutdowns without finish processing that semaphore, microhttpd thread will be blocked.

      And then the stopping of MaxScale continues, calls MHD_stop_daemon afterwhile, that will call thread_join on microhttpd threads, then dramatically hung by the blocked thread.

      You may reproduce the bug in this way:

      • First, call any Rest-API in an infinite loop (using Python):

      while True: 
          requests.get('http://127.0.0.1:30000/v1/services', auth=('admin', 'mariadb'))
      

      • Second, using maxadmin interface to call 'shutdown maxscale'.

      In most cases, the MaxScale process will be hung and stop responding to any request. (It's really easy to reproduce for me)

      And here is a sample stack info of a blocked micorhttpd thread:

      do_futex_wait.constprop 0x00007ffff7bccafb
      __new_sem_wait_slow.constprop.0 0x00007ffff7bccb8f
      sem_wait@@GLIBC_2.2.5 0x00007ffff7bccc2b
      maxbase::Semaphore::wait semaphore.hh:115
      maxbase::Worker::call(std::function<void ()>, maxbase::Worker::execute_mode_t) worker.cc:516
      resource_handle_request resource.cc:1337
      Client::process admin.cc:126
      handle_client admin.cc:266
      call_connection_handler connection.c:1834
      MHD_connection_handle_idle connection.c:2909
      call_handlers daemon.c:1154
      MHD_epoll daemon.c:4386
      MHD_select_thread daemon.c:4544
      start_thread 0x00007ffff7bc6e25
      clone 0x00007ffff1e09bad

        Attachments

          Activity

            People

            Assignee:
            markus makela markus makela
            Reporter:
            yunyong lishubing
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration