Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-7126

replication slave - deadlock in terminate_slave_thread with stop slave and show variables of replication filters and show global status

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • 5.5.37, 10.0.17
    • 5.5.43
    • Replication
    • rhel

    Description

      On a new 5.5.40 server the replication didn't even start. Databases where created. No replication could start. Identically configured server at same time was created and is replicating ok.

      SHOW STATUS of any portion of it fails to return. Show global variables like (any replication filter variable) fails to return.

      configuration:

      replicate_ignore_table as set to a list of tables. No other replication filters where set.

      results from queries:

      MariaDB [(none)]> show master status;
      +------------------------------+----------+--------------+------------------+
      | File                         | Position | Binlog_Do_DB | Binlog_Ignore_DB |
      +------------------------------+----------+--------------+------------------+
      | tulwaurp009-mysql-bin.000001 |      575 |              |                  |
      +------------------------------+----------+--------------+------------------+
      1 row in set (0.00 sec)
       
      MariaDB [(none)]> show binlog events in 'tulwaurp009-mysql-bin.000001'
          -> ;
      +------------------------------+-----+-------------+-----------+-------------+---------------------------------------------------------------------------+
      | Log_name                     | Pos | Event_type  | Server_id | End_log_pos | Info                                                                      |
      +------------------------------+-----+-------------+-----------+-------------+---------------------------------------------------------------------------+
      | tulwaurp009-mysql-bin.000001 |   4 | Format_desc |        15 |         245 | Server ver: 5.5.40-MariaDB-log, Binlog ver: 4                             |
      | tulwaurp009-mysql-bin.000001 | 245 | Query       |        15 |         369 | use `test`; create table  test ( x int unsigned not null, y varchar(300)) |
      | tulwaurp009-mysql-bin.000001 | 369 | Query       |        15 |         437 | BEGIN                                                                     |
      | tulwaurp009-mysql-bin.000001 | 437 | Query       |        15 |         548 | use `test`; insert into test values (3,'banana'), (4,'pear')              |
      | tulwaurp009-mysql-bin.000001 | 548 | Xid         |        15 |         575 | COMMIT /* xid=66401467 */                                                 |
      +------------------------------+-----+-------------+-----------+-------------+---------------------------------------------------------------------------+

      Above create table and insert was made after the replication was started as a test.

      SHOW PROCESSLIST

      +-------+-------------+---------------------+--------------------+---------+---------+----------------------------------+--------------------------------+----------+
      | Id    | User        | Host                | db                 | Command | Time    | State                            | Info                           | Progress |
      +-------+-------------+---------------------+--------------------+---------+---------+----------------------------------+--------------------------------+----------+
      |     1 | system user |                     | NULL               | Connect | 2171672 | Waiting for master to send event | NULL                           |    0.000 |
      |     2 | system user |                     | NULL               | Connect | 2044491 | closing tables                   | NULL                           |    0.000 |
      |  6122 | mysql       | localhost           | NULL               | Query   | 2044491 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6123 | mysql       | localhost           | NULL               | Killed  | 2044491 | Killing slave                    | STOP SLAVE                     |    0.000 |
      |  6124 | mysql       | 172.16.66.21:49108  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6125 | mysql       | 172.16.66.21:49117  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6126 | mysql       | localhost           | NULL               | Killed  | 2044431 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6127 | mysql       | 172.16.66.21:49124  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6128 | mysql       | localhost           | NULL               | Query   | 2044371 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6129 | mysql       | localhost           | NULL               | Query   | 2044311 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6130 | mysql       | 172.16.66.21:49142  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6131 | mysql       | 172.16.66.21:49189  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6132 | mysql       | localhost           | NULL               | Query   | 2044251 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6133 | mysql       | localhost           | NULL               | Query   | 2044191 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6134 | mysql       | 172.16.66.21:49304  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6135 | mysql       | 172.16.66.21:49311  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6136 | mysql       | localhost           | NULL               | Query   | 2044131 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6137 | mysql       | 172.16.66.21:49318  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6138 | mysql       | localhost           | NULL               | Query   | 2044071 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6139 | mysql       | 172.16.66.21:49328  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6140 | mysql       | localhost           | NULL               | Query   | 2044011 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6141 | mysql       | 172.16.66.21:49385  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6142 | mysql       | localhost           | NULL               | Query   | 2043951 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6143 | mysql       | localhost           | NULL               | Query   | 2043891 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6144 | mysql       | 172.16.66.21:49392  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6145 | mysql       | 172.16.66.21:49401  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6146 | mysql       | localhost           | NULL               | Query   | 2043831 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6147 | mysql       | localhost           | NULL               | Query   | 2043771 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6148 | mysql       | 172.16.66.21:49408  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6149 | mysql       | 172.16.66.21:49418  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6150 | mysql       | localhost           | NULL               | Query   | 2043711 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6151 | mysql       | 172.16.66.21:49473  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6152 | mysql       | localhost           | NULL               | Query   | 2043651 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6153 | mysql       | 172.16.66.21:49480  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6154 | mysql       | localhost           | NULL               | Query   | 2043591 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6155 | mysql       | localhost           | NULL               | Query   | 2043531 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6156 | mysql       | 172.16.66.21:49487  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6157 | mysql       | 172.16.66.21:49602  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6158 | mysql       | localhost           | NULL               | Query   | 2043471 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6159 | mysql       | localhost           | NULL               | Query   | 2043411 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6160 | mysql       | 172.16.66.21:49621  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6161 | mysql       | 172.16.66.21:49669  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6162 | mysql       | localhost           | NULL               | Query   | 2043351 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6163 | mysql       | 172.16.66.21:49676  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6164 | mysql       | localhost           | NULL               | Query   | 2043291 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6165 | mysql       | localhost           | NULL               | Query   | 2043231 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6166 | mysql       | 172.16.66.21:49685  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6167 | mysql       | 172.16.66.21:49692  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6168 | mysql       | localhost           | NULL               | Query   | 2043171 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6169 | mysql       | 172.16.66.21:49707  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6170 | mysql       | localhost           | NULL               | Query   | 2043111 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6171 | mysql       | 172.16.66.21:49757  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6172 | mysql       | localhost           | NULL               | Query   | 2043051 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6173 | mysql       | 172.16.66.21:49764  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6174 | mysql       | localhost           | NULL               | Query   | 2042991 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6175 | mysql       | 172.16.66.21:49771  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6176 | mysql       | localhost           | NULL               | Query   | 2042931 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6177 | mysql       | 172.16.66.21:49778  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6178 | mysql       | localhost           | NULL               | Query   | 2042871 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6179 | mysql       | 172.16.66.21:49793  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6180 | mysql       | localhost           | NULL               | Query   | 2042811 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6181 | mysql       | 172.16.66.21:49845  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6182 | mysql       | localhost           | NULL               | Query   | 2042751 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6183 | mysql       | localhost           | NULL               | Query   | 2042691 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6184 | mysql       | 172.16.66.21:49854  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6185 | mysql       | 172.16.66.21:49969  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6186 | mysql       | localhost           | NULL               | Query   | 2042631 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6187 | mysql       | 172.16.66.21:49976  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6188 | mysql       | localhost           | NULL               | Query   | 2042571 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6189 | mysql       | localhost           | NULL               | Query   | 2042511 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6190 | mysql       | 172.16.66.21:49993  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6191 | mysql       | 172.16.66.21:50042  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6192 | mysql       | localhost           | NULL               | Query   | 2042451 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6193 | mysql       | localhost           | NULL               | Query   | 2042391 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6194 | mysql       | 172.16.66.21:50049  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6195 | mysql       | localhost           | NULL               | Query   | 2042331 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6196 | mysql       | 172.16.66.21:50060  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6197 | mysql       | localhost           | NULL               | Query   | 2042271 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6198 | mysql       | 172.16.66.21:50171  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6199 | mysql       | localhost           | NULL               | Query   | 2042211 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6200 | mysql       | 172.16.66.21:50190  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6201 | mysql       | localhost           | NULL               | Query   | 2042151 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6202 | mysql       | 172.16.66.21:50238  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6203 | mysql       | localhost           | NULL               | Query   | 2042091 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6204 | mysql       | 172.16.66.21:50245  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6205 | mysql       | localhost           | NULL               | Query   | 2042031 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6206 | mysql       | 172.16.66.21:50254  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6207 | mysql       | localhost           | NULL               | Query   | 2041971 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6208 | mysql       | 172.16.66.21:50261  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6209 | mysql       | localhost           | NULL               | Query   | 2041911 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6210 | mysql       | 172.16.66.21:50273  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6211 | mysql       | localhost           | NULL               | Query   | 2041851 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6212 | mysql       | 172.16.66.21:50338  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6213 | mysql       | localhost           | NULL               | Query   | 2041791 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6214 | mysql       | 172.16.66.21:50441  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6215 | mysql       | localhost           | NULL               | Query   | 2041731 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6216 | mysql       | 172.16.66.21:50448  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6217 | mysql       | localhost           | NULL               | Query   | 2041671 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6218 | mysql       | 172.16.66.21:50455  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6219 | mysql       | localhost           | NULL               | Query   | 2041611 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6220 | mysql       | 172.16.66.21:50471  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6221 | mysql       | localhost           | NULL               | Query   | 2041551 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6222 | mysql       | 172.16.66.21:50522  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6223 | mysql       | localhost           | NULL               | Query   | 2041491 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6224 | mysql       | 172.16.66.21:50529  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6225 | mysql       | localhost           | NULL               | Query   | 2041431 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6226 | mysql       | 172.16.66.21:50538  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6227 | mysql       | localhost           | NULL               | Query   | 2041371 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6228 | mysql       | 172.16.66.21:50545  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6229 | mysql       | localhost           | NULL               | Query   | 2041311 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6230 | mysql       | 172.16.66.21:50558  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6231 | mysql       | localhost           | NULL               | Query   | 2041251 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6232 | mysql       | 172.16.66.21:50610  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6233 | mysql       | localhost           | NULL               | Query   | 2041191 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6234 | mysql       | 172.16.66.21:50617  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6235 | mysql       | localhost           | NULL               | Query   | 2041131 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6236 | mysql       | 172.16.66.21:50630  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6237 | mysql       | localhost           | NULL               | Query   | 2041071 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6238 | mysql       | 172.16.66.21:50739  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6239 | mysql       | localhost           | NULL               | Query   | 2041011 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6240 | mysql       | 172.16.66.21:50761  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6241 | mysql       | localhost           | NULL               | Query   | 2040951 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6242 | mysql       | 172.16.66.21:50804  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6243 | mysql       | localhost           | NULL               | Query   | 2040891 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6244 | mysql       | localhost           | NULL               | Query   | 2040891 | Filling schema table             | show status                    |    0.000 |
      |  6245 | mysql       | 172.16.66.21:50811  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6246 | mysql       | localhost           | NULL               | Query   | 2040831 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6247 | mysql       | 172.16.66.21:50823  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6248 | mysql       | localhost           | NULL               | Query   | 2040771 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6249 | mysql       | 172.16.66.21:50830  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6250 | mysql       | localhost           | NULL               | Query   | 2040711 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6251 | mysql       | 172.16.66.21:50845  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6252 | mysql       | localhost           | NULL               | Query   | 2040651 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6253 | mysql       | 172.16.66.21:50913  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6254 | mysql       | 172.16.66.21:51010  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6255 | mysql       | localhost           | NULL               | Query   | 2040591 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6256 | mysql       | localhost           | NULL               | Query   | 2040531 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6257 | mysql       | 172.16.66.21:51017  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6258 | mysql       | localhost           | NULL               | Query   | 2040471 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6259 | mysql       | 172.16.66.21:51024  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6260 | mysql       | 172.16.66.21:51039  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6261 | mysql       | localhost           | NULL               | Query   | 2040411 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6262 | mysql       | localhost           | NULL               | Query   | 2040351 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6263 | mysql       | 172.16.66.21:51089  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6264 | mysql       | 172.16.66.21:51096  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6265 | mysql       | localhost           | NULL               | Query   | 2040291 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6266 | mysql       | localhost           | NULL               | Query   | 2040231 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6267 | mysql       | 172.16.66.21:51105  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      |  6268 | mysql       | localhost           | NULL               | Query   | 2040171 | Filling schema table             | show /*!50002 GLOBAL */ status |    0.000 |
      |  6269 | mysql       | 172.16.66.21:51112  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      | 14881 | mysql       | 172.16.66.21:39304  | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      | 14882 | percona     | 172.17.100.40:49399 | NULL               | Killed  | 1402484 | NULL                             | SHOW SLAVE STATUS              |    0.000 |
      | 14883 | root        | localhost           | NULL               | Killed  |  465087 | NULL                             | show slave status              |    0.000 |
      | 14884 | root        | localhost           | NULL               | Killed  |  465043 | NULL                             | NULL                           |    0.000 |
      | 14885 | root        | localhost           | NULL               | Killed  |  465037 | NULL                             | NULL                           |    0.000 |
      | 14886 | root        | localhost           | NULL               | Killed  |  465028 | NULL                             | NULL                           |    0.000 |
      | 14887 | root        | localhost           | NULL               | Killed  |  464998 | NULL                             | show slave status              |    0.000 |
      | 14888 | root        | localhost           | NULL               | Killed  |  464976 | NULL                             | NULL                           |    0.000 |
      | 14889 | root        | localhost           | NULL               | Killed  |  464971 | NULL                             | NULL                           |    0.000 |
      | 14890 | root        | localhost           | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      | 14891 | root        | localhost           | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      | 14892 | root        | localhost           | NULL               | Killed  |    NULL | login                            | NULL                           |    0.000 |
      | 14893 | root        | localhost           | NULL               | Killed  |  357241 | NULL                             | show slave status              |    0.000 |
      | 14894 | root        | 172.16.66.21:47981  | NULL               | Killed  |    3529 | NULL                             | show slave status              |    0.000 |
      | 14895 | root        | 172.16.66.21:48146  | NULL               | Killed  |    3370 | NULL                             | NULL                           |    0.000 |
      | 14896 | root        | 172.16.66.21:48147  | NULL               | Killed  |    3368 | NULL                             | NULL                           |    0.000 |
      | 14897 | root        | 172.16.66.21:48148  | NULL               | Killed  |    3082 | Filling schema table             | show global variables          |    0.000 |
      | 14898 | root        | 172.16.66.21:48207  | NULL               | Killed  |    3080 | NULL                             | NULL                           |    0.000 |
      | 14899 | root        | 172.16.66.21:48208  | NULL               | Killed  |    3079 | NULL                             | NULL                           |    0.000 |
      | 14900 | root        | 172.16.66.21:48209  | NULL               | Killed  |    3075 | Filling schema table             | show variables                 |    0.000 |
      | 14901 | root        | 172.16.66.21:48319  | NULL               | Killed  |    3029 | NULL                             | NULL                           |    0.000 |
      | 14902 | root        | 172.16.66.21:48320  | NULL               | Killed  |    3028 | NULL                             | NULL                           |    0.000 |
      | 14903 | root        | 172.16.66.21:48321  | NULL               | Killed  |    2948 | Filling schema table             | show status                    |    0.000 |
      | 14904 | root        | 172.16.66.21:48324  | NULL               | Killed  |    2877 | NULL                             | NULL                           |    0.000 |
      | 14905 | root        | 172.16.66.21:48327  | NULL               | Killed  |    2872 | NULL                             | NULL                           |    0.000 |
      | 14906 | root        | 172.16.66.21:48329  | information_schema | Killed  |    2423 | NULL                             | NULL                           |    0.000 |
      | 14907 | root        | 172.16.66.21:48441  | NULL               | Killed  |    1746 | NULL                             | show slave status              |    0.000 |
      | 14908 | root        | 172.16.66.21:48667  | NULL               | Killed  |    1745 | NULL                             | NULL                           |    0.000 |
      | 14909 | root        | 172.16.66.21:48669  | NULL               | Killed  |    1727 | NULL                             | NULL                           |    0.000 |
      | 14910 | root        | 172.16.66.21:48670  | NULL               | Killed  |    1715 | NULL                             | NULL                           |    0.000 |
      | 14911 | root        | 172.16.66.21:48671  | mysql              | Killed  |     199 | NULL                             | NULL                           |    0.000 |
      | 14912 | root        | 172.16.66.21:49289  | NULL               | Killed  |     174 | NULL                             | NULL                           |    0.000 |
      | 14913 | root        | 172.16.66.21:49291  | NULL               | Query   |       0 | NULL                             | show processlist               |    0.000 |

      (several 1000 killed login and duplicate processes in same status removed)

      show databases worked.

      I am able to query databases including mysql database.

      All of these queries returned results:

      MariaDB [information_schema]> select * from KEY_CACHES;
      +----------------+----------+----------------+-----------+------------+-------------+---------------+--------------+---------------+-------+----------------+--------+
      | KEY_CACHE_NAME | SEGMENTS | SEGMENT_NUMBER | FULL_SIZE | BLOCK_SIZE | USED_BLOCKS | UNUSED_BLOCKS | DIRTY_BLOCKS | READ_REQUESTS | READS | WRITE_REQUESTS | WRITES |
      +----------------+----------+----------------+-----------+------------+-------------+---------------+--------------+---------------+-------+----------------+--------+
      | default        |     NULL |           NULL | 209715200 |       1024 |         177 |        166470 |            0 |          7253 |   171 |           1745 |   1734 |
      +----------------+----------+----------------+-----------+------------+-------------+---------------+--------------+---------------+-------+----------------+--------+
      1 row in set (0.00 sec)
       
      MariaDB [information_schema]> select * from VIEWS;
      Empty set (0.03 sec)
       
       
      MariaDB [information_schema]> select @@max_connections;
      +-------------------+
      | @@max_connections |
      +-------------------+
      |              3200 |
      +-------------------+

      MariaDB [information_schema]> show variables like '%conn%';
      Current database: information_schema
       
      +--------------------------+-----------------+
      | Variable_name            | Value           |
      +--------------------------+-----------------+
      | character_set_connection | utf8            |
      | collation_connection     | utf8_general_ci |
      | connect_timeout          | 10              |
      | extra_max_connections    | 1               |
      | init_connect             |                 |
      | max_connect_errors       | 10              |
      | max_connections          | 3200            |
      | max_user_connections     | 0               |
      +--------------------------+-----------------+
      8 rows in set (0.00 sec)
       
      MariaDB [information_schema]> show variables like '%binlog%';
      | binlog_annotate_row_events              | ON                   |
      | binlog_cache_size                       | 32768                |
      | binlog_checksum                         | NONE                 |
      | binlog_direct_non_transactional_updates | OFF                  |
      | binlog_format                           | MIXED                |
      | binlog_optimize_thread_scheduling       | ON                   |
      | binlog_stmt_cache_size                  | 32768                |
      | innodb_locks_unsafe_for_binlog          | OFF                  |
      | max_binlog_cache_size                   | 18446744073709547520 |
      | max_binlog_size                         | 1073741824           |
      | max_binlog_stmt_cache_size              | 18446744073709547520 |
      | sync_binlog                             | 0                    |
       
       
      MariaDB [(none)]> show global variables like 'Aria%';
      ...
      MariaDB [(none)]> show global variables like 'auto%';
      ...
       
      MariaDB [(none)]> show global variables like 'back%';
      +---------------+-------+
      | Variable_name | Value |
      +---------------+-------+
      | back_log      | 50    |
      +---------------+-------+
       
      MariaDB [(none)]> show global variables like 'character%';
      ...
      MariaDB [(none)]> show global variables like 'col%';
      .
       
      MariaDB [(none)]> show global variables like 'con%';
      ...
       
      MariaDB [(none)]> show global variables like 'da%';
       
       
      MariaDB [(none)]> show global variables like 'de%';
       
      show global variables like 'd%';
      ..
      show global variables like 'e%';
      .....
      show global variables like 'q%';
       
      show global variables like 'ra%';
       
      +------------------------+-------+
      | Variable_name          | Value |
      +------------------------+-------+
      | range_alloc_block_size | 4096  |
       
      show global variables like 'read%';
       
      MariaDB [(none)]> show global variables like 'relay%';
      +-----------------------+-----------------------+
      | Variable_name         | Value                 |
      +-----------------------+-----------------------+
      | relay_log             | tulwaurp009-relay-bin |
      | relay_log_index       |                       |
      | relay_log_info_file   | relay-log.info        |
      | relay_log_purge       | ON                    |
      | relay_log_recovery    | OFF                   |
      | relay_log_space_limit | 0                     |
      +-----------------------+-----------------------+
       
      MariaDB [(none)]> show global variables like 'replicate_a%';
       
      +-------------------------------+-------+
      | Variable_name                 | Value |
      +-------------------------------+-------+
      | replicate_annotate_row_events | ON    |
      +-------------------------------+-------+
      1 row in set (0.00 sec)
       
      show global variables like 'report%';
      | Variable_name   | Value       |
      +-----------------+-------------+
      | report_host     | tulwaurp009 |
      | report_password |             |
      | report_port     | 3306        |
      | report_user     |             |
       
      show global variables like 'ro%';
      show global variables like 's%';
      show global variables like 't%';
      ....
      show global variables like 'z%';

      no response:

      • show variables like '%table%';
      • show global variables like 'r%';
      • show global variables like 're%';
      • show global variables like 'repl%';
      • show global variables like 'replicate_d%';
      • show global variables like 'replicate_do_db';
      • show global variables like 'replicate_do_table;
      • show global variables like 'replicate_i%';
      • show global status like 'Aria%';
      • show global status like 'Aria_pagecache%';
      • show global status like 'Aria_trans%';
      • show global status like 'Com_%';
      • show global status like 'Bytes_%';
      • show global status like 'Busy_%';
      • show global status like 'x%';
      • show status
      • show global status
      • show variables
      • show global variables
      • select * from information_schema.session_status;
      • select * from information_schema.session_status limit 3;

      Attachments

        Issue Links

          Activity

            Hi Daniel,

            Technically replication did start, we can see both slave threads in the processlist.
            But apparently the SQL thread hang on "closing tables".
            Possibly it happened on the very first event, I suppose that's what you meant by "replication didn't even start".

            Is this instance still up hanging? If so, can you get a stack trace from the running process?

            If the instance was shutdown, then – is it reproducible? Does it hang again when you start it again?

            Did you check the obvious, like the disk space or IO errors? Closing tables has to do with writing to the disk.

            Could you please also attach the error log from the server startup and up to the time it definitely hang?

            elenst Elena Stepanova added a comment - Hi Daniel, Technically replication did start, we can see both slave threads in the processlist. But apparently the SQL thread hang on "closing tables". Possibly it happened on the very first event, I suppose that's what you meant by "replication didn't even start". Is this instance still up hanging? If so, can you get a stack trace from the running process? If the instance was shutdown, then – is it reproducible? Does it hang again when you start it again? Did you check the obvious, like the disk space or IO errors? Closing tables has to do with writing to the disk. Could you please also attach the error log from the server startup and up to the time it definitely hang?
            danblack Daniel Black added a comment -

            > Is this instance still up hanging?

            yes.

            > If so, can you get a stack trace from the running process?

            I couldn't

            >If the instance was shutdown, then – is it reproducible? Does it hang again when you start it again?

            no.

            >Did you check the obvious, like the disk space or IO errors?

            Space there and no io errors.

            > Could you please also attach the error log from the server startup and up to the time it definitely hang?

            141024 13:07:15 mysqld_safe Starting mysqld daemon with databases from /u01/mariadb
            141024 13:07:15 InnoDB: The InnoDB memory heap is disabled
            141024 13:07:15 InnoDB: Mutexes and rw_locks use GCC atomic builtins
            141024 13:07:15 InnoDB: Compressed tables use zlib 1.2.3
            141024 13:07:15 InnoDB: Using Linux native AIO
            141024 13:07:15 InnoDB: Initializing buffer pool, size = 24.0G
            141024 13:07:17 InnoDB: Completed initialization of buffer pool
            141024 13:07:17 InnoDB: highest supported file format is Barracuda.
            141024 13:07:19 InnoDB: Restoring buffer pool pages from ib_lru_dump
            141024 13:07:19  InnoDB: Waiting for the background threads to start
            141024 13:07:20 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 3338597042050
            141024 13:07:20 [Note] Plugin 'FEEDBACK' is disabled.
            141024 13:07:20 [Note] Server socket created on IP: '0.0.0.0'.
            141024 13:07:20 [Note] Slave SQL thread initialized, starting replication in log 'tulwaurp005-mysql-bin.002166' at position 11843844, relay log './tulwaurp007-relay-bin.005152' positio
            n: 11728905
            141024 13:07:20 [Note] Event Scheduler: Loaded 0 events
            141024 13:07:20 [Note] /usr/sbin/mysqld: ready for connections.
            Version: '5.5.40-MariaDB-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
            141024 13:07:20 [Note] Slave I/O thread: connected to master 'repl@tulwaurp005:3306',replication started in log 'tulwaurp005-mysql-bin.002166' at position 11843844
            141024 13:08:16 InnoDB: Completed reading buffer pool pages (requested: 1568414, read: 1561319)
            141026  0:27:01 [Note] Error reading relay log event: slave SQL thread was killed
            141126 10:51:20 [Note] Slave I/O thread killed while reading event
            141126 10:51:20 [Note] Slave I/O thread exiting, read up to log 'tulwaurp005-mysql-bin.002329', position 762362495
            141126 10:51:20 [Note] /usr/sbin/mysqld: Normal shutdown
             
            141126 10:51:20 [Note] Event Scheduler: Purging the queue. 0 events

            Happy to write this off as MDEV-7026

            danblack Daniel Black added a comment - > Is this instance still up hanging? yes. > If so, can you get a stack trace from the running process? I couldn't >If the instance was shutdown, then – is it reproducible? Does it hang again when you start it again? no. >Did you check the obvious, like the disk space or IO errors? Space there and no io errors. > Could you please also attach the error log from the server startup and up to the time it definitely hang? 141024 13:07:15 mysqld_safe Starting mysqld daemon with databases from /u01/mariadb 141024 13:07:15 InnoDB: The InnoDB memory heap is disabled 141024 13:07:15 InnoDB: Mutexes and rw_locks use GCC atomic builtins 141024 13:07:15 InnoDB: Compressed tables use zlib 1.2.3 141024 13:07:15 InnoDB: Using Linux native AIO 141024 13:07:15 InnoDB: Initializing buffer pool, size = 24.0G 141024 13:07:17 InnoDB: Completed initialization of buffer pool 141024 13:07:17 InnoDB: highest supported file format is Barracuda. 141024 13:07:19 InnoDB: Restoring buffer pool pages from ib_lru_dump 141024 13:07:19 InnoDB: Waiting for the background threads to start 141024 13:07:20 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 3338597042050 141024 13:07:20 [Note] Plugin 'FEEDBACK' is disabled. 141024 13:07:20 [Note] Server socket created on IP: '0.0.0.0'. 141024 13:07:20 [Note] Slave SQL thread initialized, starting replication in log 'tulwaurp005-mysql-bin.002166' at position 11843844, relay log './tulwaurp007-relay-bin.005152' positio n: 11728905 141024 13:07:20 [Note] Event Scheduler: Loaded 0 events 141024 13:07:20 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.40-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server 141024 13:07:20 [Note] Slave I/O thread: connected to master 'repl@tulwaurp005:3306',replication started in log 'tulwaurp005-mysql-bin.002166' at position 11843844 141024 13:08:16 InnoDB: Completed reading buffer pool pages (requested: 1568414, read: 1561319) 141026 0:27:01 [Note] Error reading relay log event: slave SQL thread was killed 141126 10:51:20 [Note] Slave I/O thread killed while reading event 141126 10:51:20 [Note] Slave I/O thread exiting, read up to log 'tulwaurp005-mysql-bin.002329', position 762362495 141126 10:51:20 [Note] /usr/sbin/mysqld: Normal shutdown   141126 10:51:20 [Note] Event Scheduler: Purging the queue. 0 events Happy to write this off as MDEV-7026

            Okay then, I think at this point it's reasonable to blame MDEV-7026, especially since we don't have much else here. Lets wait and see whether a) it happens again (if it does, maybe you can get a stack trace?), and b) if it goes away after we release 5.5.41 and you upgrade to it.
            I will close it for now as a duplicate of MDEV-7026, but please feel free to comment if you have any new information, and we'll re-open if necessary.

            elenst Elena Stepanova added a comment - Okay then, I think at this point it's reasonable to blame MDEV-7026 , especially since we don't have much else here. Lets wait and see whether a) it happens again (if it does, maybe you can get a stack trace?), and b) if it goes away after we release 5.5.41 and you upgrade to it. I will close it for now as a duplicate of MDEV-7026 , but please feel free to comment if you have any new information, and we'll re-open if necessary.

            Hopefully fixed along with MDEV-7026

            elenst Elena Stepanova added a comment - Hopefully fixed along with MDEV-7026
            danblack Daniel Black added a comment -

            This also re-occured on 5.5.39 instance (same machine).

            Id	User	Host	db	Command	Time	State	Info	Progress
            6699	system user		NULL	Connect	3488030	Queueing master event to the relay log	NULL	0.000
            6700	system user		NULL	Connect	2883230	closing tables	NULL	0.000
            26870	mysql	localhost	NULL	Query	2883230	Filling schema table	show /*!50002 GLOBAL */ status	0.000
            26871	mysql	localhost	NULL	Query	2883230	Killing slave	STOP SLAVE	0.000
            26872	mysql	localhost	NULL	Query	2883170	Filling schema table	show /*!50002 GLOBAL */ status	0.000
            26873	mysql	localhost	NULL	Query	2883110	Filling schema table	show /*!50002 GLOBAL */ status	0.000

            The show global status at the same second as the stop slave was consistent.

            2883230 seconds from the STOP SLAVE in process list corresponds to 33 days go on Dec 28.

            This machine was out of disk space however this occurred on 28 January as the SQL IO thread was still running and the relay logs filled the disk up.

            I can't see any fixed to race conditions in replication in the 5.5.40 or 5.5.41 change logs.

            I will try to reproduce with a 5.5.41 instance.

            danblack Daniel Black added a comment - This also re-occured on 5.5.39 instance (same machine). Id User Host db Command Time State Info Progress 6699 system user NULL Connect 3488030 Queueing master event to the relay log NULL 0.000 6700 system user NULL Connect 2883230 closing tables NULL 0.000 26870 mysql localhost NULL Query 2883230 Filling schema table show /*!50002 GLOBAL */ status 0.000 26871 mysql localhost NULL Query 2883230 Killing slave STOP SLAVE 0.000 26872 mysql localhost NULL Query 2883170 Filling schema table show /*!50002 GLOBAL */ status 0.000 26873 mysql localhost NULL Query 2883110 Filling schema table show /*!50002 GLOBAL */ status 0.000 The show global status at the same second as the stop slave was consistent. 2883230 seconds from the STOP SLAVE in process list corresponds to 33 days go on Dec 28. This machine was out of disk space however this occurred on 28 January as the SQL IO thread was still running and the relay logs filled the disk up. I can't see any fixed to race conditions in replication in the 5.5.40 or 5.5.41 change logs. I will try to reproduce with a 5.5.41 instance.
            arjen Arjen Lentz added a comment -

            apparently not yet fixed, as per Daniel Black's latest reports.

            arjen Arjen Lentz added a comment - apparently not yet fixed, as per Daniel Black's latest reports.
            danblack Daniel Black added a comment -

            This is a snapshot from the log file of system stats: tulwaurp009_GenInfo_20141228_0127.log.bz2 at 0127.

            mysql    55574 55569  0 00:27 ?        00:00:00 /bin/sh -c $HOME/bin/mysql_stat.sh
            mysql    55576 55570  0 00:27 ?        00:00:00 /bin/sh -c $HOME/bin/mysql_backup.sh
            mysql    55586 55574  0 00:27 ?        00:00:00 mysqladmin extended-status
            mysql    55588 55576  0 00:27 ?        00:00:00 mysqladmin stop-slave
            mysql    55589 55570  0 00:27 ?        00:00:00 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root
            mysql    55591 55589  0 00:27 ?        00:00:00 /usr/sbin/postdrop -r
            root     55593  3528  0 00:28 ?        00:00:00 CROND
            mysql    55595 55593  0 00:28 ?        00:00:00 /bin/sh -c $HOME/bin/mysql_stat.sh
            mysql    55600 55595  0 00:28 ?        00:00:00 mysqladmin extended-status
            root     55605  3528  0 00:29 ?        00:00:00 CROND
            mysql    55607 55605  0 00:29 ?        00:00:00 /bin/sh -c $HOME/bin/mysql_stat.sh
            mysql    55612 55607  0 00:29 ?        00:00:00 mysqladmin extended-status
            root     55616  3528  0 00:30 ?        00:00:00 CROND
            mysql    55620 55616  0 00:30 ?        00:00:00 /bin/sh -c $HOME/bin/mysql_stat.sh
            mysql    55625 55620  0 00:30 ?        00:00:00 mysqladmin extended-status

            As mysqladmin extended-status/stop-slave don't take an hour to run it points to a deadlock there. Also the minute /mysql_stat.sh scripts that runs the following:

            mysqladmin extended-status > $logfile
             
            #
            # Check that the slave is running
            if test `grep 'Slave_running' $logfile | grep -c 'ON'` -eq 0
            then
              #
              # get some diagnostics
              mysql --batch <<EOT >> $logfile 2>&1
            select sysdate()\\G
            show innodb status\\G
            show processlist\\G
            show slave status\\G
            EOT
            ...

            danblack Daniel Black added a comment - This is a snapshot from the log file of system stats: tulwaurp009_GenInfo_20141228_0127.log.bz2 at 0127. mysql 55574 55569 0 00:27 ? 00:00:00 /bin/sh -c $HOME/bin/mysql_stat.sh mysql 55576 55570 0 00:27 ? 00:00:00 /bin/sh -c $HOME/bin/mysql_backup.sh mysql 55586 55574 0 00:27 ? 00:00:00 mysqladmin extended-status mysql 55588 55576 0 00:27 ? 00:00:00 mysqladmin stop-slave mysql 55589 55570 0 00:27 ? 00:00:00 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root mysql 55591 55589 0 00:27 ? 00:00:00 /usr/sbin/postdrop -r root 55593 3528 0 00:28 ? 00:00:00 CROND mysql 55595 55593 0 00:28 ? 00:00:00 /bin/sh -c $HOME/bin/mysql_stat.sh mysql 55600 55595 0 00:28 ? 00:00:00 mysqladmin extended-status root 55605 3528 0 00:29 ? 00:00:00 CROND mysql 55607 55605 0 00:29 ? 00:00:00 /bin/sh -c $HOME/bin/mysql_stat.sh mysql 55612 55607 0 00:29 ? 00:00:00 mysqladmin extended-status root 55616 3528 0 00:30 ? 00:00:00 CROND mysql 55620 55616 0 00:30 ? 00:00:00 /bin/sh -c $HOME/bin/mysql_stat.sh mysql 55625 55620 0 00:30 ? 00:00:00 mysqladmin extended-status As mysqladmin extended-status/stop-slave don't take an hour to run it points to a deadlock there. Also the minute /mysql_stat.sh scripts that runs the following: mysqladmin extended-status > $logfile   # # Check that the slave is running if test `grep 'Slave_running' $logfile | grep -c 'ON'` -eq 0 then # # get some diagnostics mysql --batch <<EOT >> $logfile 2>&1 select sysdate()\\G show innodb status\\G show processlist\\G show slave status\\G EOT ...

            Was there anything else in the process list apart from the quoted 6 lines and apparently countless other SHOW GLOBAL STATUS if it kept being issued every 60 seconds – any other SQL queries?

            There were a couple deadlocks fixed in 5.5.40, but both involve several types of statements executed at once:
            MDEV-6774 (involves SHOW STATUS as well)
            MDEV-6749 (less chance it's relevant, but still)

            elenst Elena Stepanova added a comment - Was there anything else in the process list apart from the quoted 6 lines and apparently countless other SHOW GLOBAL STATUS if it kept being issued every 60 seconds – any other SQL queries? There were a couple deadlocks fixed in 5.5.40, but both involve several types of statements executed at once: MDEV-6774 (involves SHOW STATUS as well) MDEV-6749 (less chance it's relevant, but still)
            danblack Daniel Black added a comment -

            > any other SQL queries?
            many days later there is the following at the end of the show global status.

            27116	mysql	localhost	NULL	Query	546830	Filling schema table	show status	0.000
            27117	mysql	localhost	NULL	Query	464030	NULL	STOP SLAVE	0.000
            27118	mysql	localhost	NULL	Query	460429	Filling schema table	show status	0.000
            27119	mysql	localhost	NULL	Query	374030	Filling schema table	show status	0.000
            27120	mysql	localhost	NULL	Query	287629	Filling schema table	show status	0.000
            27121	mysql	localhost	NULL	Query	201230	Filling schema table	show status	0.000
            27122	mysql	localhost	NULL	Query	114830	Filling schema table	show status	0.000
            27123	mysql	localhost	NULL	Query	28430	Filling schema table	show status	0.000
            27124	root	localhost	NULL	Killed	NULL	login	NULL	0.000
            27125	munin	localhost	mysql	Killed	428	NULL	NULL	0.000
            27126	munin	localhost	mysql	Query	428	Filling schema table	SHOW GLOBAL STATUS	0.000

            > a couple deadlocks fixed in 5.5.40

            The original bug report before reopening was on 5.5.40. Have reasked client for permission to test .41/.42

            danblack Daniel Black added a comment - > any other SQL queries? many days later there is the following at the end of the show global status. 27116 mysql localhost NULL Query 546830 Filling schema table show status 0.000 27117 mysql localhost NULL Query 464030 NULL STOP SLAVE 0.000 27118 mysql localhost NULL Query 460429 Filling schema table show status 0.000 27119 mysql localhost NULL Query 374030 Filling schema table show status 0.000 27120 mysql localhost NULL Query 287629 Filling schema table show status 0.000 27121 mysql localhost NULL Query 201230 Filling schema table show status 0.000 27122 mysql localhost NULL Query 114830 Filling schema table show status 0.000 27123 mysql localhost NULL Query 28430 Filling schema table show status 0.000 27124 root localhost NULL Killed NULL login NULL 0.000 27125 munin localhost mysql Killed 428 NULL NULL 0.000 27126 munin localhost mysql Query 428 Filling schema table SHOW GLOBAL STATUS 0.000 > a couple deadlocks fixed in 5.5.40 The original bug report before reopening was on 5.5.40. Have reasked client for permission to test .41/.42
            danblack Daniel Black added a comment -

            5.5.41

             sudo cat /var/lib/mysql/mysqld.log
            150210 15:50:32 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
            150210 15:50:34 mysqld_safe Starting mysqld daemon with databases from /u01/mariadb
            150210 15:50:34 InnoDB: The InnoDB memory heap is disabled
            150210 15:50:34 InnoDB: Mutexes and rw_locks use GCC atomic builtins
            150210 15:50:34 InnoDB: Compressed tables use zlib 1.2.3
            150210 15:50:34 InnoDB: Using Linux native AIO
            150210 15:50:34 InnoDB: Initializing buffer pool, size = 24.0G
            150210 15:50:35 InnoDB: Completed initialization of buffer pool
            150210 15:50:35 InnoDB: highest supported file format is Barracuda.
            150210 15:50:35 InnoDB: Restoring buffer pool pages from ib_lru_dump
            150210 15:50:35  InnoDB: Waiting for the background threads to start
            150210 15:50:36 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 4168206481196
            150210 15:50:37 [Note] Plugin 'FEEDBACK' is disabled.
            150210 15:50:37 [Note] Server socket created on IP: '0.0.0.0'.
            150210 15:50:37 [Note] Slave SQL thread initialized, starting replication in log 'tulwaurp005-mysql-bin.002754' at position 635777838, relay log './tulwaurp009-relay-bin.000164' position: 615300928
            150210 15:50:37 [Note] Event Scheduler: Loaded 0 events
            150210 15:50:37 [Note] /usr/sbin/mysqld: ready for connections.
            Version: '5.5.41-MariaDB-log'  socket: '/u01/mariadb/mysql.sock'  port: 3306  MariaDB Server
            150210 15:50:37 [Note] Slave I/O thread: connected to master 'repl@tulwaurp005:3306',replication started in log 'tulwaurp005-mysql-bin.002754' at position 635777838
            150210 15:51:21 InnoDB: Completed reading buffer pool pages (requested: 1464750, read: 1461467)
            150210 15:54:02 [Note] Error reading relay log event: slave SQL thread was killed
            150210 15:54:02 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)
            150210 15:54:02 [Note] Slave I/O thread killed while reading event
            150210 15:54:02 [Note] Slave I/O thread exiting, read up to log 'tulwaurp005-mysql-bin.002754', position 645162051

            crontab -l
            # MDEV-6774 test
            * * * * * mysql -e 'show global status' > /dev/null 2>&1
            * * * * * mysql -e 'stop slave; start slave;'  > /dev/null 2>&1

            immediately got to the state:

            | Id | User        | Host      | db    | Command | Time | State                           | Info               | Progress |
            +----+-------------+-----------+-------+---------+------+---------------------------------+--------------------+----------+
            |  1 | system user |           | NULL  | Connect |  509 | Waiting for slave mutex on exit | NULL               |    0.000 |
            |  8 | openquery   | localhost | NULL  | Query   |  304 | Flushing relay-log info file.   | stop slave         |    0.000 |
            |  9 | openquery   | localhost | NULL  | Query   |  304 | Filling schema table            | show global status |    0.000 |
            | 10 | openquery   | localhost | NULL  | Killed  |  296 | NULL                            | stop slave         |    0.000 |
            | 11 | openquery   | localhost | NULL  | Killed  |  171 | NULL                            | show slave status  |    0.000 |
            | 12 | openquery   | localhost | NULL  | Query   |  245 | Filling schema table            | show global status |    0.000 |
            | 13 | openquery   | localhost | NULL  | Query   |  245 | NULL                            | stop slave         |    0.000 |
            | 14 | munin       | localhost | mysql | Query   |  243 | Filling schema table            | SHOW GLOBAL STATUS |    0.000 |
            | 15 | openquery   | localhost | NULL  | Query   |  185 | Filling schema table            | show global status |    0.000 |
            | 16 | openquery   | localhost | NULL  | Query   |  185 | NULL                            | stop slave         |    0.000 |
            | 17 | munin       | localhost | mysql | Query   |  180 | Filling schema table            | SHOW GLOBAL STATUS |    0.000 |
            | 18 | openquery   | localhost | NULL  | Killed  |  155 | NULL                            | NULL               |    0.000 |
            | 19 | openquery   | localhost | NULL  | Killed  |  153 | NULL                            | NULL               |    0.000 |
            | 20 | openquery   | localhost | NULL  | Killed  |  152 | NULL                            | show slave status  |    0.000 |
            | 21 | openquery   | localhost | NULL  | Killed  |  149 | NULL                            | NULL               |    0.000 |
            | 22 | openquery   | localhost | NULL  | Killed  |  147 | NULL                            | NULL               |    0.000 |
            | 23 | openquery   | localhost | NULL  | Killed  |  106 | NULL                            | show slave status  |    0.000 |
            | 24 | openquery   | localhost | NULL  | Query   |  125 | NULL                            | stop slave         |    0.000 |
            | 25 | openquery   | localhost | NULL  | Query   |  125 | Filling schema table            | show global status |    0.000 |
            | 26 | munin       | localhost | mysql | Query   |  118 | Filling schema table            | SHOW GLOBAL STATUS |    0.000 |
            | 27 | openquery   | localhost | NULL  | Killed  |  110 | NULL                            | NULL               |    0.000 |
            | 28 | openquery   | localhost | NULL  | Killed  |  109 | NULL                            | NULL               |    0.000 |
            | 29 | openquery   | localhost | NULL  | Killed  |  104 | NULL                            | NULL               |    0.000 |
            | 30 | openquery   | localhost | NULL  | Killed  |  103 | NULL                            | NULL               |    0.000 |
            | 31 | openquery   | localhost | NULL  | Killed  |   90 | NULL                            | show slave status  |    0.000 |
            | 32 | openquery   | localhost | NULL  | Killed  |   88 | NULL                            | NULL               |    0.000 |
            | 33 | openquery   | localhost | NULL  | Killed  |   87 | NULL                            | NULL               |    0.000 |
            | 34 | openquery   | localhost | NULL  | Query   |   65 | NULL                            | stop slave         |    0.000 |
            | 35 | openquery   | localhost | NULL  | Query   |   65 | Filling schema table            | show global status |    0.000 |
            | 36 | munin       | localhost | mysql | Query   |   56 | Filling schema table            | SHOW GLOBAL STATUS |    0.000 |
            | 37 | openquery   | localhost | NULL  | Query   |    5 | Filling schema table            | show global status |    0.000 |
            | 38 | openquery   | localhost | NULL  | Query   |    5 | NULL                            | stop slave         |    0.000 |
            | 39 | openquery   | localhost | NULL  | Query   |    0 | NULL                            | show processlist   |    0.000 |

            (gdb) bt full
            #0  0x0000003f8d0df293 in poll () from /lib64/libc.so.6
            No symbol table info available.
            #1  0x0000000000523653 in handle_connections_sockets() ()
            No symbol table info available.
            #2  0x00000000005273a2 in mysqld_main(int, char**) ()
            No symbol table info available.
            #3  0x0000003f8d01ecdd in __libc_start_main () from /lib64/libc.so.6
            No symbol table info available.
            #4  0x000000000051d7e9 in _start ()
            No symbol table info available.
            (gdb) quit

            danblack Daniel Black added a comment - 5.5.41 sudo cat /var/lib/mysql/mysqld.log 150210 15:50:32 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended 150210 15:50:34 mysqld_safe Starting mysqld daemon with databases from /u01/mariadb 150210 15:50:34 InnoDB: The InnoDB memory heap is disabled 150210 15:50:34 InnoDB: Mutexes and rw_locks use GCC atomic builtins 150210 15:50:34 InnoDB: Compressed tables use zlib 1.2.3 150210 15:50:34 InnoDB: Using Linux native AIO 150210 15:50:34 InnoDB: Initializing buffer pool, size = 24.0G 150210 15:50:35 InnoDB: Completed initialization of buffer pool 150210 15:50:35 InnoDB: highest supported file format is Barracuda. 150210 15:50:35 InnoDB: Restoring buffer pool pages from ib_lru_dump 150210 15:50:35 InnoDB: Waiting for the background threads to start 150210 15:50:36 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 4168206481196 150210 15:50:37 [Note] Plugin 'FEEDBACK' is disabled. 150210 15:50:37 [Note] Server socket created on IP: '0.0.0.0'. 150210 15:50:37 [Note] Slave SQL thread initialized, starting replication in log 'tulwaurp005-mysql-bin.002754' at position 635777838, relay log './tulwaurp009-relay-bin.000164' position: 615300928 150210 15:50:37 [Note] Event Scheduler: Loaded 0 events 150210 15:50:37 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.41-MariaDB-log' socket: '/u01/mariadb/mysql.sock' port: 3306 MariaDB Server 150210 15:50:37 [Note] Slave I/O thread: connected to master 'repl@tulwaurp005:3306',replication started in log 'tulwaurp005-mysql-bin.002754' at position 635777838 150210 15:51:21 InnoDB: Completed reading buffer pool pages (requested: 1464750, read: 1461467) 150210 15:54:02 [Note] Error reading relay log event: slave SQL thread was killed 150210 15:54:02 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013) 150210 15:54:02 [Note] Slave I/O thread killed while reading event 150210 15:54:02 [Note] Slave I/O thread exiting, read up to log 'tulwaurp005-mysql-bin.002754', position 645162051 crontab -l # MDEV-6774 test * * * * * mysql -e 'show global status' > /dev/null 2>&1 * * * * * mysql -e 'stop slave; start slave;' > /dev/null 2>&1 immediately got to the state: | Id | User | Host | db | Command | Time | State | Info | Progress | +----+-------------+-----------+-------+---------+------+---------------------------------+--------------------+----------+ | 1 | system user | | NULL | Connect | 509 | Waiting for slave mutex on exit | NULL | 0.000 | | 8 | openquery | localhost | NULL | Query | 304 | Flushing relay-log info file. | stop slave | 0.000 | | 9 | openquery | localhost | NULL | Query | 304 | Filling schema table | show global status | 0.000 | | 10 | openquery | localhost | NULL | Killed | 296 | NULL | stop slave | 0.000 | | 11 | openquery | localhost | NULL | Killed | 171 | NULL | show slave status | 0.000 | | 12 | openquery | localhost | NULL | Query | 245 | Filling schema table | show global status | 0.000 | | 13 | openquery | localhost | NULL | Query | 245 | NULL | stop slave | 0.000 | | 14 | munin | localhost | mysql | Query | 243 | Filling schema table | SHOW GLOBAL STATUS | 0.000 | | 15 | openquery | localhost | NULL | Query | 185 | Filling schema table | show global status | 0.000 | | 16 | openquery | localhost | NULL | Query | 185 | NULL | stop slave | 0.000 | | 17 | munin | localhost | mysql | Query | 180 | Filling schema table | SHOW GLOBAL STATUS | 0.000 | | 18 | openquery | localhost | NULL | Killed | 155 | NULL | NULL | 0.000 | | 19 | openquery | localhost | NULL | Killed | 153 | NULL | NULL | 0.000 | | 20 | openquery | localhost | NULL | Killed | 152 | NULL | show slave status | 0.000 | | 21 | openquery | localhost | NULL | Killed | 149 | NULL | NULL | 0.000 | | 22 | openquery | localhost | NULL | Killed | 147 | NULL | NULL | 0.000 | | 23 | openquery | localhost | NULL | Killed | 106 | NULL | show slave status | 0.000 | | 24 | openquery | localhost | NULL | Query | 125 | NULL | stop slave | 0.000 | | 25 | openquery | localhost | NULL | Query | 125 | Filling schema table | show global status | 0.000 | | 26 | munin | localhost | mysql | Query | 118 | Filling schema table | SHOW GLOBAL STATUS | 0.000 | | 27 | openquery | localhost | NULL | Killed | 110 | NULL | NULL | 0.000 | | 28 | openquery | localhost | NULL | Killed | 109 | NULL | NULL | 0.000 | | 29 | openquery | localhost | NULL | Killed | 104 | NULL | NULL | 0.000 | | 30 | openquery | localhost | NULL | Killed | 103 | NULL | NULL | 0.000 | | 31 | openquery | localhost | NULL | Killed | 90 | NULL | show slave status | 0.000 | | 32 | openquery | localhost | NULL | Killed | 88 | NULL | NULL | 0.000 | | 33 | openquery | localhost | NULL | Killed | 87 | NULL | NULL | 0.000 | | 34 | openquery | localhost | NULL | Query | 65 | NULL | stop slave | 0.000 | | 35 | openquery | localhost | NULL | Query | 65 | Filling schema table | show global status | 0.000 | | 36 | munin | localhost | mysql | Query | 56 | Filling schema table | SHOW GLOBAL STATUS | 0.000 | | 37 | openquery | localhost | NULL | Query | 5 | Filling schema table | show global status | 0.000 | | 38 | openquery | localhost | NULL | Query | 5 | NULL | stop slave | 0.000 | | 39 | openquery | localhost | NULL | Query | 0 | NULL | show processlist | 0.000 | (gdb) bt full #0 0x0000003f8d0df293 in poll () from /lib64/libc.so.6 No symbol table info available. #1 0x0000000000523653 in handle_connections_sockets() () No symbol table info available. #2 0x00000000005273a2 in mysqld_main(int, char**) () No symbol table info available. #3 0x0000003f8d01ecdd in __libc_start_main () from /lib64/libc.so.6 No symbol table info available. #4 0x000000000051d7e9 in _start () No symbol table info available. (gdb) quit
            elenst Elena Stepanova added a comment - - edited

            I can reproduce it on MariaDB 5.5, but not on 10.0 (and not on MySQL 5.5/5.6).

            Test case:

            --source include/master-slave.inc
             
            --connection slave
            --echo # If everything is okay, the test will end in several seconds; maybe a minute. 
            --echo # If the problem shows up, it will hang until testcase timeout is exceeded.
            --exec $MYSQL_SLAP --socket=$SLAVE_MYSOCK -q "START SLAVE; STOP SLAVE; SHOW GLOBAL STATUS" -c 2 --number-of-queries=100 --create-schema=test
            --echo # All done.

            The problem appeared in 5.5.37 with the following revision:

            revno: 4110
            revision-id: monty@mariadb.org-20140314142923-wvho2wb4i8s7wb1v
            parent: svoj@mariadb.org-20140213074049-wo2l3qdtgi0s2mjd
            committer: Michael Widenius <monty@mariadb.org>
            branch nick: maria-5.5
            timestamp: Fri 2014-03-14 16:29:23 +0200
            message:
              MDEV-5829: STOP SLAVE resets global status variables
              
              Reason for the bug was an optimization for higher connect speed where we moved when global status was updated,
              but forgot to update states when slave thread dies.
              Fixed by adding thd->add_status_to_global() before deleting slave thread's thd.

            elenst Elena Stepanova added a comment - - edited I can reproduce it on MariaDB 5.5, but not on 10.0 (and not on MySQL 5.5/5.6). Test case: --source include/master-slave.inc   --connection slave --echo # If everything is okay, the test will end in several seconds; maybe a minute. --echo # If the problem shows up, it will hang until testcase timeout is exceeded. --exec $MYSQL_SLAP --socket=$SLAVE_MYSOCK -q "START SLAVE; STOP SLAVE; SHOW GLOBAL STATUS" -c 2 --number-of-queries=100 --create-schema=test --echo # All done. The problem appeared in 5.5.37 with the following revision: revno: 4110 revision-id: monty@mariadb.org-20140314142923-wvho2wb4i8s7wb1v parent: svoj@mariadb.org-20140213074049-wo2l3qdtgi0s2mjd committer: Michael Widenius <monty@mariadb.org> branch nick: maria-5.5 timestamp: Fri 2014-03-14 16:29:23 +0200 message: MDEV-5829: STOP SLAVE resets global status variables Reason for the bug was an optimization for higher connect speed where we moved when global status was updated, but forgot to update states when slave thread dies. Fixed by adding thd->add_status_to_global() before deleting slave thread's thd.

            danblack, I'm only fixing 5.5 bug here. 10.0 deadlock is different and needs a separate bug report. Could you move it there, please?

            serg Sergei Golubchik added a comment - danblack , I'm only fixing 5.5 bug here. 10.0 deadlock is different and needs a separate bug report. Could you move it there, please?
            danblack Daniel Black added a comment -

            Could you move it there, please?

            ack - MDEV-8039

            danblack Daniel Black added a comment - Could you move it there, please? ack - MDEV-8039
            danblack Daniel Black added a comment -

            re-adding over-culled comment:

            MariaDB [(none)]> show processlist;
            +----+-------------+-----------+------+---------+------+----------------------------------+--------------------+----------+
            | Id | User        | Host      | db   | Command | Time | State                            | Info               | Progress |
            +----+-------------+-----------+------+---------+------+----------------------------------+--------------------+----------+
            |  3 | openquery   | localhost | NULL | Query   |    0 | NULL                             | show processlist   |    0.000 |
            | 38 | system user |           | NULL | Connect |   35 | Waiting for master to send event | NULL               |    0.000 |
            | 39 | system user |           | NULL | Connect |   31 | closing tables                   | NULL               |    0.000 |
            | 40 | openquery   | localhost | NULL | Query   |   31 | Filling schema table             | show global status |    0.000 |
            | 41 | openquery   | localhost | NULL | Query   |   31 | Killing slave                    | stop slave         |    0.000 |
            +----+-------------+-----------+------+---------+------+----------------------------------+--------------------+----------+
            5 rows in set (0.00 sec)

            (gdb) thread apply all bt
             
            Thread 48 (Thread 0x7f2cc83ff700 (LWP 51655)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 47 (Thread 0x7f2642bff700 (LWP 51656)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 46 (Thread 0x7f26421fe700 (LWP 51657)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 45 (Thread 0x7f26417fd700 (LWP 51658)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 44 (Thread 0x7f2640dfc700 (LWP 51659)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 43 (Thread 0x7f26403fb700 (LWP 51660)):
            ---Type <return> to continue, or q <return> to quit---
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 42 (Thread 0x7f263f9fa700 (LWP 51661)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 41 (Thread 0x7f263eff9700 (LWP 51662)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 40 (Thread 0x7f263e5f8700 (LWP 51663)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 39 (Thread 0x7f263dbf7700 (LWP 51664)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 38 (Thread 0x7f263d1f6700 (LWP 51665)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            ---Type <return> to continue, or q <return> to quit---
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 37 (Thread 0x7f263c7f5700 (LWP 51666)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 36 (Thread 0x7f263bdf4700 (LWP 51667)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 35 (Thread 0x7f263b3f3700 (LWP 51668)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 34 (Thread 0x7f263a9f2700 (LWP 51669)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 33 (Thread 0x7f2639ff1700 (LWP 51670)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            ---Type <return> to continue, or q <return> to quit---
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 32 (Thread 0x7f26395f0700 (LWP 51671)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 31 (Thread 0x7f2638bef700 (LWP 51672)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 30 (Thread 0x7f26381ee700 (LWP 51673)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 29 (Thread 0x7f26377ed700 (LWP 51674)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 28 (Thread 0x7f2636dec700 (LWP 51675)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
            ---Type <return> to continue, or q <return> to quit---
             
            Thread 27 (Thread 0x7f26363eb700 (LWP 51676)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 26 (Thread 0x7f26359ea700 (LWP 51677)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 25 (Thread 0x7f2634fe9700 (LWP 51678)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 24 (Thread 0x7f26345e8700 (LWP 51679)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 23 (Thread 0x7f2633be7700 (LWP 51680)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 22 (Thread 0x7f26331e6700 (LWP 51681)):
            ---Type <return> to continue, or q <return> to quit---
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 21 (Thread 0x7f26327e5700 (LWP 51682)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 20 (Thread 0x7f2631de4700 (LWP 51683)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 19 (Thread 0x7f26313e3700 (LWP 51684)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 18 (Thread 0x7f26309e2700 (LWP 51685)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 17 (Thread 0x7f262ffe1700 (LWP 51686)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            ---Type <return> to continue, or q <return> to quit---
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 16 (Thread 0x7f262f5e0700 (LWP 51687)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 15 (Thread 0x7f262ebdf700 (LWP 51688)):
            #0  0x0000000000950d14 in __io_getevents_0_4 ()
            #1  0x0000000000927564 in os_aio_linux_handle ()
            #2  0x00000000008da2d5 in fil_aio_wait ()
            #3  0x00000000008480b8 in io_handler_thread ()
            #4  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #5  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 14 (Thread 0x7f260a5fe700 (LWP 51690)):
            #0  0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
            #1  0x000000000092959b in os_event_wait_time_low ()
            #2  0x0000000000844a70 in srv_lock_timeout_thread ()
            #3  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #4  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 13 (Thread 0x7f260afff700 (LWP 51691)):
            #0  0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
            #1  0x000000000092959b in os_event_wait_time_low ()
            #2  0x0000000000844fc4 in srv_error_monitor_thread ()
            #3  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #4  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 12 (Thread 0x7f2609bfd700 (LWP 51692)):
            #0  0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
            #1  0x000000000092959b in os_event_wait_time_low ()
            #2  0x0000000000844783 in srv_monitor_thread ()
            #3  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #4  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            ---Type <return> to continue, or q <return> to quit---
            Thread 11 (Thread 0x7f26091fc700 (LWP 51693)):
            #0  0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
            #1  0x000000000092959b in os_event_wait_time_low ()
            #2  0x0000000000845373 in srv_LRU_dump_restore_thread ()
            #3  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #4  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 10 (Thread 0x7f26087fb700 (LWP 51694)):
            #0  0x0000003f8d40b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
            #1  0x0000000000929491 in os_event_wait_low ()
            #2  0x0000000000845eee in srv_master_thread ()
            #3  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #4  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 9 (Thread 0x7f2607dfa700 (LWP 51695)):
            #0  0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
            #1  0x000000000092959b in os_event_wait_time_low ()
            #2  0x0000000000847e91 in srv_purge_thread ()
            #3  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #4  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 8 (Thread 0x7f26073f9700 (LWP 51696)):
            #0  0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
            #1  0x00000000009a4587 in my_service_thread_sleep ()
            #2  0x000000000099ce02 in ma_checkpoint_background ()
            #3  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #4  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 7 (Thread 0x7f2cc8d3d700 (LWP 51697)):
            #0  0x0000003f8d40f2a5 in sigwait () from /lib64/libpthread.so.0
            #1  0x000000000051fdff in signal_hand ()
            #2  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #3  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 6 (Thread 0x7f2cc8c62700 (LWP 51721)):
            #0  0x0000003f8d40e54d in read () from /lib64/libpthread.so.0
            #1  0x0000000000aa4a98 in vio_read ()
            #2  0x00000000005281d1 in my_real_read(st_net*, unsigned long*) ()
            #3  0x0000000000529099 in my_net_read ()
            #4  0x00000000005a4df2 in do_command(THD*) ()
            #5  0x000000000065512b in do_handle_one_connection(THD*) ()
            #6  0x00000000006551ac in handle_one_connection ()
            ---Type <return> to continue, or q <return> to quit---
            #7  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #8  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 5 (Thread 0x7f2cc79fe700 (LWP 51738)):
            #0  0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
            #1  0x0000000000535532 in terminate_slave_thread(THD*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, bool) ()
            #2  0x000000000053d0ac in terminate_slave_threads ()
            #3  0x00000000005b691d in stop_slave(THD*, Master_info*, bool) ()
            #4  0x000000000059eebe in mysql_execute_command(THD*) ()
            #5  0x00000000005a2854 in mysql_parse ()
            #6  0x00000000005a4d08 in dispatch_command(enum_server_command, THD*, char*, unsigned int) ()
            #7  0x000000000065512b in do_handle_one_connection(THD*) ()
            #8  0x00000000006551ac in handle_one_connection ()
            #9  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #10 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 4 (Thread 0x7f2cc79b5700 (LWP 51739)):
            #0  0x0000003f8d40e054 in __lll_lock_wait () from /lib64/libpthread.so.0
            #1  0x0000003f8d4093be in _L_lock_995 () from /lib64/libpthread.so.0
            #2  0x0000003f8d409326 in pthread_mutex_lock () from /lib64/libpthread.so.0
            #3  0x000000000051f6b8 in inline_mysql_mutex_lock.constprop.126 ()
            #4  0x000000000052024c in show_heartbeat_period(THD*, st_mysql_show_var*, char*) ()
            #5  0x00000000005f38a4 in show_status_array(THD*, char const*, st_mysql_show_var*, enum_var_type, system_status_var*, char const*, TABLE*, bool, Item*) ()
            #6  0x00000000005fe618 in fill_status(THD*, TABLE_LIST*, Item*) ()
            #7  0x0000000000602d67 in get_schema_tables_result(JOIN*, enum_schema_table_state) ()
            #8  0x00000000005e9c8d in JOIN::exec() ()
            #9  0x00000000005e5525 in mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) ()
            #10 0x00000000005ec113 in handle_select(THD*, LEX*, select_result*, unsigned long) ()
            #11 0x00000000005970a6 in execute_sqlcom_select(THD*, TABLE_LIST*) ()
            #12 0x000000000059e87f in mysql_execute_command(THD*) ()
            #13 0x00000000005a2854 in mysql_parse ()
            #14 0x00000000005a4d08 in dispatch_command(enum_server_command, THD*, char*, unsigned int) ()
            #15 0x000000000065512b in do_handle_one_connection(THD*) ()
            #16 0x00000000006551ac in handle_one_connection ()
            #17 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #18 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 3 (Thread 0x7f2cc8cf4700 (LWP 51804)):
            #0  0x0000003f8d40e54d in read () from /lib64/libpthread.so.0
            #1  0x0000000000aa4a98 in vio_read ()
            #2  0x0000000000aa4b32 in vio_read_buff ()
            ---Type <return> to continue, or q <return> to quit---
            #3  0x00000000005281d1 in my_real_read(st_net*, unsigned long*) ()
            #4  0x0000000000529035 in my_net_read ()
            #5  0x00000000006a9365 in cli_safe_read ()
            #6  0x000000000053db43 in handle_slave_io ()
            #7  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #8  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 2 (Thread 0x7f2cc8cab700 (LWP 51805)):
            #0  0x0000003f8d40e054 in __lll_lock_wait () from /lib64/libpthread.so.0
            #1  0x0000003f8d4093be in _L_lock_995 () from /lib64/libpthread.so.0
            #2  0x0000003f8d409326 in pthread_mutex_lock () from /lib64/libpthread.so.0
            #3  0x0000000000534c50 in inline_mysql_mutex_lock ()
            #4  0x000000000053ee10 in THD::add_status_to_global() ()
            #5  0x000000000053c34b in handle_slave_sql ()
            #6  0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
            #7  0x0000003f8d0e894d in clone () from /lib64/libc.so.6
             
            Thread 1 (Thread 0x7f2cc8da2800 (LWP 51653)):
            #0  0x0000003f8d0df293 in poll () from /lib64/libc.so.6
            #1  0x0000000000523653 in handle_connections_sockets() ()
            #2  0x00000000005273a2 in mysqld_main(int, char**) ()
            #3  0x0000003f8d01ecdd in __libc_start_main () from /lib64/libc.so.6
            #4  0x000000000051d7e9 in _start ()

            )

            danblack Daniel Black added a comment - re-adding over-culled comment: MariaDB [(none)]> show processlist; +----+-------------+-----------+------+---------+------+----------------------------------+--------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +----+-------------+-----------+------+---------+------+----------------------------------+--------------------+----------+ | 3 | openquery | localhost | NULL | Query | 0 | NULL | show processlist | 0.000 | | 38 | system user | | NULL | Connect | 35 | Waiting for master to send event | NULL | 0.000 | | 39 | system user | | NULL | Connect | 31 | closing tables | NULL | 0.000 | | 40 | openquery | localhost | NULL | Query | 31 | Filling schema table | show global status | 0.000 | | 41 | openquery | localhost | NULL | Query | 31 | Killing slave | stop slave | 0.000 | +----+-------------+-----------+------+---------+------+----------------------------------+--------------------+----------+ 5 rows in set (0.00 sec) (gdb) thread apply all bt   Thread 48 (Thread 0x7f2cc83ff700 (LWP 51655)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 47 (Thread 0x7f2642bff700 (LWP 51656)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 46 (Thread 0x7f26421fe700 (LWP 51657)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 45 (Thread 0x7f26417fd700 (LWP 51658)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 44 (Thread 0x7f2640dfc700 (LWP 51659)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 43 (Thread 0x7f26403fb700 (LWP 51660)): ---Type <return> to continue, or q <return> to quit--- #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 42 (Thread 0x7f263f9fa700 (LWP 51661)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 41 (Thread 0x7f263eff9700 (LWP 51662)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 40 (Thread 0x7f263e5f8700 (LWP 51663)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 39 (Thread 0x7f263dbf7700 (LWP 51664)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 38 (Thread 0x7f263d1f6700 (LWP 51665)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () ---Type <return> to continue, or q <return> to quit--- #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 37 (Thread 0x7f263c7f5700 (LWP 51666)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 36 (Thread 0x7f263bdf4700 (LWP 51667)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 35 (Thread 0x7f263b3f3700 (LWP 51668)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 34 (Thread 0x7f263a9f2700 (LWP 51669)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 33 (Thread 0x7f2639ff1700 (LWP 51670)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () ---Type <return> to continue, or q <return> to quit--- #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 32 (Thread 0x7f26395f0700 (LWP 51671)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 31 (Thread 0x7f2638bef700 (LWP 51672)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 30 (Thread 0x7f26381ee700 (LWP 51673)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 29 (Thread 0x7f26377ed700 (LWP 51674)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 28 (Thread 0x7f2636dec700 (LWP 51675)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit---   Thread 27 (Thread 0x7f26363eb700 (LWP 51676)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 26 (Thread 0x7f26359ea700 (LWP 51677)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 25 (Thread 0x7f2634fe9700 (LWP 51678)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 24 (Thread 0x7f26345e8700 (LWP 51679)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 23 (Thread 0x7f2633be7700 (LWP 51680)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 22 (Thread 0x7f26331e6700 (LWP 51681)): ---Type <return> to continue, or q <return> to quit--- #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 21 (Thread 0x7f26327e5700 (LWP 51682)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 20 (Thread 0x7f2631de4700 (LWP 51683)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 19 (Thread 0x7f26313e3700 (LWP 51684)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 18 (Thread 0x7f26309e2700 (LWP 51685)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 17 (Thread 0x7f262ffe1700 (LWP 51686)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () ---Type <return> to continue, or q <return> to quit--- #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 16 (Thread 0x7f262f5e0700 (LWP 51687)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 15 (Thread 0x7f262ebdf700 (LWP 51688)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 14 (Thread 0x7f260a5fe700 (LWP 51690)): #0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000092959b in os_event_wait_time_low () #2 0x0000000000844a70 in srv_lock_timeout_thread () #3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 13 (Thread 0x7f260afff700 (LWP 51691)): #0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000092959b in os_event_wait_time_low () #2 0x0000000000844fc4 in srv_error_monitor_thread () #3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 12 (Thread 0x7f2609bfd700 (LWP 51692)): #0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000092959b in os_event_wait_time_low () #2 0x0000000000844783 in srv_monitor_thread () #3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   ---Type <return> to continue, or q <return> to quit--- Thread 11 (Thread 0x7f26091fc700 (LWP 51693)): #0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000092959b in os_event_wait_time_low () #2 0x0000000000845373 in srv_LRU_dump_restore_thread () #3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 10 (Thread 0x7f26087fb700 (LWP 51694)): #0 0x0000003f8d40b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000929491 in os_event_wait_low () #2 0x0000000000845eee in srv_master_thread () #3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 9 (Thread 0x7f2607dfa700 (LWP 51695)): #0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000092959b in os_event_wait_time_low () #2 0x0000000000847e91 in srv_purge_thread () #3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 8 (Thread 0x7f26073f9700 (LWP 51696)): #0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000009a4587 in my_service_thread_sleep () #2 0x000000000099ce02 in ma_checkpoint_background () #3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 7 (Thread 0x7f2cc8d3d700 (LWP 51697)): #0 0x0000003f8d40f2a5 in sigwait () from /lib64/libpthread.so.0 #1 0x000000000051fdff in signal_hand () #2 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #3 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 6 (Thread 0x7f2cc8c62700 (LWP 51721)): #0 0x0000003f8d40e54d in read () from /lib64/libpthread.so.0 #1 0x0000000000aa4a98 in vio_read () #2 0x00000000005281d1 in my_real_read(st_net*, unsigned long*) () #3 0x0000000000529099 in my_net_read () #4 0x00000000005a4df2 in do_command(THD*) () #5 0x000000000065512b in do_handle_one_connection(THD*) () #6 0x00000000006551ac in handle_one_connection () ---Type <return> to continue, or q <return> to quit--- #7 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #8 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 5 (Thread 0x7f2cc79fe700 (LWP 51738)): #0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000535532 in terminate_slave_thread(THD*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, bool) () #2 0x000000000053d0ac in terminate_slave_threads () #3 0x00000000005b691d in stop_slave(THD*, Master_info*, bool) () #4 0x000000000059eebe in mysql_execute_command(THD*) () #5 0x00000000005a2854 in mysql_parse () #6 0x00000000005a4d08 in dispatch_command(enum_server_command, THD*, char*, unsigned int) () #7 0x000000000065512b in do_handle_one_connection(THD*) () #8 0x00000000006551ac in handle_one_connection () #9 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #10 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 4 (Thread 0x7f2cc79b5700 (LWP 51739)): #0 0x0000003f8d40e054 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x0000003f8d4093be in _L_lock_995 () from /lib64/libpthread.so.0 #2 0x0000003f8d409326 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x000000000051f6b8 in inline_mysql_mutex_lock.constprop.126 () #4 0x000000000052024c in show_heartbeat_period(THD*, st_mysql_show_var*, char*) () #5 0x00000000005f38a4 in show_status_array(THD*, char const*, st_mysql_show_var*, enum_var_type, system_status_var*, char const*, TABLE*, bool, Item*) () #6 0x00000000005fe618 in fill_status(THD*, TABLE_LIST*, Item*) () #7 0x0000000000602d67 in get_schema_tables_result(JOIN*, enum_schema_table_state) () #8 0x00000000005e9c8d in JOIN::exec() () #9 0x00000000005e5525 in mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) () #10 0x00000000005ec113 in handle_select(THD*, LEX*, select_result*, unsigned long) () #11 0x00000000005970a6 in execute_sqlcom_select(THD*, TABLE_LIST*) () #12 0x000000000059e87f in mysql_execute_command(THD*) () #13 0x00000000005a2854 in mysql_parse () #14 0x00000000005a4d08 in dispatch_command(enum_server_command, THD*, char*, unsigned int) () #15 0x000000000065512b in do_handle_one_connection(THD*) () #16 0x00000000006551ac in handle_one_connection () #17 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #18 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 3 (Thread 0x7f2cc8cf4700 (LWP 51804)): #0 0x0000003f8d40e54d in read () from /lib64/libpthread.so.0 #1 0x0000000000aa4a98 in vio_read () #2 0x0000000000aa4b32 in vio_read_buff () ---Type <return> to continue, or q <return> to quit--- #3 0x00000000005281d1 in my_real_read(st_net*, unsigned long*) () #4 0x0000000000529035 in my_net_read () #5 0x00000000006a9365 in cli_safe_read () #6 0x000000000053db43 in handle_slave_io () #7 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #8 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 2 (Thread 0x7f2cc8cab700 (LWP 51805)): #0 0x0000003f8d40e054 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x0000003f8d4093be in _L_lock_995 () from /lib64/libpthread.so.0 #2 0x0000003f8d409326 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000534c50 in inline_mysql_mutex_lock () #4 0x000000000053ee10 in THD::add_status_to_global() () #5 0x000000000053c34b in handle_slave_sql () #6 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003f8d0e894d in clone () from /lib64/libc.so.6   Thread 1 (Thread 0x7f2cc8da2800 (LWP 51653)): #0 0x0000003f8d0df293 in poll () from /lib64/libc.so.6 #1 0x0000000000523653 in handle_connections_sockets() () #2 0x00000000005273a2 in mysqld_main(int, char**) () #3 0x0000003f8d01ecdd in __libc_start_main () from /lib64/libc.so.6 #4 0x000000000051d7e9 in _start () )

            People

              serg Sergei Golubchik
              danblack Daniel Black
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.