Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
5.5.37, 10.0.17
-
rhel
Description
On a new 5.5.40 server the replication didn't even start. Databases where created. No replication could start. Identically configured server at same time was created and is replicating ok.
SHOW STATUS of any portion of it fails to return. Show global variables like (any replication filter variable) fails to return.
configuration:
replicate_ignore_table as set to a list of tables. No other replication filters where set.
results from queries:
MariaDB [(none)]> show master status;
|
+------------------------------+----------+--------------+------------------+
|
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
|
+------------------------------+----------+--------------+------------------+
|
| tulwaurp009-mysql-bin.000001 | 575 | | |
|
+------------------------------+----------+--------------+------------------+
|
1 row in set (0.00 sec)
|
|
|
MariaDB [(none)]> show binlog events in 'tulwaurp009-mysql-bin.000001'
|
-> ;
|
+------------------------------+-----+-------------+-----------+-------------+---------------------------------------------------------------------------+
|
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |
|
+------------------------------+-----+-------------+-----------+-------------+---------------------------------------------------------------------------+
|
| tulwaurp009-mysql-bin.000001 | 4 | Format_desc | 15 | 245 | Server ver: 5.5.40-MariaDB-log, Binlog ver: 4 |
|
| tulwaurp009-mysql-bin.000001 | 245 | Query | 15 | 369 | use `test`; create table test ( x int unsigned not null, y varchar(300)) |
|
| tulwaurp009-mysql-bin.000001 | 369 | Query | 15 | 437 | BEGIN |
|
| tulwaurp009-mysql-bin.000001 | 437 | Query | 15 | 548 | use `test`; insert into test values (3,'banana'), (4,'pear') |
|
| tulwaurp009-mysql-bin.000001 | 548 | Xid | 15 | 575 | COMMIT /* xid=66401467 */ |
|
+------------------------------+-----+-------------+-----------+-------------+---------------------------------------------------------------------------+
|
Above create table and insert was made after the replication was started as a test.
SHOW PROCESSLIST
+-------+-------------+---------------------+--------------------+---------+---------+----------------------------------+--------------------------------+----------+
|
| Id | User | Host | db | Command | Time | State | Info | Progress |
|
+-------+-------------+---------------------+--------------------+---------+---------+----------------------------------+--------------------------------+----------+
|
| 1 | system user | | NULL | Connect | 2171672 | Waiting for master to send event | NULL | 0.000 |
|
| 2 | system user | | NULL | Connect | 2044491 | closing tables | NULL | 0.000 |
|
| 6122 | mysql | localhost | NULL | Query | 2044491 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6123 | mysql | localhost | NULL | Killed | 2044491 | Killing slave | STOP SLAVE | 0.000 |
|
| 6124 | mysql | 172.16.66.21:49108 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6125 | mysql | 172.16.66.21:49117 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6126 | mysql | localhost | NULL | Killed | 2044431 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6127 | mysql | 172.16.66.21:49124 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6128 | mysql | localhost | NULL | Query | 2044371 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6129 | mysql | localhost | NULL | Query | 2044311 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6130 | mysql | 172.16.66.21:49142 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6131 | mysql | 172.16.66.21:49189 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6132 | mysql | localhost | NULL | Query | 2044251 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6133 | mysql | localhost | NULL | Query | 2044191 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6134 | mysql | 172.16.66.21:49304 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6135 | mysql | 172.16.66.21:49311 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6136 | mysql | localhost | NULL | Query | 2044131 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6137 | mysql | 172.16.66.21:49318 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6138 | mysql | localhost | NULL | Query | 2044071 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6139 | mysql | 172.16.66.21:49328 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6140 | mysql | localhost | NULL | Query | 2044011 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6141 | mysql | 172.16.66.21:49385 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6142 | mysql | localhost | NULL | Query | 2043951 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6143 | mysql | localhost | NULL | Query | 2043891 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6144 | mysql | 172.16.66.21:49392 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6145 | mysql | 172.16.66.21:49401 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6146 | mysql | localhost | NULL | Query | 2043831 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6147 | mysql | localhost | NULL | Query | 2043771 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6148 | mysql | 172.16.66.21:49408 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6149 | mysql | 172.16.66.21:49418 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6150 | mysql | localhost | NULL | Query | 2043711 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6151 | mysql | 172.16.66.21:49473 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6152 | mysql | localhost | NULL | Query | 2043651 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6153 | mysql | 172.16.66.21:49480 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6154 | mysql | localhost | NULL | Query | 2043591 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6155 | mysql | localhost | NULL | Query | 2043531 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6156 | mysql | 172.16.66.21:49487 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6157 | mysql | 172.16.66.21:49602 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6158 | mysql | localhost | NULL | Query | 2043471 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6159 | mysql | localhost | NULL | Query | 2043411 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6160 | mysql | 172.16.66.21:49621 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6161 | mysql | 172.16.66.21:49669 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6162 | mysql | localhost | NULL | Query | 2043351 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6163 | mysql | 172.16.66.21:49676 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6164 | mysql | localhost | NULL | Query | 2043291 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6165 | mysql | localhost | NULL | Query | 2043231 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6166 | mysql | 172.16.66.21:49685 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6167 | mysql | 172.16.66.21:49692 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6168 | mysql | localhost | NULL | Query | 2043171 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6169 | mysql | 172.16.66.21:49707 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6170 | mysql | localhost | NULL | Query | 2043111 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6171 | mysql | 172.16.66.21:49757 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6172 | mysql | localhost | NULL | Query | 2043051 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6173 | mysql | 172.16.66.21:49764 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6174 | mysql | localhost | NULL | Query | 2042991 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6175 | mysql | 172.16.66.21:49771 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6176 | mysql | localhost | NULL | Query | 2042931 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6177 | mysql | 172.16.66.21:49778 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6178 | mysql | localhost | NULL | Query | 2042871 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6179 | mysql | 172.16.66.21:49793 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6180 | mysql | localhost | NULL | Query | 2042811 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6181 | mysql | 172.16.66.21:49845 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6182 | mysql | localhost | NULL | Query | 2042751 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6183 | mysql | localhost | NULL | Query | 2042691 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6184 | mysql | 172.16.66.21:49854 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6185 | mysql | 172.16.66.21:49969 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6186 | mysql | localhost | NULL | Query | 2042631 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6187 | mysql | 172.16.66.21:49976 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6188 | mysql | localhost | NULL | Query | 2042571 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6189 | mysql | localhost | NULL | Query | 2042511 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6190 | mysql | 172.16.66.21:49993 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6191 | mysql | 172.16.66.21:50042 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6192 | mysql | localhost | NULL | Query | 2042451 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6193 | mysql | localhost | NULL | Query | 2042391 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6194 | mysql | 172.16.66.21:50049 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6195 | mysql | localhost | NULL | Query | 2042331 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6196 | mysql | 172.16.66.21:50060 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6197 | mysql | localhost | NULL | Query | 2042271 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6198 | mysql | 172.16.66.21:50171 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6199 | mysql | localhost | NULL | Query | 2042211 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6200 | mysql | 172.16.66.21:50190 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6201 | mysql | localhost | NULL | Query | 2042151 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6202 | mysql | 172.16.66.21:50238 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6203 | mysql | localhost | NULL | Query | 2042091 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6204 | mysql | 172.16.66.21:50245 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6205 | mysql | localhost | NULL | Query | 2042031 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6206 | mysql | 172.16.66.21:50254 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6207 | mysql | localhost | NULL | Query | 2041971 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6208 | mysql | 172.16.66.21:50261 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6209 | mysql | localhost | NULL | Query | 2041911 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6210 | mysql | 172.16.66.21:50273 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6211 | mysql | localhost | NULL | Query | 2041851 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6212 | mysql | 172.16.66.21:50338 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6213 | mysql | localhost | NULL | Query | 2041791 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6214 | mysql | 172.16.66.21:50441 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6215 | mysql | localhost | NULL | Query | 2041731 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6216 | mysql | 172.16.66.21:50448 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6217 | mysql | localhost | NULL | Query | 2041671 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6218 | mysql | 172.16.66.21:50455 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6219 | mysql | localhost | NULL | Query | 2041611 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6220 | mysql | 172.16.66.21:50471 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6221 | mysql | localhost | NULL | Query | 2041551 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6222 | mysql | 172.16.66.21:50522 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6223 | mysql | localhost | NULL | Query | 2041491 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6224 | mysql | 172.16.66.21:50529 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6225 | mysql | localhost | NULL | Query | 2041431 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6226 | mysql | 172.16.66.21:50538 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6227 | mysql | localhost | NULL | Query | 2041371 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6228 | mysql | 172.16.66.21:50545 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6229 | mysql | localhost | NULL | Query | 2041311 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6230 | mysql | 172.16.66.21:50558 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6231 | mysql | localhost | NULL | Query | 2041251 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6232 | mysql | 172.16.66.21:50610 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6233 | mysql | localhost | NULL | Query | 2041191 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6234 | mysql | 172.16.66.21:50617 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6235 | mysql | localhost | NULL | Query | 2041131 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6236 | mysql | 172.16.66.21:50630 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6237 | mysql | localhost | NULL | Query | 2041071 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6238 | mysql | 172.16.66.21:50739 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6239 | mysql | localhost | NULL | Query | 2041011 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6240 | mysql | 172.16.66.21:50761 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6241 | mysql | localhost | NULL | Query | 2040951 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6242 | mysql | 172.16.66.21:50804 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6243 | mysql | localhost | NULL | Query | 2040891 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6244 | mysql | localhost | NULL | Query | 2040891 | Filling schema table | show status | 0.000 |
|
| 6245 | mysql | 172.16.66.21:50811 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6246 | mysql | localhost | NULL | Query | 2040831 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6247 | mysql | 172.16.66.21:50823 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6248 | mysql | localhost | NULL | Query | 2040771 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6249 | mysql | 172.16.66.21:50830 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6250 | mysql | localhost | NULL | Query | 2040711 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6251 | mysql | 172.16.66.21:50845 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6252 | mysql | localhost | NULL | Query | 2040651 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6253 | mysql | 172.16.66.21:50913 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6254 | mysql | 172.16.66.21:51010 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6255 | mysql | localhost | NULL | Query | 2040591 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6256 | mysql | localhost | NULL | Query | 2040531 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6257 | mysql | 172.16.66.21:51017 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6258 | mysql | localhost | NULL | Query | 2040471 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6259 | mysql | 172.16.66.21:51024 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6260 | mysql | 172.16.66.21:51039 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6261 | mysql | localhost | NULL | Query | 2040411 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6262 | mysql | localhost | NULL | Query | 2040351 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6263 | mysql | 172.16.66.21:51089 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6264 | mysql | 172.16.66.21:51096 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6265 | mysql | localhost | NULL | Query | 2040291 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6266 | mysql | localhost | NULL | Query | 2040231 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6267 | mysql | 172.16.66.21:51105 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 6268 | mysql | localhost | NULL | Query | 2040171 | Filling schema table | show /*!50002 GLOBAL */ status | 0.000 |
|
| 6269 | mysql | 172.16.66.21:51112 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 14881 | mysql | 172.16.66.21:39304 | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 14882 | percona | 172.17.100.40:49399 | NULL | Killed | 1402484 | NULL | SHOW SLAVE STATUS | 0.000 |
|
| 14883 | root | localhost | NULL | Killed | 465087 | NULL | show slave status | 0.000 |
|
| 14884 | root | localhost | NULL | Killed | 465043 | NULL | NULL | 0.000 |
|
| 14885 | root | localhost | NULL | Killed | 465037 | NULL | NULL | 0.000 |
|
| 14886 | root | localhost | NULL | Killed | 465028 | NULL | NULL | 0.000 |
|
| 14887 | root | localhost | NULL | Killed | 464998 | NULL | show slave status | 0.000 |
|
| 14888 | root | localhost | NULL | Killed | 464976 | NULL | NULL | 0.000 |
|
| 14889 | root | localhost | NULL | Killed | 464971 | NULL | NULL | 0.000 |
|
| 14890 | root | localhost | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 14891 | root | localhost | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 14892 | root | localhost | NULL | Killed | NULL | login | NULL | 0.000 |
|
| 14893 | root | localhost | NULL | Killed | 357241 | NULL | show slave status | 0.000 |
|
| 14894 | root | 172.16.66.21:47981 | NULL | Killed | 3529 | NULL | show slave status | 0.000 |
|
| 14895 | root | 172.16.66.21:48146 | NULL | Killed | 3370 | NULL | NULL | 0.000 |
|
| 14896 | root | 172.16.66.21:48147 | NULL | Killed | 3368 | NULL | NULL | 0.000 |
|
| 14897 | root | 172.16.66.21:48148 | NULL | Killed | 3082 | Filling schema table | show global variables | 0.000 |
|
| 14898 | root | 172.16.66.21:48207 | NULL | Killed | 3080 | NULL | NULL | 0.000 |
|
| 14899 | root | 172.16.66.21:48208 | NULL | Killed | 3079 | NULL | NULL | 0.000 |
|
| 14900 | root | 172.16.66.21:48209 | NULL | Killed | 3075 | Filling schema table | show variables | 0.000 |
|
| 14901 | root | 172.16.66.21:48319 | NULL | Killed | 3029 | NULL | NULL | 0.000 |
|
| 14902 | root | 172.16.66.21:48320 | NULL | Killed | 3028 | NULL | NULL | 0.000 |
|
| 14903 | root | 172.16.66.21:48321 | NULL | Killed | 2948 | Filling schema table | show status | 0.000 |
|
| 14904 | root | 172.16.66.21:48324 | NULL | Killed | 2877 | NULL | NULL | 0.000 |
|
| 14905 | root | 172.16.66.21:48327 | NULL | Killed | 2872 | NULL | NULL | 0.000 |
|
| 14906 | root | 172.16.66.21:48329 | information_schema | Killed | 2423 | NULL | NULL | 0.000 |
|
| 14907 | root | 172.16.66.21:48441 | NULL | Killed | 1746 | NULL | show slave status | 0.000 |
|
| 14908 | root | 172.16.66.21:48667 | NULL | Killed | 1745 | NULL | NULL | 0.000 |
|
| 14909 | root | 172.16.66.21:48669 | NULL | Killed | 1727 | NULL | NULL | 0.000 |
|
| 14910 | root | 172.16.66.21:48670 | NULL | Killed | 1715 | NULL | NULL | 0.000 |
|
| 14911 | root | 172.16.66.21:48671 | mysql | Killed | 199 | NULL | NULL | 0.000 |
|
| 14912 | root | 172.16.66.21:49289 | NULL | Killed | 174 | NULL | NULL | 0.000 |
|
| 14913 | root | 172.16.66.21:49291 | NULL | Query | 0 | NULL | show processlist | 0.000 |
|
(several 1000 killed login and duplicate processes in same status removed)
show databases worked.
I am able to query databases including mysql database.
All of these queries returned results:
MariaDB [information_schema]> select * from KEY_CACHES;
|
+----------------+----------+----------------+-----------+------------+-------------+---------------+--------------+---------------+-------+----------------+--------+
|
| KEY_CACHE_NAME | SEGMENTS | SEGMENT_NUMBER | FULL_SIZE | BLOCK_SIZE | USED_BLOCKS | UNUSED_BLOCKS | DIRTY_BLOCKS | READ_REQUESTS | READS | WRITE_REQUESTS | WRITES |
|
+----------------+----------+----------------+-----------+------------+-------------+---------------+--------------+---------------+-------+----------------+--------+
|
| default | NULL | NULL | 209715200 | 1024 | 177 | 166470 | 0 | 7253 | 171 | 1745 | 1734 |
|
+----------------+----------+----------------+-----------+------------+-------------+---------------+--------------+---------------+-------+----------------+--------+
|
1 row in set (0.00 sec)
|
|
|
MariaDB [information_schema]> select * from VIEWS;
|
Empty set (0.03 sec)
|
|
|
|
|
MariaDB [information_schema]> select @@max_connections;
|
+-------------------+
|
| @@max_connections |
|
+-------------------+
|
| 3200 |
|
+-------------------+
|
MariaDB [information_schema]> show variables like '%conn%';
|
Current database: information_schema
|
|
|
+--------------------------+-----------------+
|
| Variable_name | Value |
|
+--------------------------+-----------------+
|
| character_set_connection | utf8 |
|
| collation_connection | utf8_general_ci |
|
| connect_timeout | 10 |
|
| extra_max_connections | 1 |
|
| init_connect | |
|
| max_connect_errors | 10 |
|
| max_connections | 3200 |
|
| max_user_connections | 0 |
|
+--------------------------+-----------------+
|
8 rows in set (0.00 sec)
|
|
|
MariaDB [information_schema]> show variables like '%binlog%';
|
| binlog_annotate_row_events | ON |
|
| binlog_cache_size | 32768 |
|
| binlog_checksum | NONE |
|
| binlog_direct_non_transactional_updates | OFF |
|
| binlog_format | MIXED |
|
| binlog_optimize_thread_scheduling | ON |
|
| binlog_stmt_cache_size | 32768 |
|
| innodb_locks_unsafe_for_binlog | OFF |
|
| max_binlog_cache_size | 18446744073709547520 |
|
| max_binlog_size | 1073741824 |
|
| max_binlog_stmt_cache_size | 18446744073709547520 |
|
| sync_binlog | 0 |
|
|
|
|
|
MariaDB [(none)]> show global variables like 'Aria%';
|
...
|
MariaDB [(none)]> show global variables like 'auto%';
|
...
|
|
|
MariaDB [(none)]> show global variables like 'back%';
|
+---------------+-------+
|
| Variable_name | Value |
|
+---------------+-------+
|
| back_log | 50 |
|
+---------------+-------+
|
|
|
MariaDB [(none)]> show global variables like 'character%';
|
...
|
MariaDB [(none)]> show global variables like 'col%';
|
.
|
|
|
MariaDB [(none)]> show global variables like 'con%';
|
...
|
|
|
MariaDB [(none)]> show global variables like 'da%';
|
|
|
|
|
MariaDB [(none)]> show global variables like 'de%';
|
|
|
show global variables like 'd%';
|
..
|
show global variables like 'e%';
|
.....
|
show global variables like 'q%';
|
|
|
show global variables like 'ra%';
|
|
|
+------------------------+-------+
|
| Variable_name | Value |
|
+------------------------+-------+
|
| range_alloc_block_size | 4096 |
|
|
|
show global variables like 'read%';
|
|
|
MariaDB [(none)]> show global variables like 'relay%';
|
+-----------------------+-----------------------+
|
| Variable_name | Value |
|
+-----------------------+-----------------------+
|
| relay_log | tulwaurp009-relay-bin |
|
| relay_log_index | |
|
| relay_log_info_file | relay-log.info |
|
| relay_log_purge | ON |
|
| relay_log_recovery | OFF |
|
| relay_log_space_limit | 0 |
|
+-----------------------+-----------------------+
|
|
|
MariaDB [(none)]> show global variables like 'replicate_a%';
|
|
|
+-------------------------------+-------+
|
| Variable_name | Value |
|
+-------------------------------+-------+
|
| replicate_annotate_row_events | ON |
|
+-------------------------------+-------+
|
1 row in set (0.00 sec)
|
|
|
show global variables like 'report%';
|
| Variable_name | Value |
|
+-----------------+-------------+
|
| report_host | tulwaurp009 |
|
| report_password | |
|
| report_port | 3306 |
|
| report_user | |
|
|
|
show global variables like 'ro%';
|
show global variables like 's%';
|
show global variables like 't%';
|
....
|
show global variables like 'z%';
|
no response:
- show variables like '%table%';
- show global variables like 'r%';
- show global variables like 're%';
- show global variables like 'repl%';
- show global variables like 'replicate_d%';
- show global variables like 'replicate_do_db';
- show global variables like 'replicate_do_table;
- show global variables like 'replicate_i%';
- show global status like 'Aria%';
- show global status like 'Aria_pagecache%';
- show global status like 'Aria_trans%';
- show global status like 'Com_%';
- show global status like 'Bytes_%';
- show global status like 'Busy_%';
- show global status like 'x%';
- show status
- show global status
- show variables
- show global variables
- select * from information_schema.session_status;
- select * from information_schema.session_status limit 3;
Attachments
Issue Links
- relates to
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
- links to
Activity
| Field | Original Value | New Value |
|---|---|---|
| Assignee | Elena Stepanova [ elenst ] |
| Due Date | 2014-12-09 |
> Is this instance still up hanging?
yes.
> If so, can you get a stack trace from the running process?
I couldn't
>If the instance was shutdown, then – is it reproducible? Does it hang again when you start it again?
no.
>Did you check the obvious, like the disk space or IO errors?
Space there and no io errors.
> Could you please also attach the error log from the server startup and up to the time it definitely hang?
141024 13:07:15 mysqld_safe Starting mysqld daemon with databases from /u01/mariadb
|
141024 13:07:15 InnoDB: The InnoDB memory heap is disabled
|
141024 13:07:15 InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
141024 13:07:15 InnoDB: Compressed tables use zlib 1.2.3
|
141024 13:07:15 InnoDB: Using Linux native AIO
|
141024 13:07:15 InnoDB: Initializing buffer pool, size = 24.0G
|
141024 13:07:17 InnoDB: Completed initialization of buffer pool
|
141024 13:07:17 InnoDB: highest supported file format is Barracuda.
|
141024 13:07:19 InnoDB: Restoring buffer pool pages from ib_lru_dump
|
141024 13:07:19 InnoDB: Waiting for the background threads to start
|
141024 13:07:20 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 3338597042050
|
141024 13:07:20 [Note] Plugin 'FEEDBACK' is disabled.
|
141024 13:07:20 [Note] Server socket created on IP: '0.0.0.0'.
|
141024 13:07:20 [Note] Slave SQL thread initialized, starting replication in log 'tulwaurp005-mysql-bin.002166' at position 11843844, relay log './tulwaurp007-relay-bin.005152' positio
|
n: 11728905
|
141024 13:07:20 [Note] Event Scheduler: Loaded 0 events
|
141024 13:07:20 [Note] /usr/sbin/mysqld: ready for connections.
|
Version: '5.5.40-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
|
141024 13:07:20 [Note] Slave I/O thread: connected to master 'repl@tulwaurp005:3306',replication started in log 'tulwaurp005-mysql-bin.002166' at position 11843844
|
141024 13:08:16 InnoDB: Completed reading buffer pool pages (requested: 1568414, read: 1561319)
|
141026 0:27:01 [Note] Error reading relay log event: slave SQL thread was killed
|
141126 10:51:20 [Note] Slave I/O thread killed while reading event
|
141126 10:51:20 [Note] Slave I/O thread exiting, read up to log 'tulwaurp005-mysql-bin.002329', position 762362495
|
141126 10:51:20 [Note] /usr/sbin/mysqld: Normal shutdown
|
|
|
141126 10:51:20 [Note] Event Scheduler: Purging the queue. 0 events
|
Happy to write this off as MDEV-7026
Okay then, I think at this point it's reasonable to blame MDEV-7026, especially since we don't have much else here. Lets wait and see whether a) it happens again (if it does, maybe you can get a stack trace?), and b) if it goes away after we release 5.5.41 and you upgrade to it.
I will close it for now as a duplicate of MDEV-7026, but please feel free to comment if you have any new information, and we'll re-open if necessary.
| Fix Version/s | 5.5.41 [ 17600 ] | |
| Resolution | Fixed [ 1 ] | |
| Status | Open [ 1 ] | Closed [ 6 ] |
| Assignee | Elena Stepanova [ elenst ] |
| Affects Version/s | 5.5.39 [ 16301 ] |
This also re-occured on 5.5.39 instance (same machine).
Id User Host db Command Time State Info Progress
|
6699 system user NULL Connect 3488030 Queueing master event to the relay log NULL 0.000
|
6700 system user NULL Connect 2883230 closing tables NULL 0.000
|
26870 mysql localhost NULL Query 2883230 Filling schema table show /*!50002 GLOBAL */ status 0.000
|
26871 mysql localhost NULL Query 2883230 Killing slave STOP SLAVE 0.000
|
26872 mysql localhost NULL Query 2883170 Filling schema table show /*!50002 GLOBAL */ status 0.000
|
26873 mysql localhost NULL Query 2883110 Filling schema table show /*!50002 GLOBAL */ status 0.000
|
The show global status at the same second as the stop slave was consistent.
2883230 seconds from the STOP SLAVE in process list corresponds to 33 days go on Dec 28.
This machine was out of disk space however this occurred on 28 January as the SQL IO thread was still running and the relay logs filled the disk up.
I can't see any fixed to race conditions in replication in the 5.5.40 or 5.5.41 change logs.
I will try to reproduce with a 5.5.41 instance.
| Resolution | Fixed [ 1 ] | |
| Status | Closed [ 6 ] | Stalled [ 10000 ] |
This is a snapshot from the log file of system stats: tulwaurp009_GenInfo_20141228_0127.log.bz2 at 0127.
mysql 55574 55569 0 00:27 ? 00:00:00 /bin/sh -c $HOME/bin/mysql_stat.sh
|
mysql 55576 55570 0 00:27 ? 00:00:00 /bin/sh -c $HOME/bin/mysql_backup.sh
|
mysql 55586 55574 0 00:27 ? 00:00:00 mysqladmin extended-status
|
mysql 55588 55576 0 00:27 ? 00:00:00 mysqladmin stop-slave
|
mysql 55589 55570 0 00:27 ? 00:00:00 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root
|
mysql 55591 55589 0 00:27 ? 00:00:00 /usr/sbin/postdrop -r
|
root 55593 3528 0 00:28 ? 00:00:00 CROND
|
mysql 55595 55593 0 00:28 ? 00:00:00 /bin/sh -c $HOME/bin/mysql_stat.sh
|
mysql 55600 55595 0 00:28 ? 00:00:00 mysqladmin extended-status
|
root 55605 3528 0 00:29 ? 00:00:00 CROND
|
mysql 55607 55605 0 00:29 ? 00:00:00 /bin/sh -c $HOME/bin/mysql_stat.sh
|
mysql 55612 55607 0 00:29 ? 00:00:00 mysqladmin extended-status
|
root 55616 3528 0 00:30 ? 00:00:00 CROND
|
mysql 55620 55616 0 00:30 ? 00:00:00 /bin/sh -c $HOME/bin/mysql_stat.sh
|
mysql 55625 55620 0 00:30 ? 00:00:00 mysqladmin extended-status
|
As mysqladmin extended-status/stop-slave don't take an hour to run it points to a deadlock there. Also the minute /mysql_stat.sh scripts that runs the following:
mysqladmin extended-status > $logfile
|
|
|
#
|
# Check that the slave is running
|
if test `grep 'Slave_running' $logfile | grep -c 'ON'` -eq 0
|
then
|
#
|
# get some diagnostics
|
mysql --batch <<EOT >> $logfile 2>&1
|
select sysdate()\\G
|
show innodb status\\G
|
show processlist\\G
|
show slave status\\G
|
EOT
|
...
|
| Fix Version/s | 5.5 [ 15800 ] | |
| Fix Version/s | 5.5.41 [ 17600 ] |
| Assignee | Elena Stepanova [ elenst ] |
Was there anything else in the process list apart from the quoted 6 lines and apparently countless other SHOW GLOBAL STATUS if it kept being issued every 60 seconds – any other SQL queries?
There were a couple deadlocks fixed in 5.5.40, but both involve several types of statements executed at once:
MDEV-6774 (involves SHOW STATUS as well)
MDEV-6749 (less chance it's relevant, but still)
| Due Date | 2014-12-09 | 2015-02-23 |
| Fix Version/s | N/A [ 14700 ] | |
| Fix Version/s | 5.5 [ 15800 ] |
> any other SQL queries?
many days later there is the following at the end of the show global status.
27116 mysql localhost NULL Query 546830 Filling schema table show status 0.000
|
27117 mysql localhost NULL Query 464030 NULL STOP SLAVE 0.000
|
27118 mysql localhost NULL Query 460429 Filling schema table show status 0.000
|
27119 mysql localhost NULL Query 374030 Filling schema table show status 0.000
|
27120 mysql localhost NULL Query 287629 Filling schema table show status 0.000
|
27121 mysql localhost NULL Query 201230 Filling schema table show status 0.000
|
27122 mysql localhost NULL Query 114830 Filling schema table show status 0.000
|
27123 mysql localhost NULL Query 28430 Filling schema table show status 0.000
|
27124 root localhost NULL Killed NULL login NULL 0.000
|
27125 munin localhost mysql Killed 428 NULL NULL 0.000
|
27126 munin localhost mysql Query 428 Filling schema table SHOW GLOBAL STATUS 0.000
|
> a couple deadlocks fixed in 5.5.40
The original bug report before reopening was on 5.5.40. Have reasked client for permission to test .41/.42
5.5.41
sudo cat /var/lib/mysql/mysqld.log
|
150210 15:50:32 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
|
150210 15:50:34 mysqld_safe Starting mysqld daemon with databases from /u01/mariadb
|
150210 15:50:34 InnoDB: The InnoDB memory heap is disabled
|
150210 15:50:34 InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
150210 15:50:34 InnoDB: Compressed tables use zlib 1.2.3
|
150210 15:50:34 InnoDB: Using Linux native AIO
|
150210 15:50:34 InnoDB: Initializing buffer pool, size = 24.0G
|
150210 15:50:35 InnoDB: Completed initialization of buffer pool
|
150210 15:50:35 InnoDB: highest supported file format is Barracuda.
|
150210 15:50:35 InnoDB: Restoring buffer pool pages from ib_lru_dump
|
150210 15:50:35 InnoDB: Waiting for the background threads to start
|
150210 15:50:36 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 4168206481196
|
150210 15:50:37 [Note] Plugin 'FEEDBACK' is disabled.
|
150210 15:50:37 [Note] Server socket created on IP: '0.0.0.0'.
|
150210 15:50:37 [Note] Slave SQL thread initialized, starting replication in log 'tulwaurp005-mysql-bin.002754' at position 635777838, relay log './tulwaurp009-relay-bin.000164' position: 615300928
|
150210 15:50:37 [Note] Event Scheduler: Loaded 0 events
|
150210 15:50:37 [Note] /usr/sbin/mysqld: ready for connections.
|
Version: '5.5.41-MariaDB-log' socket: '/u01/mariadb/mysql.sock' port: 3306 MariaDB Server
|
150210 15:50:37 [Note] Slave I/O thread: connected to master 'repl@tulwaurp005:3306',replication started in log 'tulwaurp005-mysql-bin.002754' at position 635777838
|
150210 15:51:21 InnoDB: Completed reading buffer pool pages (requested: 1464750, read: 1461467)
|
150210 15:54:02 [Note] Error reading relay log event: slave SQL thread was killed
|
150210 15:54:02 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)
|
150210 15:54:02 [Note] Slave I/O thread killed while reading event
|
150210 15:54:02 [Note] Slave I/O thread exiting, read up to log 'tulwaurp005-mysql-bin.002754', position 645162051
|
crontab -l
|
# MDEV-6774 test
|
* * * * * mysql -e 'show global status' > /dev/null 2>&1
|
* * * * * mysql -e 'stop slave; start slave;' > /dev/null 2>&1
|
immediately got to the state:
| Id | User | Host | db | Command | Time | State | Info | Progress |
|
+----+-------------+-----------+-------+---------+------+---------------------------------+--------------------+----------+
|
| 1 | system user | | NULL | Connect | 509 | Waiting for slave mutex on exit | NULL | 0.000 |
|
| 8 | openquery | localhost | NULL | Query | 304 | Flushing relay-log info file. | stop slave | 0.000 |
|
| 9 | openquery | localhost | NULL | Query | 304 | Filling schema table | show global status | 0.000 |
|
| 10 | openquery | localhost | NULL | Killed | 296 | NULL | stop slave | 0.000 |
|
| 11 | openquery | localhost | NULL | Killed | 171 | NULL | show slave status | 0.000 |
|
| 12 | openquery | localhost | NULL | Query | 245 | Filling schema table | show global status | 0.000 |
|
| 13 | openquery | localhost | NULL | Query | 245 | NULL | stop slave | 0.000 |
|
| 14 | munin | localhost | mysql | Query | 243 | Filling schema table | SHOW GLOBAL STATUS | 0.000 |
|
| 15 | openquery | localhost | NULL | Query | 185 | Filling schema table | show global status | 0.000 |
|
| 16 | openquery | localhost | NULL | Query | 185 | NULL | stop slave | 0.000 |
|
| 17 | munin | localhost | mysql | Query | 180 | Filling schema table | SHOW GLOBAL STATUS | 0.000 |
|
| 18 | openquery | localhost | NULL | Killed | 155 | NULL | NULL | 0.000 |
|
| 19 | openquery | localhost | NULL | Killed | 153 | NULL | NULL | 0.000 |
|
| 20 | openquery | localhost | NULL | Killed | 152 | NULL | show slave status | 0.000 |
|
| 21 | openquery | localhost | NULL | Killed | 149 | NULL | NULL | 0.000 |
|
| 22 | openquery | localhost | NULL | Killed | 147 | NULL | NULL | 0.000 |
|
| 23 | openquery | localhost | NULL | Killed | 106 | NULL | show slave status | 0.000 |
|
| 24 | openquery | localhost | NULL | Query | 125 | NULL | stop slave | 0.000 |
|
| 25 | openquery | localhost | NULL | Query | 125 | Filling schema table | show global status | 0.000 |
|
| 26 | munin | localhost | mysql | Query | 118 | Filling schema table | SHOW GLOBAL STATUS | 0.000 |
|
| 27 | openquery | localhost | NULL | Killed | 110 | NULL | NULL | 0.000 |
|
| 28 | openquery | localhost | NULL | Killed | 109 | NULL | NULL | 0.000 |
|
| 29 | openquery | localhost | NULL | Killed | 104 | NULL | NULL | 0.000 |
|
| 30 | openquery | localhost | NULL | Killed | 103 | NULL | NULL | 0.000 |
|
| 31 | openquery | localhost | NULL | Killed | 90 | NULL | show slave status | 0.000 |
|
| 32 | openquery | localhost | NULL | Killed | 88 | NULL | NULL | 0.000 |
|
| 33 | openquery | localhost | NULL | Killed | 87 | NULL | NULL | 0.000 |
|
| 34 | openquery | localhost | NULL | Query | 65 | NULL | stop slave | 0.000 |
|
| 35 | openquery | localhost | NULL | Query | 65 | Filling schema table | show global status | 0.000 |
|
| 36 | munin | localhost | mysql | Query | 56 | Filling schema table | SHOW GLOBAL STATUS | 0.000 |
|
| 37 | openquery | localhost | NULL | Query | 5 | Filling schema table | show global status | 0.000 |
|
| 38 | openquery | localhost | NULL | Query | 5 | NULL | stop slave | 0.000 |
|
| 39 | openquery | localhost | NULL | Query | 0 | NULL | show processlist | 0.000 |
|
(gdb) bt full
|
#0 0x0000003f8d0df293 in poll () from /lib64/libc.so.6
|
No symbol table info available.
|
#1 0x0000000000523653 in handle_connections_sockets() ()
|
No symbol table info available.
|
#2 0x00000000005273a2 in mysqld_main(int, char**) ()
|
No symbol table info available.
|
#3 0x0000003f8d01ecdd in __libc_start_main () from /lib64/libc.so.6
|
No symbol table info available.
|
#4 0x000000000051d7e9 in _start ()
|
No symbol table info available.
|
(gdb) quit
|
| Affects Version/s | 5.5.41 [ 17600 ] |
| Due Date | 2015-02-23 |
I can reproduce it on MariaDB 5.5, but not on 10.0 (and not on MySQL 5.5/5.6).
Test case:
--source include/master-slave.inc
|
|
|
--connection slave
|
--echo # If everything is okay, the test will end in several seconds; maybe a minute.
|
--echo # If the problem shows up, it will hang until testcase timeout is exceeded.
|
--exec $MYSQL_SLAP --socket=$SLAVE_MYSOCK -q "START SLAVE; STOP SLAVE; SHOW GLOBAL STATUS" -c 2 --number-of-queries=100 --create-schema=test
|
--echo # All done.
|
The problem appeared in 5.5.37 with the following revision:
revno: 4110
|
revision-id: monty@mariadb.org-20140314142923-wvho2wb4i8s7wb1v
|
parent: svoj@mariadb.org-20140213074049-wo2l3qdtgi0s2mjd
|
committer: Michael Widenius <monty@mariadb.org>
|
branch nick: maria-5.5
|
timestamp: Fri 2014-03-14 16:29:23 +0200
|
message:
|
MDEV-5829: STOP SLAVE resets global status variables
|
|
Reason for the bug was an optimization for higher connect speed where we moved when global status was updated,
|
but forgot to update states when slave thread dies.
|
Fixed by adding thd->add_status_to_global() before deleting slave thread's thd.
|
| Fix Version/s | 5.5 [ 15800 ] | |
| Fix Version/s | N/A [ 14700 ] | |
| Affects Version/s | 5.5.37 [ 15000 ] | |
| Affects Version/s | 5.5.39 [ 16301 ] | |
| Affects Version/s | 5.5.40 [ 17100 ] | |
| Affects Version/s | 5.5.41 [ 17600 ] |
| Assignee | Elena Stepanova [ elenst ] | Sergei Golubchik [ serg ] |
| Labels | verified |
| Priority | Major [ 3 ] | Critical [ 2 ] |
| Attachment | gdb.txt [ 37606 ] |
| Affects Version/s | 10.0.17 [ 18300 ] |
| Summary | replication slave - show variables of replication filters and all show status fail to return | replication slave - deadlock with stop slave and show variables of replication filters and all show status fail to return |
| Attachment | gdb-10.0-terminate_slave_thread.txt [ 37611 ] |
| Summary | replication slave - deadlock with stop slave and show variables of replication filters and all show status fail to return | replication slave - deadlock in terminate_slave_thread with stop slave and show variables of replication filters and show global status |
| Status | Stalled [ 10000 ] | In Progress [ 3 ] |
| Status | In Progress [ 3 ] | Stalled [ 10000 ] |
danblack, I'm only fixing 5.5 bug here. 10.0 deadlock is different and needs a separate bug report. Could you move it there, please?
| Attachment | gdb-10.0-terminate_slave_thread.txt [ 37611 ] |
| Attachment | gdb.txt [ 37606 ] |
| Comment |
[ reproduced on 10.0.17: its quite likely this had an active IO thread and stopped SQL thread (was running pt-slave-delay --delay 24h --interval 1hr) {noformat} mysql -S /var/lib/mariadb/mysql-10.sock Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 1438 Server version: 10.0.17-MariaDB-log MariaDB Server Copyright (c) 2000, 2015, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> stop slave; set global slave_domain_parallel_threads=15; start slave; {noformat} {noformat} ]$ mysql -u openquery -S /var/lib/mariadb/mysql-10.sock Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 1443 Server version: 10.0.17-MariaDB-log MariaDB Server Copyright (c) 2000, 2015, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show processlist; +------+-------------+-----------+--------------------+---------+--------+--------------------------------------------------------------------------------+--------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +------+-------------+-----------+--------------------+---------+--------+--------------------------------------------------------------------------------+--------------------+----------+ | 3 | system user | | NULL | Connect | 264856 | Waiting for work from SQL thread | NULL | 0.000 | | 4 | system user | | NULL | Connect | 264856 | Waiting for work from SQL thread | NULL | 0.000 | | 5 | system user | | NULL | Connect | 264856 | Waiting for work from SQL thread | NULL | 0.000 | | 6 | system user | | NULL | Connect | 264856 | Waiting for work from SQL thread | NULL | 0.000 | | 7 | system user | | NULL | Connect | 264856 | Waiting for work from SQL thread | NULL | 0.000 | | 8 | system user | | NULL | Connect | 264856 | Waiting for work from SQL thread | NULL | 0.000 | | 9 | system user | | NULL | Connect | 264856 | Waiting for work from SQL thread | NULL | 0.000 | | 10 | system user | | NULL | Connect | 264856 | Waiting for work from SQL thread | NULL | 0.000 | | 11 | system user | | NULL | Connect | 264856 | Waiting for work from SQL thread | NULL | 0.000 | | 12 | system user | | NULL | Connect | 264856 | Waiting for work from SQL thread | NULL | 0.000 | | 13 | system user | | NULL | Connect | 114397 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 14 | system user | | NULL | Connect | 114397 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 15 | system user | | NULL | Connect | 114397 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 16 | system user | | NULL | Connect | 114396 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 17 | system user | | NULL | Connect | 114398 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 18 | system user | | NULL | Connect | 114396 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 19 | system user | | NULL | Connect | 114398 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 20 | system user | | NULL | Connect | 114398 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 21 | system user | | NULL | Connect | 114396 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 22 | system user | | NULL | Connect | 114397 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 24 | system user | | NULL | Connect | 264856 | Waiting for master to send event | NULL | 0.000 | | 39 | openquery | localhost | NULL | Sleep | 907 | | NULL | 0.000 | | 1349 | system user | | NULL | Connect | 26107 | Reading event from the relay log | NULL | 0.000 | | 1438 | root | localhost | NULL | Query | 147 | Killing slave | stop slave | 0.000 | | 1440 | munin | localhost | information_schema | Query | 76 | Filling schema table | SHOW GLOBAL STATUS | 0.000 | | 1441 | root | localhost | NULL | Query | 41 | init | show slave status | 0.000 | | 1442 | munin | localhost | information_schema | Query | 14 | Filling schema table | SHOW GLOBAL STATUS | 0.000 | | 1443 | openquery | localhost | NULL | Query | 0 | init | show processlist | 0.000 | +------+-------------+-----------+--------------------+---------+--------+--------------------------------------------------------------------------------+--------------------+----------+ {noformat} ] |
| Comment |
[ {noformat} MariaDB [(none)]> show processlist; +----+-------------+-----------+------+---------+------+----------------------------------+--------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +----+-------------+-----------+------+---------+------+----------------------------------+--------------------+----------+ | 3 | openquery | localhost | NULL | Query | 0 | NULL | show processlist | 0.000 | | 38 | system user | | NULL | Connect | 35 | Waiting for master to send event | NULL | 0.000 | | 39 | system user | | NULL | Connect | 31 | closing tables | NULL | 0.000 | | 40 | openquery | localhost | NULL | Query | 31 | Filling schema table | show global status | 0.000 | | 41 | openquery | localhost | NULL | Query | 31 | Killing slave | stop slave | 0.000 | +----+-------------+-----------+------+---------+------+----------------------------------+--------------------+----------+ 5 rows in set (0.00 sec) {noformat} {noformat} (gdb) thread apply all bt Thread 48 (Thread 0x7f2cc83ff700 (LWP 51655)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 47 (Thread 0x7f2642bff700 (LWP 51656)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 46 (Thread 0x7f26421fe700 (LWP 51657)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 45 (Thread 0x7f26417fd700 (LWP 51658)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 44 (Thread 0x7f2640dfc700 (LWP 51659)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7f26403fb700 (LWP 51660)): ---Type <return> to continue, or q <return> to quit--- #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7f263f9fa700 (LWP 51661)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7f263eff9700 (LWP 51662)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7f263e5f8700 (LWP 51663)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7f263dbf7700 (LWP 51664)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7f263d1f6700 (LWP 51665)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () ---Type <return> to continue, or q <return> to quit--- #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7f263c7f5700 (LWP 51666)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7f263bdf4700 (LWP 51667)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f263b3f3700 (LWP 51668)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f263a9f2700 (LWP 51669)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f2639ff1700 (LWP 51670)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () ---Type <return> to continue, or q <return> to quit--- #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f26395f0700 (LWP 51671)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f2638bef700 (LWP 51672)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f26381ee700 (LWP 51673)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f26377ed700 (LWP 51674)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f2636dec700 (LWP 51675)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 27 (Thread 0x7f26363eb700 (LWP 51676)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f26359ea700 (LWP 51677)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f2634fe9700 (LWP 51678)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f26345e8700 (LWP 51679)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f2633be7700 (LWP 51680)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f26331e6700 (LWP 51681)): ---Type <return> to continue, or q <return> to quit--- #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f26327e5700 (LWP 51682)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f2631de4700 (LWP 51683)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f26313e3700 (LWP 51684)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f26309e2700 (LWP 51685)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f262ffe1700 (LWP 51686)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () ---Type <return> to continue, or q <return> to quit--- #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f262f5e0700 (LWP 51687)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f262ebdf700 (LWP 51688)): #0 0x0000000000950d14 in __io_getevents_0_4 () #1 0x0000000000927564 in os_aio_linux_handle () #2 0x00000000008da2d5 in fil_aio_wait () #3 0x00000000008480b8 in io_handler_thread () #4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f260a5fe700 (LWP 51690)): #0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000092959b in os_event_wait_time_low () #2 0x0000000000844a70 in srv_lock_timeout_thread () #3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f260afff700 (LWP 51691)): #0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000092959b in os_event_wait_time_low () #2 0x0000000000844fc4 in srv_error_monitor_thread () #3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f2609bfd700 (LWP 51692)): #0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000092959b in os_event_wait_time_low () #2 0x0000000000844783 in srv_monitor_thread () #3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 11 (Thread 0x7f26091fc700 (LWP 51693)): #0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000092959b in os_event_wait_time_low () #2 0x0000000000845373 in srv_LRU_dump_restore_thread () #3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f26087fb700 (LWP 51694)): #0 0x0000003f8d40b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000929491 in os_event_wait_low () #2 0x0000000000845eee in srv_master_thread () #3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f2607dfa700 (LWP 51695)): #0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000092959b in os_event_wait_time_low () #2 0x0000000000847e91 in srv_purge_thread () #3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f26073f9700 (LWP 51696)): #0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000009a4587 in my_service_thread_sleep () #2 0x000000000099ce02 in ma_checkpoint_background () #3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f2cc8d3d700 (LWP 51697)): #0 0x0000003f8d40f2a5 in sigwait () from /lib64/libpthread.so.0 #1 0x000000000051fdff in signal_hand () #2 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #3 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f2cc8c62700 (LWP 51721)): #0 0x0000003f8d40e54d in read () from /lib64/libpthread.so.0 #1 0x0000000000aa4a98 in vio_read () #2 0x00000000005281d1 in my_real_read(st_net*, unsigned long*) () #3 0x0000000000529099 in my_net_read () #4 0x00000000005a4df2 in do_command(THD*) () #5 0x000000000065512b in do_handle_one_connection(THD*) () #6 0x00000000006551ac in handle_one_connection () ---Type <return> to continue, or q <return> to quit--- #7 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #8 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f2cc79fe700 (LWP 51738)): #0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000535532 in terminate_slave_thread(THD*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, bool) () #2 0x000000000053d0ac in terminate_slave_threads () #3 0x00000000005b691d in stop_slave(THD*, Master_info*, bool) () #4 0x000000000059eebe in mysql_execute_command(THD*) () #5 0x00000000005a2854 in mysql_parse () #6 0x00000000005a4d08 in dispatch_command(enum_server_command, THD*, char*, unsigned int) () #7 0x000000000065512b in do_handle_one_connection(THD*) () #8 0x00000000006551ac in handle_one_connection () #9 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #10 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f2cc79b5700 (LWP 51739)): #0 0x0000003f8d40e054 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x0000003f8d4093be in _L_lock_995 () from /lib64/libpthread.so.0 #2 0x0000003f8d409326 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x000000000051f6b8 in inline_mysql_mutex_lock.constprop.126 () #4 0x000000000052024c in show_heartbeat_period(THD*, st_mysql_show_var*, char*) () #5 0x00000000005f38a4 in show_status_array(THD*, char const*, st_mysql_show_var*, enum_var_type, system_status_var*, char const*, TABLE*, bool, Item*) () #6 0x00000000005fe618 in fill_status(THD*, TABLE_LIST*, Item*) () #7 0x0000000000602d67 in get_schema_tables_result(JOIN*, enum_schema_table_state) () #8 0x00000000005e9c8d in JOIN::exec() () #9 0x00000000005e5525 in mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) () #10 0x00000000005ec113 in handle_select(THD*, LEX*, select_result*, unsigned long) () #11 0x00000000005970a6 in execute_sqlcom_select(THD*, TABLE_LIST*) () #12 0x000000000059e87f in mysql_execute_command(THD*) () #13 0x00000000005a2854 in mysql_parse () #14 0x00000000005a4d08 in dispatch_command(enum_server_command, THD*, char*, unsigned int) () #15 0x000000000065512b in do_handle_one_connection(THD*) () #16 0x00000000006551ac in handle_one_connection () #17 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #18 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f2cc8cf4700 (LWP 51804)): #0 0x0000003f8d40e54d in read () from /lib64/libpthread.so.0 #1 0x0000000000aa4a98 in vio_read () #2 0x0000000000aa4b32 in vio_read_buff () ---Type <return> to continue, or q <return> to quit--- #3 0x00000000005281d1 in my_real_read(st_net*, unsigned long*) () #4 0x0000000000529035 in my_net_read () #5 0x00000000006a9365 in cli_safe_read () #6 0x000000000053db43 in handle_slave_io () #7 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #8 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f2cc8cab700 (LWP 51805)): #0 0x0000003f8d40e054 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x0000003f8d4093be in _L_lock_995 () from /lib64/libpthread.so.0 #2 0x0000003f8d409326 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000534c50 in inline_mysql_mutex_lock () #4 0x000000000053ee10 in THD::add_status_to_global() () #5 0x000000000053c34b in handle_slave_sql () #6 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003f8d0e894d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f2cc8da2800 (LWP 51653)): #0 0x0000003f8d0df293 in poll () from /lib64/libc.so.6 #1 0x0000000000523653 in handle_connections_sockets() () #2 0x00000000005273a2 in mysqld_main(int, char**) () #3 0x0000003f8d01ecdd in __libc_start_main () from /lib64/libc.so.6 #4 0x000000000051d7e9 in _start () {noformat} ] |
| Comment |
[ SQL thread was stopped - set password on non-existent user. {noformat} MariaDB [(none)]> set global sql_slave_skip_counter=1; start slave; ERROR 1198 (HY000): This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first Query OK, 0 rows affected, 1 warning (0.00 sec) MariaDB [(none)]> stop slave;set global sql_slave_skip_counter=1; start slave; {noformat} {noformat} erver version: 10.0.17-MariaDB-log MariaDB Server Copyright (c) 2000, 2015, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show processlist; | Id | User | Host | db | Command | Time | State | Info | Progress | | 3 | system user | | NULL | Connect | 1010764 | Waiting for work from SQL thread | NULL | 0.000 | | 4 | system user | | NULL | Connect | 1010764 | Waiting for work from SQL thread | NULL | 0.000 | | 5 | system user | | NULL | Connect | 1010764 | Waiting for work from SQL thread | NULL | 0.000 | | 6 | system user | | NULL | Connect | 1010764 | Waiting for work from SQL thread | NULL | 0.000 | | 7 | system user | | NULL | Connect | 1010764 | Waiting for work from SQL thread | NULL | 0.000 | | 8 | system user | | NULL | Connect | 944326 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 9 | system user | | NULL | Connect | 944324 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 10 | system user | | NULL | Connect | 944327 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 11 | system user | | NULL | Connect | 944326 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 12 | system user | | NULL | Connect | 944327 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 13 | system user | | NULL | Connect | 944326 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 14 | system user | | NULL | Connect | 944325 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 15 | system user | | NULL | Connect | 944328 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 16 | system user | | NULL | Connect | 944328 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 17 | system user | | NULL | Connect | 944324 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 18 | system user | | NULL | Connect | 944328 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 19 | system user | | NULL | Connect | 944324 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 20 | system user | | NULL | Connect | 944326 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 21 | system user | | NULL | Connect | 944324 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 22 | system user | | NULL | Connect | 944324 | Waiting for prior transaction to start commit before starting next transaction | NULL | 0.000 | | 23 | system user | | NULL | Connect | 1010763 | Waiting for master to send event | NULL | 0.000 | | 28 | openquery | localhost | NULL | Killed | 127489 | init | SHOW SLAVE STATUS | 0.000 | | 291 | system user | | NULL | Connect | 854691 | Reading event from the relay log | NULL | 0.000 | | 2711 | openquery | localhost | NULL | Query | 129024 | Killing slave | stop slave | 0.000 | | 2715 | munin | localhost | information_schema | Killed | 128765 | Filling schema table | SHOW GLOBAL STATUS | 0.000 | | 2716 | root | localhost | NULL | Killed | 128709 | init | show slave status | 0.000 | | 2719 | munin | localhost | information_schema | Killed | 128703 | Filling schema table | SHOW GLOBAL STATUS | 0.000 | | 2741 | openquery | localhost | NULL | Query | 0 | init | show processlist | 0.000 | 28 rows in set (0.00 sec) {noformat} Threads where well and truly halted before i killed off the global status threads. {noformat} (gdb) where #0 0x000000348e40b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000056167f in terminate_slave_thread () #2 0x0000000000564d45 in terminate_slave_threads(Master_info*, int, bool) () #3 0x00000000005efabf in stop_slave(THD*, Master_info*, bool) () #4 0x00000000005d3fe2 in mysql_execute_command(THD*) () #5 0x00000000005d8552 in mysql_parse () #6 0x00000000005da7b0 in dispatch_command(enum_server_command, THD*, char*, unsigned int) () #7 0x00000000006963d3 in do_handle_one_connection(THD*) () #8 0x00000000006964a2 in handle_one_connection () #9 0x000000348e4079d1 in start_thread () from /lib64/libpthread.so.0 #10 0x000000348e0e8b6d in clone () from /lib64/libc.so.6 {noformat} full bt in gdb-10.0-terminate_slave_thread.txt ] |
| Comment |
[ Looking at the show slave status this is at the pthread_mutex_lock position 0x63551a in fill_status though a copy show pthread_mutex lock is from {{fill_status(635419)\->show_status_array(62a94c)->show_heartbeat_period(54a982)}}. show_heartbeat_period is blocked acquiring the LOCK_active_mi. fill_status(63551a) is waiting on LOCK_show_status just before show_status_array which means its waiting on the fill_status that reached show_heartbeat_period. 0x5d5953 in mysql_execute_command is waiting on a lock LOCK_active_mi for a show slave status command. terminate_slave_thread looking at the disassembled code the following is the only cond_timedwait {noformat} mysql_cond_timedwait(term_cond, term_lock, &abstime) {noformat} and its waiting here. thd->LOCK_thd_data is unlocked. term_lock is sql_lock (in terminate_slave_threads) based on the position in the terminate_slave_threads, which is (mi->rli.run_lock) (which is locked) in stop_slave (by calling lock_slave_thread). term_cond is mi->rli.stop_cond We can see this is terminating the sql thread. In both stack traces have handle_slave_sql(line 4676) calling in rpl_parallel::wait_for_done(6d52ec) at mysql_cond_wait(&e->COND_parallel_entry, &rpt->LOCK_rpl_thread); gdb-10.0-terminate_slave_thread.txt has two of these threads in the same place which strikes me as odd -(might have been multisource)- (wasn't - just remembered which server it was on). So the blocking {{rpl_parallel::wait_for_done}} at a lock point and is preventing the {{mysql_cond_broadcast(&rli->stop_cond); }}at the bottom of the handle_slave_sql thread. So the point of being stuck is at {{rpl_parallel::wait_for_done(6d52ec)}} at {{mysql_cond_wait(&e->COND_parallel_entry, &rpt->LOCK_rpl_thread)}} Which is obviously a different bug to the original 5.5.X bug. ] |
re-adding over-culled comment:
MariaDB [(none)]> show processlist;
|
+----+-------------+-----------+------+---------+------+----------------------------------+--------------------+----------+
|
| Id | User | Host | db | Command | Time | State | Info | Progress |
|
+----+-------------+-----------+------+---------+------+----------------------------------+--------------------+----------+
|
| 3 | openquery | localhost | NULL | Query | 0 | NULL | show processlist | 0.000 |
|
| 38 | system user | | NULL | Connect | 35 | Waiting for master to send event | NULL | 0.000 |
|
| 39 | system user | | NULL | Connect | 31 | closing tables | NULL | 0.000 |
|
| 40 | openquery | localhost | NULL | Query | 31 | Filling schema table | show global status | 0.000 |
|
| 41 | openquery | localhost | NULL | Query | 31 | Killing slave | stop slave | 0.000 |
|
+----+-------------+-----------+------+---------+------+----------------------------------+--------------------+----------+
|
5 rows in set (0.00 sec)
|
(gdb) thread apply all bt
|
|
|
Thread 48 (Thread 0x7f2cc83ff700 (LWP 51655)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 47 (Thread 0x7f2642bff700 (LWP 51656)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 46 (Thread 0x7f26421fe700 (LWP 51657)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 45 (Thread 0x7f26417fd700 (LWP 51658)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 44 (Thread 0x7f2640dfc700 (LWP 51659)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 43 (Thread 0x7f26403fb700 (LWP 51660)):
|
---Type <return> to continue, or q <return> to quit---
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 42 (Thread 0x7f263f9fa700 (LWP 51661)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 41 (Thread 0x7f263eff9700 (LWP 51662)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 40 (Thread 0x7f263e5f8700 (LWP 51663)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 39 (Thread 0x7f263dbf7700 (LWP 51664)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 38 (Thread 0x7f263d1f6700 (LWP 51665)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
---Type <return> to continue, or q <return> to quit---
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 37 (Thread 0x7f263c7f5700 (LWP 51666)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 36 (Thread 0x7f263bdf4700 (LWP 51667)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 35 (Thread 0x7f263b3f3700 (LWP 51668)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 34 (Thread 0x7f263a9f2700 (LWP 51669)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 33 (Thread 0x7f2639ff1700 (LWP 51670)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
---Type <return> to continue, or q <return> to quit---
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 32 (Thread 0x7f26395f0700 (LWP 51671)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 31 (Thread 0x7f2638bef700 (LWP 51672)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 30 (Thread 0x7f26381ee700 (LWP 51673)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 29 (Thread 0x7f26377ed700 (LWP 51674)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 28 (Thread 0x7f2636dec700 (LWP 51675)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
---Type <return> to continue, or q <return> to quit---
|
|
|
Thread 27 (Thread 0x7f26363eb700 (LWP 51676)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 26 (Thread 0x7f26359ea700 (LWP 51677)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 25 (Thread 0x7f2634fe9700 (LWP 51678)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 24 (Thread 0x7f26345e8700 (LWP 51679)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 23 (Thread 0x7f2633be7700 (LWP 51680)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 22 (Thread 0x7f26331e6700 (LWP 51681)):
|
---Type <return> to continue, or q <return> to quit---
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 21 (Thread 0x7f26327e5700 (LWP 51682)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 20 (Thread 0x7f2631de4700 (LWP 51683)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 19 (Thread 0x7f26313e3700 (LWP 51684)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 18 (Thread 0x7f26309e2700 (LWP 51685)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 17 (Thread 0x7f262ffe1700 (LWP 51686)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
---Type <return> to continue, or q <return> to quit---
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 16 (Thread 0x7f262f5e0700 (LWP 51687)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 15 (Thread 0x7f262ebdf700 (LWP 51688)):
|
#0 0x0000000000950d14 in __io_getevents_0_4 ()
|
#1 0x0000000000927564 in os_aio_linux_handle ()
|
#2 0x00000000008da2d5 in fil_aio_wait ()
|
#3 0x00000000008480b8 in io_handler_thread ()
|
#4 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#5 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 14 (Thread 0x7f260a5fe700 (LWP 51690)):
|
#0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
|
#1 0x000000000092959b in os_event_wait_time_low ()
|
#2 0x0000000000844a70 in srv_lock_timeout_thread ()
|
#3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 13 (Thread 0x7f260afff700 (LWP 51691)):
|
#0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
|
#1 0x000000000092959b in os_event_wait_time_low ()
|
#2 0x0000000000844fc4 in srv_error_monitor_thread ()
|
#3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 12 (Thread 0x7f2609bfd700 (LWP 51692)):
|
#0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
|
#1 0x000000000092959b in os_event_wait_time_low ()
|
#2 0x0000000000844783 in srv_monitor_thread ()
|
#3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
---Type <return> to continue, or q <return> to quit---
|
Thread 11 (Thread 0x7f26091fc700 (LWP 51693)):
|
#0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
|
#1 0x000000000092959b in os_event_wait_time_low ()
|
#2 0x0000000000845373 in srv_LRU_dump_restore_thread ()
|
#3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 10 (Thread 0x7f26087fb700 (LWP 51694)):
|
#0 0x0000003f8d40b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
|
#1 0x0000000000929491 in os_event_wait_low ()
|
#2 0x0000000000845eee in srv_master_thread ()
|
#3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 9 (Thread 0x7f2607dfa700 (LWP 51695)):
|
#0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
|
#1 0x000000000092959b in os_event_wait_time_low ()
|
#2 0x0000000000847e91 in srv_purge_thread ()
|
#3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 8 (Thread 0x7f26073f9700 (LWP 51696)):
|
#0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
|
#1 0x00000000009a4587 in my_service_thread_sleep ()
|
#2 0x000000000099ce02 in ma_checkpoint_background ()
|
#3 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#4 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 7 (Thread 0x7f2cc8d3d700 (LWP 51697)):
|
#0 0x0000003f8d40f2a5 in sigwait () from /lib64/libpthread.so.0
|
#1 0x000000000051fdff in signal_hand ()
|
#2 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#3 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 6 (Thread 0x7f2cc8c62700 (LWP 51721)):
|
#0 0x0000003f8d40e54d in read () from /lib64/libpthread.so.0
|
#1 0x0000000000aa4a98 in vio_read ()
|
#2 0x00000000005281d1 in my_real_read(st_net*, unsigned long*) ()
|
#3 0x0000000000529099 in my_net_read ()
|
#4 0x00000000005a4df2 in do_command(THD*) ()
|
#5 0x000000000065512b in do_handle_one_connection(THD*) ()
|
#6 0x00000000006551ac in handle_one_connection ()
|
---Type <return> to continue, or q <return> to quit---
|
#7 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#8 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 5 (Thread 0x7f2cc79fe700 (LWP 51738)):
|
#0 0x0000003f8d40b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
|
#1 0x0000000000535532 in terminate_slave_thread(THD*, st_mysql_mutex*, st_mysql_cond*, unsigned int volatile*, bool) ()
|
#2 0x000000000053d0ac in terminate_slave_threads ()
|
#3 0x00000000005b691d in stop_slave(THD*, Master_info*, bool) ()
|
#4 0x000000000059eebe in mysql_execute_command(THD*) ()
|
#5 0x00000000005a2854 in mysql_parse ()
|
#6 0x00000000005a4d08 in dispatch_command(enum_server_command, THD*, char*, unsigned int) ()
|
#7 0x000000000065512b in do_handle_one_connection(THD*) ()
|
#8 0x00000000006551ac in handle_one_connection ()
|
#9 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#10 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 4 (Thread 0x7f2cc79b5700 (LWP 51739)):
|
#0 0x0000003f8d40e054 in __lll_lock_wait () from /lib64/libpthread.so.0
|
#1 0x0000003f8d4093be in _L_lock_995 () from /lib64/libpthread.so.0
|
#2 0x0000003f8d409326 in pthread_mutex_lock () from /lib64/libpthread.so.0
|
#3 0x000000000051f6b8 in inline_mysql_mutex_lock.constprop.126 ()
|
#4 0x000000000052024c in show_heartbeat_period(THD*, st_mysql_show_var*, char*) ()
|
#5 0x00000000005f38a4 in show_status_array(THD*, char const*, st_mysql_show_var*, enum_var_type, system_status_var*, char const*, TABLE*, bool, Item*) ()
|
#6 0x00000000005fe618 in fill_status(THD*, TABLE_LIST*, Item*) ()
|
#7 0x0000000000602d67 in get_schema_tables_result(JOIN*, enum_schema_table_state) ()
|
#8 0x00000000005e9c8d in JOIN::exec() ()
|
#9 0x00000000005e5525 in mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) ()
|
#10 0x00000000005ec113 in handle_select(THD*, LEX*, select_result*, unsigned long) ()
|
#11 0x00000000005970a6 in execute_sqlcom_select(THD*, TABLE_LIST*) ()
|
#12 0x000000000059e87f in mysql_execute_command(THD*) ()
|
#13 0x00000000005a2854 in mysql_parse ()
|
#14 0x00000000005a4d08 in dispatch_command(enum_server_command, THD*, char*, unsigned int) ()
|
#15 0x000000000065512b in do_handle_one_connection(THD*) ()
|
#16 0x00000000006551ac in handle_one_connection ()
|
#17 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#18 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 3 (Thread 0x7f2cc8cf4700 (LWP 51804)):
|
#0 0x0000003f8d40e54d in read () from /lib64/libpthread.so.0
|
#1 0x0000000000aa4a98 in vio_read ()
|
#2 0x0000000000aa4b32 in vio_read_buff ()
|
---Type <return> to continue, or q <return> to quit---
|
#3 0x00000000005281d1 in my_real_read(st_net*, unsigned long*) ()
|
#4 0x0000000000529035 in my_net_read ()
|
#5 0x00000000006a9365 in cli_safe_read ()
|
#6 0x000000000053db43 in handle_slave_io ()
|
#7 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#8 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 2 (Thread 0x7f2cc8cab700 (LWP 51805)):
|
#0 0x0000003f8d40e054 in __lll_lock_wait () from /lib64/libpthread.so.0
|
#1 0x0000003f8d4093be in _L_lock_995 () from /lib64/libpthread.so.0
|
#2 0x0000003f8d409326 in pthread_mutex_lock () from /lib64/libpthread.so.0
|
#3 0x0000000000534c50 in inline_mysql_mutex_lock ()
|
#4 0x000000000053ee10 in THD::add_status_to_global() ()
|
#5 0x000000000053c34b in handle_slave_sql ()
|
#6 0x0000003f8d407851 in start_thread () from /lib64/libpthread.so.0
|
#7 0x0000003f8d0e894d in clone () from /lib64/libc.so.6
|
|
|
Thread 1 (Thread 0x7f2cc8da2800 (LWP 51653)):
|
#0 0x0000003f8d0df293 in poll () from /lib64/libc.so.6
|
#1 0x0000000000523653 in handle_connections_sockets() ()
|
#2 0x00000000005273a2 in mysqld_main(int, char**) ()
|
#3 0x0000003f8d01ecdd in __libc_start_main () from /lib64/libc.so.6
|
#4 0x000000000051d7e9 in _start ()
|
)
| Fix Version/s | 5.5.43 [ 18601 ] | |
| Fix Version/s | 5.5 [ 15800 ] | |
| Resolution | Fixed [ 1 ] | |
| Status | Stalled [ 10000 ] | Closed [ 6 ] |
| Remote Link | This issue links to "Applied Patch (Web Link)" [ 23206 ] |
| Workflow | MariaDB v2 [ 58609 ] | MariaDB v3 [ 65374 ] |
| Workflow | MariaDB v3 [ 65374 ] | MariaDB v4 [ 148480 ] |
Hi Daniel,
Technically replication did start, we can see both slave threads in the processlist.
But apparently the SQL thread hang on "closing tables".
Possibly it happened on the very first event, I suppose that's what you meant by "replication didn't even start".
Is this instance still up hanging? If so, can you get a stack trace from the running process?
If the instance was shutdown, then – is it reproducible? Does it hang again when you start it again?
Did you check the obvious, like the disk space or IO errors? Closing tables has to do with writing to the disk.
Could you please also attach the error log from the server startup and up to the time it definitely hang?