[MDEV-13082] Upgrade Instructions 10.1 to 10.2 Result In Indefinite Shutdown Duration Created: 2017-06-13  Updated: 2017-08-18  Resolved: 2017-08-18

Status: Closed
Project: MariaDB Server
Component/s: Documentation, Storage Engine - InnoDB
Affects Version/s: 10.1.23
Fix Version/s: 10.2.5

Type: Bug Priority: Major
Reporter: James Bowery Assignee: Ian Gilfillan
Resolution: Fixed Votes: 0
Labels: Launchpad, innodb
Environment:
  1. cat /etc/debian_version
    9.0

Attachments: File stracemysqlshutdown.log    
Issue Links:
Relates
relates to MDEV-9663 InnoDB assertion failure: *cursor->in... Closed
relates to MDEV-12289 Keep 128 persistent rollback segments... Closed

 Description   

The directions to upgrade from 10.1 to 10.2.

Step 2 spent nearly 2 days shutting down before I killed it (-9).

Every minute, the mysql/error.log emits a message like:

InnoDB: number of bytes of change buffer just merged: 18433

The number changes a bit but not much each time.

I can't query the database's system variables to discover the size of the change buffer so I can't estimate how long it's going to take.

The change buffer defaults to 25% of the buffer pool size which defaults to 128M, which would imply over 24 hours to shut down. This seems to be a bug – particularly since taking down a database server for 24 hours must be scheduled into a maintenance cycle and all services, such as websites, are down for the duration.

This symptom has cropped up with the percona server upgrade as well in the last few days.

Of possible relevance is the fact that I had, previously, updated my apt sources.list.d to contain:

# cat /etc/apt/sources.list.d/MariaDB.list 
# MariaDB 10.2 repository list - created 2017-06-13 00:34 UTC
 
# http://downloads.mariadb.org/mariadb/repositories/
 
deb [arch=i386,amd64] https://mirrors.evowise.com/mariadb/repo/10.2/debian stretch main
 
deb-src https://mirrors.evowise.com/mariadb/repo/10.2/debian stretch main

And done an apt-get update and apt-get upgrade to install 10.2, not understanding that this would not be completed automatically – and that the aforelinked upgrade instructions from 10.1 to 10.2 must be followed.

I've attached an strace of the threads involved.

Here is a list of the logs accessed by mysql currently.

# lsof -nc mysqld | grep -vE '(.so(..*)?$|.frm|.MY?|.ibd|ib_logfile|ibdata|TCP)'
 
mysqld  21386 mysql  cwd    DIR    8,0       4096   49910 /var/lib/mysql
 
mysqld  21386 mysql  rtd    DIR    8,0       4096       2 /
 
mysqld  21386 mysql  txt    REG    8,0   17504392   10289 /usr/sbin/mysqld
 
mysqld  21386 mysql  mem    REG    8,0      24576   49290 /var/lib/mysql/tc.log
 
mysqld  21386 mysql  DEL    REG   0,12            1535549 /[aio]
 
mysqld  21386 mysql  DEL    REG   0,12            1535548 /[aio]
 
mysqld  21386 mysql  DEL    REG   0,12            1535547 /[aio]
 
mysqld  21386 mysql  DEL    REG   0,12            1535546 /[aio]
 
mysqld  21386 mysql  DEL    REG   0,12            1535545 /[aio]
 
mysqld  21386 mysql  DEL    REG   0,12            1535544 /[aio]
 
mysqld  21386 mysql  DEL    REG   0,12            1535543 /[aio]
 
mysqld  21386 mysql  DEL    REG   0,12            1535542 /[aio]
 
mysqld  21386 mysql  DEL    REG   0,12            1535541 /[aio]
 
mysqld  21386 mysql  DEL    REG   0,12            1535540 /[aio]
 
mysqld  21386 mysql  DEL    REG   0,12            1535539 /[aio]
 
mysqld  21386 mysql  DEL    REG   0,12            1535538 /[aio]
 
mysqld  21386 mysql    0r   CHR    1,3        0t0 1532830 /dev/null
 
mysqld  21386 mysql    1w   REG    8,0     238675   65782 /var/log/mysql/error.log.1
 
mysqld  21386 mysql    2w   REG    8,0     238675   65782 /var/log/mysql/error.log.1
 
mysqld  21386 mysql    3uW  REG    8,0         52   49333 /var/lib/mysql/aria_log_control
 
mysqld  21386 mysql    4r   DIR    8,0       4096   49910 /var/lib/mysql
 
mysqld  21386 mysql    5u   REG    8,0      16384   49498 /var/lib/mysql/aria_log.00000001
 
mysqld  21386 mysql    7u   REG    8,0          0  303444 /tmp/ibyVu6X2 (deleted)
 
mysqld  21386 mysql    9u   REG    8,0          0  303542 /tmp/ibg6db2S (deleted)
 
mysqld  21386 mysql   10u   REG    8,0          0  303543 /tmp/ib6iaebJ (deleted)
 
mysqld  21386 mysql   14u   REG    8,0          0  303544 /tmp/ibC6qilE (deleted)
 
mysqld  21386 mysql  129u   REG    8,0      24576   49290 /var/lib/mysql/tc.log

Here are the relevant packages that are installed according to apt-show-versions:

# apt-show-versions|grep maria
 
libmariadbclient18:amd64/stretch 10.1.23-9+deb9u1 upgradeable to 10.2.6+maria~stretch
 
libmariadbclient18:i386 not installed
 
mariadb-client-10.1:amd64/stretch 10.1.23-9+deb9u1 uptodate
 
mariadb-client-10.1:i386 not installed
 
mariadb-client-core-10.1:amd64/stretch 10.1.23-9+deb9u1 uptodate
 
mariadb-client-core-10.1:i386 not installed
 
mariadb-common:all/stretch 10.2.6+maria~stretch uptodate
 
mariadb-server-10.1:amd64/stretch 10.1.23-9+deb9u1 uptodate
 
mariadb-server-10.1:i386 not installed
 
mariadb-server-core-10.1:amd64/stretch 10.1.23-9+deb9u1 uptodate
 
mariadb-server-core-10.1:i386 not installed
 
mysql-common:all/stretch 10.2.6+maria~stretch uptodate
 
# apt-show-versions|grep mysql
 
dbconfig-mysql:all/stretch 2.0.8 uptodate
 
default-mysql-server:all/stretch 1.0.2 uptodate
 
erlang-p1-mysql:amd64/stretch 1.0.1-4 uptodate
 
erlang-p1-mysql:i386 not installed
 
libdbd-mysql-perl:amd64/stretch 4.041-2 uptodate
 
libdbd-mysql-perl:i386 not installed
 
mysql-common:all/stretch 10.2.6+maria~stretch uptodate
 
php-mysql:all/stretch 1:7.1+52+0~20170504085116.17+stretch~1.gbpd2ab76 uptodate
 
php5-mysqlnd:amd64 5.6.30+dfsg-0+deb8u1 installed: No available version in archive
 
php5-mysqlnd-ms:amd64 1.6.0-1+b1 installed: No available version in archive
 
php5.6-mysql:amd64/stretch 5.6.30-12+0~20170609083959.35+stretch~1.gbp12ca9d uptodate
 
php5.6-mysql:i386 not installed
 
php7.0-mysql:amd64/stretch 7.0.20-1+0~20170609083149.33+stretch~1.gbp2cd6b6 uptodate
 
php7.0-mysql:i386 not installed
 
php7.1-mysql:amd64/stretch 7.1.6-1+0~20170609083224.26+stretch~1.gbpa2f43a uptodate
 
php7.1-mysql:i386 not installed



 Comments   
Comment by Marko Mäkelä [ 2017-08-17 ]

jabowery, thank you for the report.
Thanks to MDEV-12289, the first step of the upgrade instructions is unnecessary. You can perform a normal shutdown without setting innodb_fast_shutdown=0. The default value is 1, which will write all dirty pages from the buffer pool to the data files and make the redo log files logically empty.

The slow shutdown would empty the change buffer and the undo logs. Without MDEV-12289, InnoDB would require the undo logs to be empty. If you were upgrading to MySQL 5.7, you would definitely need a slow shutdown.

The change buffer merge is not necessary before upgrade, because the format has not changed since I last changed it in MySQL 5.5 by introducing the delete buffering. But see also MDEV-9663; the use of the change buffer might cause corruption of secondary indexes.

Comment by Ian Gilfillan [ 2017-08-18 ]

Upgrading instructions updated

Generated at Thu Feb 08 08:02:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.