Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9039

Can't upgrade MariaDB to to 10.1.8 version from 10.0.21

Details

    Description

      After removing old packages and install new.
      Server couldn't run again.

      [root@vmi25208 mysql]# systemctl status mariadb.service -l
      ● mariadb.service - MariaDB database server
         Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
        Drop-In: /etc/systemd/system/mariadb.service.d
                 └─migrated-from-my.cnf-settings.conf
         Active: failed (Result: exit-code) since Пт 2015-10-30 03:27:11 YEKT; 3min 12s ago
        Process: 28986 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS (code=exited, status=1/FAILURE)
       Main PID: 28986 (code=exited, status=1/FAILURE)
         Status: "MariaDB server is down"
       
      окт 30 03:27:09 vmi25208.contabo.net mysqld[28986]: 2015-10-30  3:27:09 139992865175296 [Note] InnoDB: Dumping buffer pool(s) not yet started
      окт 30 03:27:09 vmi25208.contabo.net mysqld[28986]: 2015-10-30  3:27:09 139993609361600 [Note] Recovering after a crash using tc.log
      окт 30 03:27:09 vmi25208.contabo.net mysqld[28986]: 2015-10-30  3:27:09 139993609361600 [ERROR] Recovery failed! You must enable exactly 2 storage engines that support two-phase commit protocol
      окт 30 03:27:09 vmi25208.contabo.net mysqld[28986]: 2015-10-30  3:27:09 139993609361600 [ERROR] Crash recovery failed. Either correct the problem (if it's, for example, out of memory error) and restart, or delete tc log and start mysqld with --tc-heuristic-recover={commit|rollback}
      окт 30 03:27:09 vmi25208.contabo.net mysqld[28986]: 2015-10-30  3:27:09 139993609361600 [ERROR] Can't init tc log
      окт 30 03:27:09 vmi25208.contabo.net mysqld[28986]: 2015-10-30  3:27:09 139993609361600 [ERROR] Aborting
      окт 30 03:27:11 vmi25208.contabo.net systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
      окт 30 03:27:11 vmi25208.contabo.net systemd[1]: Failed to start MariaDB database server.
      окт 30 03:27:11 vmi25208.contabo.net systemd[1]: Unit mariadb.service entered failed state.
      окт 30 03:27:11 vmi25208.contabo.net systemd[1]: mariadb.service failed.
       

      Attachments

        Issue Links

          Activity

            mikhail Mikhail Gavrilov created issue -

            Hi Mikhail,

            You've hit the bug MDEV-8794.
            To start the server, it should be enough just to remove tc.log from your datadir.

            elenst Elena Stepanova added a comment - Hi Mikhail, You've hit the bug MDEV-8794 . To start the server, it should be enough just to remove tc.log from your datadir.

            mikhail,

            I would like to see which engines caused the problem in your upgrade. Could you please quote or attach the whole error log, and also output of SHOW ENGINES after you start the server?

            elenst Elena Stepanova added a comment - mikhail , I would like to see which engines caused the problem in your upgrade. Could you please quote or attach the whole error log, and also output of SHOW ENGINES after you start the server?

            Could you please quote or attach the whole error log

            I see that new server not add anything to /var/log/mariadb/mariadb.log

            151030  3:07:13 [Note] /usr/libexec/mysqld: ready for connections.
             
            <- here must be new server logging
             
            Version: '10.0.21-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
            151030 03:36:29 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

            and also output of SHOW ENGINES after you start the server?

            How do it? New server not still not run ...
            I can showing that the output of the previous server.

            Engine              Support  Comment                                                                     Transactions  XA      Savepoints  
            ------------------  -------  --------------------------------------------------------------------------  ------------  ------  ------------
            MyISAM              YES      MyISAM storage engine                                                       NO            NO      NO          
            CSV                 YES      CSV storage engine                                                          NO            NO      NO          
            MRG_MyISAM          YES      Collection of identical MyISAM tables                                       NO            NO      NO          
            BLACKHOLE           YES      /dev/null storage engine (anything you write to it disappears)              NO            NO      NO          
            FEDERATED           YES      FederatedX pluggable storage engine                                         YES           NO      YES         
            SEQUENCE            YES      Generated tables filled with sequential values                              YES           YES     YES         
            PERFORMANCE_SCHEMA  YES      Performance Schema                                                          NO            NO      NO          
            ARCHIVE             YES      Archive storage engine                                                      NO            NO      NO          
            MEMORY              YES      Hash based, stored in memory, useful for temporary tables                   NO            NO      NO          
            InnoDB              DEFAULT  Percona-XtraDB, Supports transactions, row-level locking, and foreign keys  YES           YES     YES         
            Aria                YES      Crash-safe tables with MyISAM heritage                                      NO            NO      NO      

            mikhail Mikhail Gavrilov added a comment - Could you please quote or attach the whole error log I see that new server not add anything to /var/log/mariadb/mariadb.log 151030 3:07:13 [Note] /usr/libexec/mysqld: ready for connections.   <- here must be new server logging   Version: '10.0.21-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server 151030 03:36:29 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql and also output of SHOW ENGINES after you start the server? How do it? New server not still not run ... I can showing that the output of the previous server. Engine Support Comment Transactions XA Savepoints ------------------ ------- -------------------------------------------------------------------------- ------------ ------ ------------ MyISAM YES MyISAM storage engine NO NO NO CSV YES CSV storage engine NO NO NO MRG_MyISAM YES Collection of identical MyISAM tables NO NO NO BLACKHOLE YES /dev/null storage engine (anything you write to it disappears) NO NO NO FEDERATED YES FederatedX pluggable storage engine YES NO YES SEQUENCE YES Generated tables filled with sequential values YES YES YES PERFORMANCE_SCHEMA YES Performance Schema NO NO NO ARCHIVE YES Archive storage engine NO NO NO MEMORY YES Hash based, stored in memory, useful for temporary tables NO NO NO InnoDB DEFAULT Percona-XtraDB, Supports transactions, row-level locking, and foreign keys YES YES YES Aria YES Crash-safe tables with MyISAM heritage NO NO NO
            mikhail Mikhail Gavrilov made changes -
            Field Original Value New Value
            Attachment mariadb.log [ 40310 ]

            mikhail,
            Thanks.

            Did you see my first comment?
            You've hit the bug MDEV-8794.
            To start the server, it should be enough just to remove tc.log from your datadir.

            elenst Elena Stepanova added a comment - mikhail , Thanks. Did you see my first comment? You've hit the bug MDEV-8794 . To start the server, it should be enough just to remove tc.log from your datadir.

            To start the server, it should be enough just to remove tc.log from your datadir.

            Very strange I make second attempt without removing tc.log and server start fine.
            Gremlins?

            # dnf remove mariadb*
            Dependencies resolved.
            ===================================================================================================================================================================================================================
             Package                                               Arch                                          Version                                                 Repository                                       Size
            ===================================================================================================================================================================================================================
            Removing:
             mariadb                                               x86_64                                        1:10.0.21-1.fc22                                        @updates                                         37 M
             mariadb-common                                        x86_64                                        1:10.0.21-1.fc22                                        @updates                                        328 k
             mariadb-config                                        x86_64                                        1:10.0.21-1.fc22                                        @updates                                        577  
             mariadb-errmsg                                        x86_64                                        1:10.0.21-1.fc22                                        @updates                                        1.9 M
             mariadb-libs                                          x86_64                                        1:10.0.21-1.fc22                                        @updates                                        3.4 M
             mariadb-server                                        x86_64                                        1:10.0.21-1.fc22                                        @updates                                         86 M
             perl-DBD-MySQL                                        x86_64                                        4.031-1.fc22                                            @fedora                                         347 k
             
            Transaction Summary
            ===================================================================================================================================================================================================================
            Remove  7 Packages
             
            Installed size: 130 M
            Is this ok [y/N]: y
            Running transaction check
            Transaction check succeeded.
            Running transaction test
            Transaction test succeeded.
            Running transaction
              Erasing     : mariadb-server-1:10.0.21-1.fc22.x86_64                                                                                                                                                         1/7 
              Erasing     : mariadb-errmsg-1:10.0.21-1.fc22.x86_64                                                                                                                                                         2/7 
              Erasing     : mariadb-1:10.0.21-1.fc22.x86_64                                                                                                                                                                3/7 
              Erasing     : perl-DBD-MySQL-4.031-1.fc22.x86_64                                                                                                                                                             4/7 
              Erasing     : mariadb-libs-1:10.0.21-1.fc22.x86_64                                                                                                                                                           5/7 
              Erasing     : mariadb-common-1:10.0.21-1.fc22.x86_64                                                                                                                                                         6/7 
              Erasing     : mariadb-config-1:10.0.21-1.fc22.x86_64                                                                                                                                                         7/7 
              Verifying   : mariadb-1:10.0.21-1.fc22.x86_64                                                                                                                                                                1/7 
              Verifying   : mariadb-common-1:10.0.21-1.fc22.x86_64                                                                                                                                                         2/7 
              Verifying   : mariadb-config-1:10.0.21-1.fc22.x86_64                                                                                                                                                         3/7 
              Verifying   : mariadb-errmsg-1:10.0.21-1.fc22.x86_64                                                                                                                                                         4/7 
              Verifying   : mariadb-libs-1:10.0.21-1.fc22.x86_64                                                                                                                                                           5/7 
              Verifying   : mariadb-server-1:10.0.21-1.fc22.x86_64                                                                                                                                                         6/7 
              Verifying   : perl-DBD-MySQL-4.031-1.fc22.x86_64                                                                                                                                                             7/7 
             
            Removed:
              mariadb.x86_64 1:10.0.21-1.fc22           mariadb-common.x86_64 1:10.0.21-1.fc22    mariadb-config.x86_64 1:10.0.21-1.fc22    mariadb-errmsg.x86_64 1:10.0.21-1.fc22    mariadb-libs.x86_64 1:10.0.21-1.fc22   
              mariadb-server.x86_64 1:10.0.21-1.fc22    perl-DBD-MySQL.x86_64 4.031-1.fc22       
             
            Complete!
             
            [root@vmi25208 yum.repos.d]# dnf install MariaDB-server
            Last metadata expiration check performed 0:43:53 ago on Fri Oct 30 17:30:22 2015.
            Dependencies resolved.
            ===================================================================================================================================================================================================================
             Package                                                Arch                                           Version                                               Repository                                       Size
            ===================================================================================================================================================================================================================
            Installing:
             MariaDB-client                                         x86_64                                         10.1.8-1.fc21                                         mariadb                                          34 M
             MariaDB-common                                         x86_64                                         10.1.8-1.fc21                                         mariadb                                          47 k
             MariaDB-server                                         x86_64                                         10.1.8-1.fc21                                         mariadb                                          87 M
             jemalloc                                               x86_64                                         3.6.0-8.fc22                                          fedora                                          116 k
             
            Transaction Summary
            ===================================================================================================================================================================================================================
            Install  4 Packages
             
            Total download size: 121 M
            Installed size: 529 M
            Is this ok [y/N]: y
            Downloading Packages:
            (1/4): MariaDB-10.1.8-fedora21-x86_64-common.rpm                                                                                                                                    50 kB/s |  47 kB     00:00    
            (2/4): jemalloc-3.6.0-8.fc22.x86_64.rpm                                                                                                                                            279 kB/s | 116 kB     00:00    
            (3/4): MariaDB-10.1.8-fedora21-x86_64-client.rpm                                                                                                                                   3.6 MB/s |  34 MB     00:09    
            (4/4): MariaDB-10.1.8-fedora21-x86_64-server.rpm                                                                                                                                   2.9 MB/s |  87 MB     00:30    
            -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
            Total                                                                                                                                                                              3.8 MB/s | 121 MB     00:31     
            Running transaction check
            Transaction check succeeded.
            Running transaction test
            Transaction test succeeded.
            Running transaction
              Installing  : MariaDB-common-10.1.8-1.fc21.x86_64                                                                                                                                                            1/4 
              Installing  : MariaDB-client-10.1.8-1.fc21.x86_64                                                                                                                                                            2/4 
              Installing  : jemalloc-3.6.0-8.fc22.x86_64                                                                                                                                                                   3/4 
              Installing  : MariaDB-server-10.1.8-1.fc21.x86_64                                                                                                                                                            4/4 
              Verifying   : MariaDB-server-10.1.8-1.fc21.x86_64                                                                                                                                                            1/4 
              Verifying   : MariaDB-client-10.1.8-1.fc21.x86_64                                                                                                                                                            2/4 
              Verifying   : MariaDB-common-10.1.8-1.fc21.x86_64                                                                                                                                                            3/4 
              Verifying   : jemalloc-3.6.0-8.fc22.x86_64                                                                                                                                                                   4/4 
             
            Installed:
              MariaDB-client.x86_64 10.1.8-1.fc21                   MariaDB-common.x86_64 10.1.8-1.fc21                   MariaDB-server.x86_64 10.1.8-1.fc21                   jemalloc.x86_64 3.6.0-8.fc22                  
             
            Complete!
            [root@vmi25208 yum.repos.d]# systemctl start mariadb.service
             

            mikhail Mikhail Gavrilov added a comment - To start the server, it should be enough just to remove tc.log from your datadir. Very strange I make second attempt without removing tc.log and server start fine. Gremlins? # dnf remove mariadb* Dependencies resolved. =================================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================================== Removing: mariadb x86_64 1:10.0.21-1.fc22 @updates 37 M mariadb-common x86_64 1:10.0.21-1.fc22 @updates 328 k mariadb-config x86_64 1:10.0.21-1.fc22 @updates 577 mariadb-errmsg x86_64 1:10.0.21-1.fc22 @updates 1.9 M mariadb-libs x86_64 1:10.0.21-1.fc22 @updates 3.4 M mariadb-server x86_64 1:10.0.21-1.fc22 @updates 86 M perl-DBD-MySQL x86_64 4.031-1.fc22 @fedora 347 k   Transaction Summary =================================================================================================================================================================================================================== Remove 7 Packages   Installed size: 130 M Is this ok [y/N]: y Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Erasing : mariadb-server-1:10.0.21-1.fc22.x86_64 1/7 Erasing : mariadb-errmsg-1:10.0.21-1.fc22.x86_64 2/7 Erasing : mariadb-1:10.0.21-1.fc22.x86_64 3/7 Erasing : perl-DBD-MySQL-4.031-1.fc22.x86_64 4/7 Erasing : mariadb-libs-1:10.0.21-1.fc22.x86_64 5/7 Erasing : mariadb-common-1:10.0.21-1.fc22.x86_64 6/7 Erasing : mariadb-config-1:10.0.21-1.fc22.x86_64 7/7 Verifying : mariadb-1:10.0.21-1.fc22.x86_64 1/7 Verifying : mariadb-common-1:10.0.21-1.fc22.x86_64 2/7 Verifying : mariadb-config-1:10.0.21-1.fc22.x86_64 3/7 Verifying : mariadb-errmsg-1:10.0.21-1.fc22.x86_64 4/7 Verifying : mariadb-libs-1:10.0.21-1.fc22.x86_64 5/7 Verifying : mariadb-server-1:10.0.21-1.fc22.x86_64 6/7 Verifying : perl-DBD-MySQL-4.031-1.fc22.x86_64 7/7   Removed: mariadb.x86_64 1:10.0.21-1.fc22 mariadb-common.x86_64 1:10.0.21-1.fc22 mariadb-config.x86_64 1:10.0.21-1.fc22 mariadb-errmsg.x86_64 1:10.0.21-1.fc22 mariadb-libs.x86_64 1:10.0.21-1.fc22 mariadb-server.x86_64 1:10.0.21-1.fc22 perl-DBD-MySQL.x86_64 4.031-1.fc22   Complete!   [root@vmi25208 yum.repos.d]# dnf install MariaDB-server Last metadata expiration check performed 0:43:53 ago on Fri Oct 30 17:30:22 2015. Dependencies resolved. =================================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================================== Installing: MariaDB-client x86_64 10.1.8-1.fc21 mariadb 34 M MariaDB-common x86_64 10.1.8-1.fc21 mariadb 47 k MariaDB-server x86_64 10.1.8-1.fc21 mariadb 87 M jemalloc x86_64 3.6.0-8.fc22 fedora 116 k   Transaction Summary =================================================================================================================================================================================================================== Install 4 Packages   Total download size: 121 M Installed size: 529 M Is this ok [y/N]: y Downloading Packages: (1/4): MariaDB-10.1.8-fedora21-x86_64-common.rpm 50 kB/s | 47 kB 00:00 (2/4): jemalloc-3.6.0-8.fc22.x86_64.rpm 279 kB/s | 116 kB 00:00 (3/4): MariaDB-10.1.8-fedora21-x86_64-client.rpm 3.6 MB/s | 34 MB 00:09 (4/4): MariaDB-10.1.8-fedora21-x86_64-server.rpm 2.9 MB/s | 87 MB 00:30 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 3.8 MB/s | 121 MB 00:31 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Installing : MariaDB-common-10.1.8-1.fc21.x86_64 1/4 Installing : MariaDB-client-10.1.8-1.fc21.x86_64 2/4 Installing : jemalloc-3.6.0-8.fc22.x86_64 3/4 Installing : MariaDB-server-10.1.8-1.fc21.x86_64 4/4 Verifying : MariaDB-server-10.1.8-1.fc21.x86_64 1/4 Verifying : MariaDB-client-10.1.8-1.fc21.x86_64 2/4 Verifying : MariaDB-common-10.1.8-1.fc21.x86_64 3/4 Verifying : jemalloc-3.6.0-8.fc22.x86_64 4/4   Installed: MariaDB-client.x86_64 10.1.8-1.fc21 MariaDB-common.x86_64 10.1.8-1.fc21 MariaDB-server.x86_64 10.1.8-1.fc21 jemalloc.x86_64 3.6.0-8.fc22   Complete! [root@vmi25208 yum.repos.d]# systemctl start mariadb.service  

            Well, it's a little surprising, but rather fortunate under the circumstances.

            The problem (MDEV-8794) appears when three events coincide:

            • you had at least two XA-capable engines enabled (so that tc.log was created);
            • your server was not shut down gracefully;
            • you attempt to start 10.1 server with a different number of XA-capable engines.

            It specifically affects upgrade to 10.1, because in 10.1 the SEQUENCE engine is enabled by default, so it changes the number of engines automatically.

            There are two solutions when it happens: either remove tc.log, or get the server start and shut down gracefully. As soon as either of these happens, it's fine to proceed.
            I suppose it's possible that at some point during re-installation server was able to start up and shut down properly, which solved the problem.

            elenst Elena Stepanova added a comment - Well, it's a little surprising, but rather fortunate under the circumstances. The problem ( MDEV-8794 ) appears when three events coincide: you had at least two XA-capable engines enabled (so that tc.log was created); your server was not shut down gracefully; you attempt to start 10.1 server with a different number of XA-capable engines. It specifically affects upgrade to 10.1, because in 10.1 the SEQUENCE engine is enabled by default, so it changes the number of engines automatically. There are two solutions when it happens: either remove tc.log, or get the server start and shut down gracefully. As soon as either of these happens, it's fine to proceed. I suppose it's possible that at some point during re-installation server was able to start up and shut down properly, which solved the problem.
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            Fix Version/s 10.1 [ 16100 ]
            Affects Version/s 10.1 [ 16100 ]
            elenst Elena Stepanova made changes -
            Status Open [ 1 ] Confirmed [ 10101 ]
            elenst Elena Stepanova made changes -
            Assignee Elena Stepanova [ elenst ]
            serg Sergei Golubchik made changes -
            Assignee Elena Stepanova [ elenst ] Sergei Golubchik [ serg ]
            serg Sergei Golubchik made changes -
            Status Confirmed [ 10101 ] In Progress [ 3 ]
            serg Sergei Golubchik made changes -
            Component/s Storage Engine - Sequence [ 13000 ]
            Component/s XA [ 12400 ]
            Fix Version/s 10.1.9 [ 20301 ]
            Fix Version/s 10.1 [ 16100 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 72342 ] MariaDB v4 [ 149776 ]

            People

              serg Sergei Golubchik
              mikhail Mikhail Gavrilov
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.