Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Duplicate
    • 5.5.36, 10.0.6
    • 5.5.37, 10.0.11
    • None
    • debian 7 64 bits

    Description

      root@production3:/var/lib/mysql# service mysql start
      [FAIL] Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . failed!

      but in fact if you wait 60 seconds, and try "mysql", the software is up.
      The issue is using a large amount of memory and memlock=1
      It seems that the OS takes time to lock the memory and the software reports a failure to start when in fact, the process has not finished yet.
      The issue may be compounded by the fact that the machine is virtual, Vmware, and it may take longer to acquire a lock on memory.
      I use large-pages=1.
      I have no idea if the memory is actually locked or not

      cat /proc/meminfo | grep -i pages
      AnonPages:      11784856 kB
      AnonHugePages:         0 kB
      HugePages_Total:   27000
      HugePages_Free:     1668
      HugePages_Rsvd:        0
      HugePages_Surp:        0
      Hugepagesize:       2048 kB

      Attachments

        Issue Links

          Activity

            I don't know what we can do here. I suppose one can always create a configuration where MariaDB will take a lot of time to start. I'm not sure it's possible in all cases to distinguish that from a hanged or locked up server (caused by some bug in MariaDB or even the Linux kernel).

            Perhaps all you need to do is to increase MYSQLD_STARTUP_TIMEOUT, like in

            root@production3:/var/lib/mysql# MYSQLD_STARTUP_TIMEOUT=60 service mysql start

            serg Sergei Golubchik added a comment - I don't know what we can do here. I suppose one can always create a configuration where MariaDB will take a lot of time to start. I'm not sure it's possible in all cases to distinguish that from a hanged or locked up server (caused by some bug in MariaDB or even the Linux kernel). Perhaps all you need to do is to increase MYSQLD_STARTUP_TIMEOUT , like in root@production3:/var/lib/mysql# MYSQLD_STARTUP_TIMEOUT=60 service mysql start
            philip_38 Philip orleans added a comment - - edited

            The solution did not work. I exported a new variable with 90, but it
            did not affect the outcome.
            What worked is this. In /etc/into.d/mysql, change this line

            for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-30}"); do
            for
            for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-90}"); do

            Maybe we should change it at least to 60?

            Also, the innodb stat tables do not get created, on a brand new
            install from the repository. It took me some time to figure out that
            the solution was to create them manually,
            The tables are called
            `innodb_index_stats`
            `innodb_table_stats`
            and this one also is missing
            `gtid_slave_pos`

            but maybe we should fix this in the product.

            philip_38 Philip orleans added a comment - - edited The solution did not work. I exported a new variable with 90, but it did not affect the outcome. What worked is this. In /etc/into.d/mysql, change this line for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-30}"); do for for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-90}"); do Maybe we should change it at least to 60? Also, the innodb stat tables do not get created, on a brand new install from the repository. It took me some time to figure out that the solution was to create them manually, The tables are called `innodb_index_stats` `innodb_table_stats` and this one also is missing `gtid_slave_pos` but maybe we should fix this in the product.

            about 30 vs. 60. vs 90 — I think an appropriate solution would be to source /etc/default/mysql and let MYSQLD_STARTUP_TIMEOUT to be set there.

            serg Sergei Golubchik added a comment - about 30 vs. 60. vs 90 — I think an appropriate solution would be to source /etc/default/mysql and let MYSQLD_STARTUP_TIMEOUT to be set there.
            frettled Jan Ingvoldstad added a comment - - edited

            Sourcing /etc/default/mysql would be very welcome, yes. See below for example code based on /etc/init.d/skeleton.

            But IMHO, this should be solved differently, by permitting an upgrade of the package without enforcing a restart of the daemon at postinst.

            As an example, one of my test cases uses a few minutes to start. This would be nice to do at a different time than package upgrade.

            The work-around is to hold the mariadb-server packages and only upgrade these manually, rather than automatically.

            NAME=mysql
             
            # ...
             
            # Read configuration variable file if it is present
            [ -r /etc/default/$NAME ] && . /etc/default/$NAME
             
            # Load the VERBOSE setting and other rcS variables
            . /lib/init/vars.sh

            frettled Jan Ingvoldstad added a comment - - edited Sourcing /etc/default/mysql would be very welcome, yes. See below for example code based on /etc/init.d/skeleton . But IMHO, this should be solved differently, by permitting an upgrade of the package without enforcing a restart of the daemon at postinst. As an example, one of my test cases uses a few minutes to start. This would be nice to do at a different time than package upgrade. The work-around is to hold the mariadb-server packages and only upgrade these manually, rather than automatically. NAME=mysql   # ...   # Read configuration variable file if it is present [ -r /etc/default/$NAME ] && . /etc/default/$NAME   # Load the VERBOSE setting and other rcS variables . /lib/init/vars.sh

            On second thought: don't load /lib/init/vars.sh, since that will break service mysql [start|stop], at least as it stands now.

            frettled Jan Ingvoldstad added a comment - On second thought: don't load /lib/init/vars.sh , since that will break service mysql [start|stop] , at least as it stands now.

            Here's a patch:

            --- 10.0.9/etc/init.d/mysql     2014-03-21 11:21:13.848030546 +0100
            +++ 10.0.10/etc/init.d/mysql    2014-03-21 11:51:11.284712299 +0100
            @@ -17,8 +17,19 @@
             set -u
             ${DEBIAN_SCRIPT_DEBUG:+ set -v -x}
             
            -test -x /usr/sbin/mysqld || exit 0
            -
            +DESC="MariaDB database server"
            +NAME=mysql
            +DAEMON=/usr/sbin/mysqld
            +
            +# Exit if the package is not installed
            +[ -x "$DAEMON" ] || exit 0
            +
            +# Read configuration variable file if it is present
            +[ -r /etc/default/$NAME ] && . /etc/default/$NAME
            +
            +# Define LSB log_* functions.
            +# Depend on lsb-base (>= 3.2-14) to ensure that this file is present
            +# and status_of_proc is working.
             . /lib/lsb/init-functions
             
             SELF=$(cd $(dirname $0); pwd -P)/$(basename $0)
            @@ -26,7 +37,7 @@
             MYADMIN="/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf"
             
             # priority can be overriden and "-s" adds output to stderr
            -ERR_LOGGER="logger -p daemon.err -t /etc/init.d/mysql -i"
            +ERR_LOGGER="logger -p daemon.err -t ${SELF} -i"
             
             # Safeguard (relative paths, core dumps..)
             cd /
            @@ -41,7 +52,7 @@
             #
             # Usage: void mysqld_get_param option
             mysqld_get_param() {
            -       /usr/sbin/mysqld --print-defaults \
            +       ${DAEMON} --print-defaults \
                            | tr " " "\n" \
                            | grep -- "--$1" \
                            | tail -n 1 \
            @@ -97,7 +108,7 @@
               'start')
                    sanity_checks;
                    # Start daemon
            -       log_daemon_msg "Starting MariaDB database server" "mysqld"
            +       log_daemon_msg "Starting ${DESC}" "mysqld"
                    if mysqld_status check_alive nowarn; then
                       log_progress_msg "already running"
                       log_end_msg 0
            @@ -131,7 +142,7 @@
                    # at least for cron, we can rely on it here, too. (although we have 
                    # to specify it explicit as e.g. sudo environments points to the normal
                    # users home and not /root)
            -       log_daemon_msg "Stopping MariaDB database server" "mysqld"
            +       log_daemon_msg "Stopping ${DESC}" "mysqld"
                    if ! mysqld_status check_dead nowarn; then
                      set +e
                      shutdown_out=`$MYADMIN shutdown 2>&1`; r=$?
            @@ -139,7 +150,7 @@
                      if [ "$r" -ne 0 ]; then
                        log_end_msg 1
                        [ "$VERBOSE" != "no" ] && log_failure_msg "Error: $shutdown_out"
            -           log_daemon_msg "Killing MariaDB database server by signal" "mysqld"
            +           log_daemon_msg "Killing ${DESC} by signal" "mysqld"
                        killall -15 mysqld
                         server_down=
                        for i in `seq 1 600`; do
            @@ -165,7 +176,7 @@
                    ;;
             
               'reload'|'force-reload')
            -       log_daemon_msg "Reloading MariaDB database server" "mysqld"
            +       log_daemon_msg "Reloading ${DESC}" "mysqld"
                    $MYADMIN reload
                    log_end_msg 0
                    ;;

            frettled Jan Ingvoldstad added a comment - Here's a patch: --- 10.0.9/etc/init.d/mysql 2014-03-21 11:21:13.848030546 +0100 +++ 10.0.10/etc/init.d/mysql 2014-03-21 11:51:11.284712299 +0100 @@ -17,8 +17,19 @@ set -u ${DEBIAN_SCRIPT_DEBUG:+ set -v -x} -test -x /usr/sbin/mysqld || exit 0 - +DESC="MariaDB database server" +NAME=mysql +DAEMON=/usr/sbin/mysqld + +# Exit if the package is not installed +[ -x "$DAEMON" ] || exit 0 + +# Read configuration variable file if it is present +[ -r /etc/default/$NAME ] && . /etc/default/$NAME + +# Define LSB log_* functions. +# Depend on lsb-base (>= 3.2-14) to ensure that this file is present +# and status_of_proc is working. . /lib/lsb/init-functions SELF=$(cd $(dirname $0); pwd -P)/$(basename $0) @@ -26,7 +37,7 @@ MYADMIN="/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf" # priority can be overriden and "-s" adds output to stderr -ERR_LOGGER="logger -p daemon.err -t /etc/init.d/mysql -i" +ERR_LOGGER="logger -p daemon.err -t ${SELF} -i" # Safeguard (relative paths, core dumps..) cd / @@ -41,7 +52,7 @@ # # Usage: void mysqld_get_param option mysqld_get_param() { - /usr/sbin/mysqld --print-defaults \ + ${DAEMON} --print-defaults \ | tr " " "\n" \ | grep -- "--$1" \ | tail -n 1 \ @@ -97,7 +108,7 @@ 'start') sanity_checks; # Start daemon - log_daemon_msg "Starting MariaDB database server" "mysqld" + log_daemon_msg "Starting ${DESC}" "mysqld" if mysqld_status check_alive nowarn; then log_progress_msg "already running" log_end_msg 0 @@ -131,7 +142,7 @@ # at least for cron, we can rely on it here, too. (although we have # to specify it explicit as e.g. sudo environments points to the normal # users home and not /root) - log_daemon_msg "Stopping MariaDB database server" "mysqld" + log_daemon_msg "Stopping ${DESC}" "mysqld" if ! mysqld_status check_dead nowarn; then set +e shutdown_out=`$MYADMIN shutdown 2>&1`; r=$? @@ -139,7 +150,7 @@ if [ "$r" -ne 0 ]; then log_end_msg 1 [ "$VERBOSE" != "no" ] && log_failure_msg "Error: $shutdown_out" - log_daemon_msg "Killing MariaDB database server by signal" "mysqld" + log_daemon_msg "Killing ${DESC} by signal" "mysqld" killall -15 mysqld server_down= for i in `seq 1 600`; do @@ -165,7 +176,7 @@ ;; 'reload'|'force-reload') - log_daemon_msg "Reloading MariaDB database server" "mysqld" + log_daemon_msg "Reloading ${DESC}" "mysqld" $MYADMIN reload log_end_msg 0 ;;

            Fixed in MDEV-5068

            serg Sergei Golubchik added a comment - Fixed in MDEV-5068
            nils.meyer Nils Meyer added a comment -

            The fix in MDEV-5068 doesn't apply to Debian.

            nils.meyer Nils Meyer added a comment - The fix in MDEV-5068 doesn't apply to Debian.

            People

              serg Sergei Golubchik
              philip_38 Philip orleans
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.