[CONJ-86] getTimestamp use client timezone and fails on Daylight Savings Time Created: 2014-03-31  Updated: 2016-08-24  Resolved: 2015-09-21

Status: Closed
Project: MariaDB Connector/J
Component/s: Other
Affects Version/s: 1.1.6
Fix Version/s: 1.3.0

Type: Bug Priority: Minor
Reporter: Anders Karlsson Assignee: Diego Dupin
Resolution: Fixed Votes: 1
Labels: None

Attachments: Text File CONJ-86_lenient_true_fix.patch     Java Source File TimestampTest.java    
Issue Links:
Relates
relates to CONJ-143 ResultSet.getTime() shouldn't convert... Closed
Sprint: Sprint connector/j 1.3.0

 Description   

It seems that the getTimestamp of the ResultSet class use the default client timestamp for time conversions. This works in most cases, but when Daylight Saving Time is used, it fails at certain times. For example if the TimeZone is Europe/Stockholm and the time is '2014-03-30 02:15:00' this is an invalid time (as the hour between 2 and 3 is "lost"). Using an explicit TimeZone works, and what seems like a workaround would be to use the UTC timezone instead of the default one when an explicit timezone is not set.



 Comments   
Comment by Lennart Schedin [ 2014-04-01 ]

JUint test cases for getTimestamp()

Comment by Lennart Schedin [ 2014-04-01 ]

I attached the file TimestampTest.java. It must be run against a Mysql/MariaDB with UTC timezone to trigger the specific exception:
TimestampTest
org.mariadb.jdbc.TimestampTest
testGetTimestampWhenDaylightSavingRemovesHour(org.mariadb.jdbc.TimestampTest)
java.sql.SQLException: Could not parse column as timestamp, was: "2014-03-30 02:15:00"
at org.mariadb.jdbc.internal.SQLExceptionMapper.getSQLException(SQLExceptionMapper.java:154)
at org.mariadb.jdbc.MySQLResultSet.getTimestamp(MySQLResultSet.java:3506)
at org.mariadb.jdbc.TimestampTest.testGetTimestampWhenDaylightSavingRemovesHour(TimestampTest.java:83)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.text.ParseException: Unparseable date: "2014-03-30 02:15:00"
at java.text.DateFormat.parse(DateFormat.java:357)
at org.mariadb.jdbc.internal.common.AbstractValueObject.getTimestamp(AbstractValueObject.java:252)
at org.mariadb.jdbc.MySQLResultSet.getTimestamp(MySQLResultSet.java:3504)
... 25 more

When using MariaDB JDBC connector my test method testGetTimestampWhenDaylightSavingRemovesHour() will fail with the above exception. When using Mysql JDBC connector it works.

I also tried the implied fix by Anders Karlsson, to add this in org.mariadb.jdbc.internal.common.AbstractValueObject.getTimestamp():
sdf.setCalendar(Calendar.getInstance(TimeZone.getTimeZone("utc")));
That prevents the exception, but makes the times wrong. I think?! Time zones are a complex issue and it is very easy to get lost. I don't think Mysql is the best reference on how it should work. Especially since they have added some JDBC driver properties over the years (for example "useJDBCCompliantTimezoneShift").

My guess is that MariaDB JDBC connector needs to use the timezone information provided by the server to perform the correct conversion.

Comment by Vladislav Vaintroub [ 2014-04-01 ]

1. In JDBC API , there is a getTimestamp() that accepts Calendar instance. Did you try that
2. There is a serverTimezone parameter, implemented in https://mariadb.atlassian.net/browse/CONJ-65

Would not that help?

Comment by Anders Karlsson [ 2014-04-02 ]

1) Using a specific Calendar for getTimestamp sure works, but that was
never the issue.
2) This also seems to work, but this parameter isn't documented in the
MariaDB JDBC docs: https://mariadb.com/kb/en/about-the-mariadb-java-client/

In general, that the default is to use the Client TimeZone though seems
like a distinctly bad idea. The data is located on the server. On the
server this is a valid date.

I'll check 2) above with the customer. Also, we need to get better at
keeping up with docs. Pointing to MySQL Connector/J docs is not what I
want to do, as this is a different connector.

/Karlsson

Anders Karlsson, Senior Sales Engineer
SkySQL | t: +46 708-608-121 | Skype: drdatabase

Comment by Lennart Schedin [ 2014-04-02 ]

The method ResultSet.getTimestamp(int columnIndex, Calendar cal) is not optimal for TIMESTAMP values. It is a legacy in the JDBC API (http://docs.oracle.com/javase/7/docs/api/java/sql/ResultSet.html#getTimestamp%28int,%20java.util.Calendar%29) :
“Retrieves the value of the designated column in the current row of this ResultSet object as a java.sql.Timestamp object in the Java programming language. This method uses the given calendar to construct an appropriate millisecond value for the timestamp if the underlying database does not store timezone information.”

What database does not store timezone information? Well maybe MySQL for the strange DATETIME datatype. But for that it is probably better to use getString() since it represents a time of day and not a point in time.

I’m not that keen on adding a JDBC parameter. There is a danger on added more and more parameters. The MySQL JDBC parameter list has gotten a bit too long…

My test case shows that the MariaDB JDBC driver is not a drop-in-replacement for MySQL since MySQL handles it. When I sniff the traffic with wireshark it looks like it is the server that has performed the conversion (but wrong!?). I’m confused. It is however clear that MariaDB and MySQL uses different code to parse the value.

A simple solution I will try is to add this to getTimestamp():
sdf.setLenient(true);
It will at least prevent the exception. I’m not sure about other effects.

Comment by Lennart Schedin [ 2014-04-03 ]

Attaching my setLenient(true)-fix. I have tested it and for my purposes it works fine. Note: I have only tested it with Swedish timezone for the Java client and UTC timezone for server. As have debugged GregorianCalendar it looks like the fix will make the Calendar class accept and correct times that probably are wrong. It corrected the time 02:15 to 03:15 (30th of March 2014).

Comment by Vladislav Vaintroub [ 2014-04-03 ]

While accepting wrong input and helpfully fixing it is inline with original MySQL philosophy, many people myself included came to think that defined behavior is better, i.e it is better to get exception and be warned than silently accept everything and produce incorrect results

lenient=false is not well defined.
serverTimezone parameter is well defined, extra parameter Calendar in getTimestamp() is well defined. If serverTimezone is not set, current behavior is well defined. Always assuming timestamp in UTC would also be well defined, unfortunately it has the potential to ran havoc on current users applications - current behavior was there since ever, and now we have the first complaint about the behavior.

IMO, everything is better compared to lenient=false. But this is just my opinion, and guys who are currently contributing to this driver will do whatever they think is better

Comment by Lennart Schedin [ 2014-06-24 ]

I agree that the ”helpfully fixing” MySQL philosophy causes some problems. From my perspective it is mostly that the MySQL JDBC need some URL properties to properly work since backwards compatibility “must” be preserved.

I suspect that the problem in this ticket is that the MySQL/MariaDB server sends the wrong timestamp (an invalid timestamp). I suspect that this is a bug in the server that has been smoothed over in the MySQL Java client by using a variant of my lenient fix.

The suggested fix in https://code.launchpad.net/~massimo-siani/mariadb-java-client/CONJ-86 will only make any impact for the ResultSet.getTimestamp(int, Calendar) method? My testcase (in the CONJ-86_lenient_true_fix.patch does) will have no benefit of the fix since it uses the ResultSet.getTimestamp(int) method?

Comment by Massimo Siani (Inactive) [ 2014-06-24 ]

Hi Lennart,
the suggested fix on launchpad will only affect the ResultSet.getTimestamp(int) method. I believe that if you set a calendar, there should be no guess if the timestamp cannot be parsed.
So, whenever the calendar has not been set and the timestamp cannot be parsed, we'll guess that the latter is in UTC. Why not always assuming UTC? There two reasons, in my opinion. First, backward compatibility. Secondly, it is reasonable that, if you do not explicitly set the parameter, you may be happy with the default one (server timezone equals to the client one) and I do not want to override that.

I'd like to tell you that your test case has been included in the code. Thank you.

Any comment is very welcome.

Comment by Lennart Schedin [ 2014-09-11 ]

I’ve had the gut feeling that the suggested fix in https://code.launchpad.net/~massimo-siani/mariadb-java-client/CONJ-86 did not work in all cases. I think I have found such a case now:

TimezoneDaylightSavingTimeTest.java

package org.mariadb.jdbc;
 
import static org.junit.Assert.*;
 
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Timestamp;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;
import java.util.Locale;
import java.util.TimeZone;
 
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
 
public class TimezoneDaylightSavingTimeTest extends BaseTest {
 
    private Locale previousFormatLocale;
    private TimeZone previousTimeZone;
    private TimeZone utcTimeZone;
    private SimpleDateFormat utcDateFormatISO8601;
    private SimpleDateFormat utcDateFormatSimple;
    private TimeZone istanbulTimeZone;
 
    @Before
    public void setUp() throws SQLException {
        //Save the previous FORMAT locate so we can restore it later
        previousFormatLocale = Locale.getDefault();
        //Save the previous timezone so we can restore it later
        previousTimeZone = TimeZone.getDefault();
        
        //I have tried to represent all times written in the code in the UTC time zone
        utcTimeZone = TimeZone.getTimeZone("utc");
        
        //For this test case I choose the Istanbul timezone because it show the fault in a good way.
        istanbulTimeZone = TimeZone.getTimeZone("Europe/Istanbul");
        TimeZone.setDefault(istanbulTimeZone);
        
        //Use a date formatter for UTC timezone in ISO 8601 so users in different
        //timezones can compare the test results easier.
        utcDateFormatISO8601 = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ");
        utcDateFormatISO8601.setTimeZone(utcTimeZone);
        
        utcDateFormatSimple = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        utcDateFormatSimple.setTimeZone(utcTimeZone);
        
        Statement statement = connection.createStatement();
        statement.execute("drop table if exists timestamptest");
        statement.execute("create table timestamptest (id int not null primary key auto_increment, tm timestamp)");
    }
    
    @After
    public void tearDown() {
        //Reset the FORMAT locate so other test cases are not disturbed.
        Locale.setDefault(previousFormatLocale);
        //Reset the timezone so so other test cases are not disturbed.
        TimeZone.setDefault(previousTimeZone);
    }
    
    
    /**
     * This method is designed to correctly insert a timestamp in the database. It manually forces
     * the session to be in UTC and transfers the timestamp as a String to minimize the impact the
     * JDBC code may have over it.
     */
    private void insertTimestampInWithUTCSessionTimeZone(Calendar timestamp) throws SQLException {
        //Totally reset of the connection to be sure everything is clean
        connection.close();
        
        before(); //Force a new Connection (may be replaced by a getConnection() method if CONJ-112 is implemented
        Statement statement = connection.createStatement();
 
        setSessionTimeZone(connection, "+00:00");
        
        //Use the "wrong" way to insert a timestamp: as a string. This is done to avoid possible bugs
        //in the PreparedStatement.setTimestamp(int parameterIndex, Timestamp x) method and to make
        //sure the value is a correct UTC time.
        statement.execute("insert into timestamptest values(null, '" 
                        + utcDateFormatSimple.format(new Date(timestamp.getTimeInMillis())) + "')");
        
        //Totally reset of the connection to be sure everything is clean
        statement.close();
        connection.close();
        before();
    }
    
    @Test
    public void testGetTimestampWhenDaylightSavingRemovesHour() throws SQLException {
        Calendar _0015 = Calendar.getInstance(TimeZone.getTimeZone("utc"));
        _0015.clear();
        _0015.set(2014, 2, 30, 0, 15, 0);
        String _0015String =  utcDateFormatISO8601.format(new Date(_0015.getTimeInMillis()));
        
        insertTimestampInWithUTCSessionTimeZone(_0015);
 
        Statement statement = connection.createStatement();
        setSessionTimeZone(connection, istanbulTimeZone);
 
        //Verify with ResultSet.getTimestamp() that it is correct
        ResultSet rs = statement.executeQuery("select * from timestamptest");
 
        assertTrue(rs.next());
        System.out.println(rs.getString("tm"));
        Timestamp timestamp = rs.getTimestamp("tm");
        assertEquals(_0015String, utcDateFormatISO8601.format(timestamp));
    }
    
    @Test
    public void testGetTimestampWithoutDaylightSavingIssue() throws SQLException {
        Calendar _0115 = Calendar.getInstance(TimeZone.getTimeZone("utc"));
        _0115.clear();
        _0115.set(2014, 2, 30, 1, 15, 0);
        String _0115String =  utcDateFormatISO8601.format(new Date(_0115.getTimeInMillis()));
        
        insertTimestampInWithUTCSessionTimeZone(_0115);
 
        Statement statement = connection.createStatement();
        setSessionTimeZone(connection, istanbulTimeZone);
 
        //Verify with ResultSet.getTimestamp() that it is correct
        ResultSet rs = statement.executeQuery("select * from timestamptest");
 
        assertTrue(rs.next());
        System.out.println(rs.getString("tm"));
        Timestamp timestamp = rs.getTimestamp("tm");
        assertEquals(_0115String, utcDateFormatISO8601.format(timestamp));
    }
    
    /**
     * Return the session timezone in a "+02:00" or "-05:00" format. 
     */
    private String getSessionTimeZone() throws SQLException {
        Statement statement = connection.createStatement();
        ResultSet resultSet = statement.executeQuery(
            "select CONVERT(timediff(now(),convert_tz(now(),@@session.time_zone,'+00:00')), CHAR)");
        
        resultSet.next();
        
        String timeZoneString = resultSet.getString(1);
        
        timeZoneString = timeZoneString.replaceAll(":00$", "");
        
        if (!timeZoneString.startsWith("-")) {
            timeZoneString = "+" + timeZoneString;
        }
        statement.close();
        
        return timeZoneString;
    }
    
    private void setSessionTimeZone(Connection connection, String timeZone) throws SQLException {
        Statement statement = connection.createStatement();
        statement.execute("set @@session.time_zone = '" + timeZone + "'");
        statement.close();
    }
 
    private void setSessionTimeZone(Connection connection, TimeZone timeZone) throws SQLException {
        int offsetInMs = timeZone.getOffset(System.currentTimeMillis());
        int offsetInHours = offsetInMs/3600/1000;
        
        boolean leadingZero = false;
        if (Math.abs(offsetInHours) < 10) {
            //Add a leading 0
             leadingZero = true;
        }
        
        boolean positive = true;
        if (offsetInHours < 0) {
            positive = false; 
        }
        
        String offsetString =  (positive ? "+" : "-") +
                        (leadingZero ? "0" : "") +
                        Math.abs(offsetInHours) +
                        ":00";
 
        setSessionTimeZone(connection, offsetString);
    }
    
}
 
 

The test case testGetTimestampWithoutDaylightSavingIssue() uses a time without any daylight saving effects and will pass. The testGetTimestampWhenDaylightSavingRemovesHour() is very similar, it only differs on the time but will fail. The time is a daylight-saving-problem-hour. I have tried to make the Junit tests to behave the same regardless of with time zone the server of client uses.

I don’t have any proposed good fix for this problem. A fix should probably be synced with CONJ-109.

Comment by Rasmus Johansson (Inactive) [ 2015-06-05 ]

Please verify if this bug still exists (I think it does) and try to fix

Comment by Diego Dupin [ 2015-09-21 ]

testing if the problem still exist with :

    TimeZone parisTimeZone;
 
    @Before
    public void init() throws SQLException {
        parisTimeZone = TimeZone.getTimeZone("Europe/Paris");
        TimeZone.setDefault(parisTimeZone);
    }
 
    @Test
    public void testDayLight() throws SQLException {
        setConnection("&serverTimezone=Europe/Paris");
        Statement st = connection.createStatement();
        st.executeQuery("DROP TABLE IF EXISTS daylight");
        st.executeQuery("CREATE TABLE daylight(id int, tt TIMESTAMP(6))");
 
        Calendar quarterBeforeChangingHour = Calendar.getInstance(TimeZone.getTimeZone("utc"));
        quarterBeforeChangingHour.clear();
        quarterBeforeChangingHour.set(2015, 2, 29, 0, 45, 0);
        int offsetBefore = parisTimeZone.getOffset(quarterBeforeChangingHour.getTimeInMillis());
        Assert.assertEquals(offsetBefore, 3600000);
 
        Calendar quarterAfterChangingHour = Calendar.getInstance(TimeZone.getTimeZone("utc"));
        quarterAfterChangingHour.clear();
        quarterAfterChangingHour.set(2015, 2, 29, 1, 15, 0);
        int offsetAfter = parisTimeZone.getOffset(quarterAfterChangingHour.getTimeInMillis());
        Assert.assertEquals(offsetAfter, 7200000);
 
 
        PreparedStatement pst = connection.prepareStatement("INSERT INTO daylight VALUES (?, ?)");
        pst.setInt(1, 1);
        pst.setTimestamp(2, new Timestamp(quarterBeforeChangingHour.getTimeInMillis()));
        pst.addBatch();
        pst.setInt(1, 2);
        pst.setTimestamp(2, new Timestamp(quarterAfterChangingHour.getTimeInMillis()));
        pst.addBatch();
        pst.setInt(1, 3);
        pst.setString(2, "2015-03-29 02:15:00");
        pst.addBatch();
        try {
            pst.executeBatch();
        } catch (SQLException e) {
            assertTrue(e.getMessage().startsWith("Incorrect datetime value"));
        }
 
        ResultSet rs = st.executeQuery("SELECT * from daylight");
        rs.next();
        assertEquals(rs.getTimestamp(2).getTime(), quarterBeforeChangingHour.getTimeInMillis());
        rs.next();
        assertEquals(rs.getTimestamp(2).getTime(), quarterAfterChangingHour.getTimeInMillis());
        assertFalse(rs.next());
    }
 
@Test
    public void testDayLightWithClientTimeZoneDifferent() throws SQLException {
        setConnection("&serverTimezone=UTC");
        Statement st = connection.createStatement();
        st.executeQuery("DROP TABLE IF EXISTS daylight");
        st.executeQuery("CREATE TABLE daylight(id int, tt TIMESTAMP(6))");
 
        Calendar quarterBeforeChangingHour = Calendar.getInstance(TimeZone.getTimeZone("utc"));
        quarterBeforeChangingHour.clear();
        quarterBeforeChangingHour.set(2015, 2, 29, 0, 45, 0);
        int offsetBefore = parisTimeZone.getOffset(quarterBeforeChangingHour.getTimeInMillis());
        Assert.assertEquals(offsetBefore, 3600000);
 
        Calendar quarterAfterChangingHour = Calendar.getInstance(TimeZone.getTimeZone("utc"));
        quarterAfterChangingHour.clear();
        quarterAfterChangingHour.set(2015, 2, 29, 1, 15, 0);
        int offsetAfter = parisTimeZone.getOffset(quarterAfterChangingHour.getTimeInMillis());
        Assert.assertEquals(offsetAfter, 7200000);
 
        PreparedStatement pst = connection.prepareStatement("INSERT INTO daylight VALUES (?, ?)");
        pst.setInt(1, 1);
        pst.setTimestamp(2, new Timestamp(quarterBeforeChangingHour.getTimeInMillis()));
        pst.addBatch();
        pst.setInt(1, 2);
        pst.setTimestamp(2, new Timestamp(quarterAfterChangingHour.getTimeInMillis()));
        pst.addBatch();
        pst.executeBatch();
 
        //test with text protocol
        ResultSet rs = st.executeQuery("SELECT * from daylight");
        rs.next();
        Timestamp tBefore = rs.getTimestamp(2);
        assertEquals(tBefore.toString(), "2015-03-29 01:45:00.0");
        rs.next();
        Timestamp tAfter = rs.getTimestamp(2);
        assertEquals(tAfter.toString(), "2015-03-29 03:15:00.0");
 
        //test with binary protocol
        pst = connection.prepareStatement("SELECT * from daylight where id = ?");
        pst.setInt(1, 1);
        pst.addBatch();
        rs = pst.executeQuery();
        rs.next();
        tBefore = rs.getTimestamp(2);
        assertEquals(tBefore.toString(), "2015-03-29 01:45:00.0");
 
        pst.setInt(1, 2);
        pst.addBatch();
        rs = pst.executeQuery();
        rs.next();
        tAfter = rs.getTimestamp(2);
        assertEquals(tAfter.toString(), "2015-03-29 03:15:00.0");
    }
 

and all is working well.
closing the issue.

Generated at Thu Feb 08 03:13:04 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.