Uploaded image for project: 'MariaDB Connector/J'
  1. MariaDB Connector/J
  2. CONJ-826

After roughly 24 hours of moderate DB load, "No such column" errors on Aurora

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Cannot Reproduce
    • 2.6.2
    • N/A
    • aurora
    • None

    Description

      We are facing a issue that has only cropped up recently in correlation with increased traffic.

      About 24 hours after our server restart, we begin to receive a large number of "No such column" exceptions

      {{Sep 4 14:07:41 ip-10-0-0-254 FreshWebService[15887]: [dw-1143 - GET /fresh/web/users/clockedIn] com.fresh.web.FreshExceptionMapper org.hibernate.exception.SQLGrammarException: error executing work
      at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:63)
      at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
      at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
      at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
      at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:319)
      at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2333)
      at org.hibernate.internal.SessionImpl.doReturningWork(SessionImpl.java:2329)
      at com.fresh.dao.TimeCardDAO.getOpenTimeCards(TimeCardDAO.java:155)
      at com.fresh.resources.UserResource.getClockedInUsers(UserResource.java:878)
      at sun.reflect.GeneratedMethodAccessor290.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
      at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
      at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
      at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
      at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
      at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
      at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
      at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
      at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
      at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
      at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
      at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
      at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
      at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
      at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
      at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
      at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
      at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
      at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
      at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
      at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
      at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
      at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
      at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:34)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:50)
      at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:44)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:241)
      at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
      at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
      at io.dropwizard.jetty.BiDiGzipHandler.handle(BiDiGzipHandler.java:69)
      at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56)
      at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      at org.eclipse.jetty.server.Server.handle(Server.java:531)
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
      at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
      at java.lang.Thread.run(Thread.java:748)#012Caused by: java.sql.SQLSyntaxErrorException: No such column: 'time_card_id'. 'time_card_id' must be in [minimum_2_97_0_, this_.recommen3_97_0_, system_status_id, system_status.minimum_terminal_build_number, system_s1_97_0_, minimum_terminal_build_number, recommen3_97_0_, system_status.recommended_terminal_build_number, system_status.system_status_id, recommended_terminal_build_number, this_.system_s1_97_0_, this_.minimum_2_97_0_]
      at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:62)
      at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:187)
      at org.mariadb.jdbc.internal.com.read.dao.ColumnLabelIndexer.getIndex(ColumnLabelIndexer.java:137)
      at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.findColumn(SelectResultSet.java:1318)
      at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.getLong(SelectResultSet.java:995)
      at com.fresh.dao.TimeCardDAO$1.execute(TimeCardDAO.java:173)
      at com.fresh.dao.TimeCardDAO$1.execute(TimeCardDAO.java:155)
      at org.hibernate.jdbc.WorkExecutor.executeReturningWork(WorkExecutor.java:55)
      at org.hibernate.internal.SessionImpl$2.accept(SessionImpl.java:2326)
      at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:314)
      ... 69 more}}

      Upon some investigation, it seems some of the queries the Aurora database is responding, to not correlate to the actual query sent. For example, the query requesting "time_card_id" is defined as

      {{ String sqlString = "select tc.time_card_id as time_card_id "
      + "from time_card tc "
      + "left join company c on c.company_id = tc.company_id "
      + "where tc.company_id = ? and tc.end_time is null";}}

      and yet the column array the exception is indicating "time_card_id" has to be in, seems to be the columns of a completely different table corresponding to a completely different query. The above query works flawlessly for the first 24 hours.

      Our connection string is

      jdbc:mariadb:aurora://hostname/dbname
      we have also tried

      jdbc:mariadb:aurora://hostname/dbname?useBatchMultiSend=false&usePipelineAuth=false
      as well as

      jdbc:mysql://hostname/dbname?useBatchMultiSend=false&usePipelineAuth=false

      Attachments

        Issue Links

          Activity

            People

              diego dupin Diego Dupin
              mkonecny Martin Konecny
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.