Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-20889

MariaDB unrecoverable crash on update query

    XMLWordPrintable

Details

    Description

      MariaDB server was crashed and unable to recover post crash.
      Brought a failover server as master, failover also crashed with the same query and same error.

      Note: Both the servers unable to start post crash.

      From the coredump, we are suspecting the below

      m_body_utf8 = 0x0, m_body_utf8_ptr = 0x2020 <Address 0x2020 out of bounds>, m_cpp_utf8_processed_ptr = 0x0, next_state = MY_LEX_END, found_semicolon = 0x0, tok_bitmap = 63 '?',
      ignore_space = false, stmt_prepare_mode = false, multi_statements = true, in_comment = NO_COMMENT, in_comment_saved = (unknown: 4160914512), m_cpp_text_start = 0x7f47f8010b95 "86278'",
      m_cpp_text_end = 0x7f47f8010b9a "'", m_underscore_cs = 0x0, m_digest = 0x0}, m_yacc = {yacc_yyss = 0x0, yacc_yyvs = 0x0, m_set_signal_info = {m_item = {0x0 <repeats 12 times>}},
      m_lock_type = TL_READ_DEFAULT, m_mdl_type = MDL_SHARED_READ}, m_digest_psi = 0x0}

      Error log

      2019-10-23 11:58:17 0x7ef5b85d4700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/storage/innobase/row/row0upd.cc line 1877
      InnoDB: Failing assertion: dict_index_is_clust(index) || ind_field->prefix_len <= dfield_len
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
       
      191023 11:58:17 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.2.12-MariaDB-log
      key_buffer_size=2147483648
      read_buffer_size=2097152
      max_used_connections=2255
      max_threads=5002
      thread_count=2259
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22713427 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7ef93c023f28
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7ef5b85d3e98 thread_stack 0x49000
      (my_addr_resolve failure: fork)
      /usr/local/mariadb/bin/mysqld(my_print_stacktrace+0x2e) [0xde610e]
      /usr/local/mariadb/bin/mysqld(handle_fatal_signal+0x471) [0x7de601]
      /lib64/libpthread.so.0(+0xf6d0) [0x7f1c894bb6d0]
      /lib64/libc.so.6(gsignal+0x37) [0x7f1c882c7277]
      /lib64/libc.so.6(abort+0x148) [0x7f1c882c8968]
      /usr/local/mariadb/bin/mysqld() [0xbafc13]
      /usr/local/mariadb/bin/mysqld() [0xb4912c]
      /usr/local/mariadb/bin/mysqld() [0xb4c07e]
      /usr/local/mariadb/bin/mysqld() [0xb4d4b1]
      /usr/local/mariadb/bin/mysqld() [0xb15076]
      /usr/local/mariadb/bin/mysqld() [0xa4556b]
      /usr/local/mariadb/bin/mysqld(handler::ha_update_row(unsigned char const*, unsigned char*)+0x16a) [0x7e845a]
      /usr/local/mariadb/bin/mysqld(mysql_update(THD*, TABLE_LIST*, List<Item>&, List<Item>&, Item*, unsigned int, st_order*, unsigned long long, enum_duplicates, bool, unsigned long long*, unsigned long long*)+0x1433) [0x69d403]
      /usr/local/mariadb/bin/mysqld(mysql_execute_command(THD*)+0x89eb) [0x5f1cbb]
      /usr/local/mariadb/bin/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool)+0x3a2) [0x5f4272]
      /usr/local/mariadb/bin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool)+0x26f0) [0x5f7190]
      /usr/local/mariadb/bin/mysqld(do_command(THD*)+0x15d) [0x5f84cd]
      /usr/local/mariadb/bin/mysqld(do_handle_one_connection(CONNECT*)+0x235) [0x6dc865]
      /usr/local/mariadb/bin/mysqld(handle_one_connection+0x3f) [0x6dca2f]
      /usr/local/mariadb/bin/mysqld() [0xa054a9]
      /lib64/libpthread.so.0(+0x7e25) [0x7f1c894b3e25]
      /lib64/libc.so.6(clone+0x6d) [0x7f1c8838fbad]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7ef93c0e70f0): is an invalid pointer
      Connection ID (thread ID): 975738669
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
      {code}
       
      Following query is the culprit one, as we intentionally fired on one of the backup server to test the crash. Same crash is happened and MariaDB is unable to come up on the backup server as well.
      

      Query

      UPDATE `yt_category_meta` SET `url` = 'india-tour-packages/holidays-in-coorg', `meta_title` = 'default', `meta_keywords` = 'default', `meta_description` = '', `robots` = 'index,follow', `canonical` = 'default', `content` = '\n\nSet amidst the enchanting hills, deep valleys and huge coffee plantations, <a href=\"https://www.mytravelindia.com/india-tourism/coorg-travel-guide\" target=\"_blank\">Coorg</a> is one of the perfect getaway places for a micro vacation to take a quick escape from \n\nhustle and bustle of urban life. Coorg can be described as a complete travel destination for a nature lover and for the ones residing in the Southern parts of India, \n\nit offers a pleasant escape from the hot and humid climate of the region which in itself is a change to enjoy. Today, we are going to provide a complete travel guide \n\nfor Coorg with tips to getting best tour package for Coorg and furthermore, we will list out some of the activities and Best <a href=\"https://www.mytravelindia.com/india-\n\ntourism/places-to-visit-in-coorg\" target=\"_blank\">places to visit in Coorg</a> that you should definitely not miss out on. So, let us get right onto the main \n\ndiscussion.<br/><br/>\n\n<span class=\"bold\">What to pack?</span><br/>\nAre you wondering what to pack in your travel bag for a Coorg trip? Well, it all comes down to the fact that after all coorg is a hill station and you should pack \n\nexpecting a bit of cold but definitely no hot sun rather a pleasant and soothing weather. Your luggage will differ depending upon what kind of traveler you are, like \n\nyou will surely try to pack a lot less if you plan for a quick trip or a trip with lots of activity involved but you will pack relatively more if you are travelling \n\nwith a family or a group or might be your trip is a bit long. But don't worry we got you covered in either of the cases and below, we have mentioned every bit of \n\ndetail for what to carry on your trip to coorg. <br/><br/>\n\nIf you want to keep your luggage light and easy to carry on the go, then apart from the essentials like tickets and itinerary and their print outs, Travel insurance \n\ndocs and relevant stuff, pack in a leather jacket (you can always look cool in a leather jacket even if it goes against the weather which will probably not be the case \n\nin Coorg), a set of thermals, boots. Comfortable cotton clothing will definitely go with the climate of coorg and will also keep the weight less. A pair of sports \n\njoggers and T-shirt will be handy if you plan to go for adventure sports.<br/><br/>\n \nFor those of you for whom the weight is not a concern but missing out on anything is, start with increasing the quantity of everything we mentioned earlier until it \n\nsatisfies your mind. Then you can add up essentials you always take along with you on a trip like a mp3 player, DVD player, a fridge etc. No, definitely you're not \n\ngonna pack any of these things or are you? For Sure, we would recommend you to keep your luggage light and if you still want to add somethings then keep electronic \n\nessentials like a portable battery pack and some cables. Also, do not forget to keep some toiletries and a first aid kit.<br/><br/>\n\n<span class=\"bold\"><a href=\"https://www.mytravelindia.com/india-tourism/mangalore/how-to-reach\" target=\"_blank\">How to Reach</a>?</span><br/>\nYou can reach Coorg via the nearest cities which are Mysore, Mangalore and Bengaluru. Below we have put up the information for how to reach coorg via different modes \n\nof transport.<br/><br/>\n\n<span class=\"bold\">Via Air</span><br/>\n<a href=\"https://www.mytravelindia.com/domestic-airports/bajpe-airport_mangalore\" target=\"_blank\">Mangalore international airport</a> is the nearest domestic airport to coorg \n\nwhich is about 160km from it. If you are from abroad, then nearest international airport is located in Bengaluru and is approximately 260km from coorg. <br/><br/>\n\n<span class=\"bold\">Via Train</span><br/>\nAbout 95kms away, the Railway Station in Mysore is the nearest railway station to coorg.<br/><br/>\n\n<span class=\"bold\">Via Bus</span><br/>\nYou can take the bus route if you feel like and book <a href=\"https://www.mytravelindia.com/bus-operators-india/ksrtc\" target=\"_blank\">KSRTC Online Booking</a> at mytravelindia.com which has its daily trip from Bengaluru, Mangalore and Mysore.<br/><br/>\n\n<span class=\"bold\">Via Road</span><br/><br/>\nJust open google maps and get on the road in a 4 wheeler or a 2 wheeler with a helmet. <br/><br/>\nWell, in my opinion the best tour package for Coorg out of all <strong>coorg tour packages</strong> is the one done of road with best partners.<br/><br/>\n\n<span class=\"bold\">About the Place</span><br/><br/>\n\n●&nbsp;<span class=\"bold\">Plantations</span><br/>\nOne of the unique attraction about the <strong>Coorg holiday packages</strong> is its coffee \n\nplantation tours. You can get a complete insight right from the seed stage upto the processing of coffee that ends up in your house. These types of tours can last upto \n\n4 hours and will offer a great one of a kind experience to any person who loves their coffee. You get to learn about techniques like picking, drying and pruning which \n\nare the part of complete process. You should definitely add a visit to one coffee estate in things to do on your coorg tour package.<br/><br/>\n\n●&nbsp;<span class=\"bold\">Best places to Visit</span><br/>\nLike e mentioned in our introduction that coorg is a complete destination for a nature lover and we really meant it and to show it to you, below, we have mentioned a \n\nfew places, each with its own set of experiences and fun.<br/><br/>\n\n●&nbsp;<span class=\"bold\">Abbey Falls</span><br/>\nIn any collection of best places to visit in coorg, you will certainly come across the name of Abbey falls which is surely the best place to visit in the coorg. The \n\nwaterfall is formed by the combination of multiple small streams and heavy rainfall which finally ends up in the River Kaveri. A 30 minutes' drive from the main town \n\nMadikeri located 10kms away will get you at the Abbey Falls. After this you will have to take down a 10minutes trek from a coffee plantation and a spice estate to \n\nfinally reach at the destination. The sound of waterfalls will provide you a calm and you can enjoy the majestic views of the falls from a hanging bridge. Cross the \n\nbridge and you will get to a kali Mata Mandir where a lot of tourists gathers.<br/><br/>\nIf you happen to travel during monsoon, then you will have a totally amazing experience on the waterfalls.<br/><br/>\n\n●&nbsp;<span class=\"bold\">Golden Temple</span><br/><br/>\nSituated at a distance of 34km from Madikeri, The Golden temple is a Buddhist Monastery also known as the Namdroling Monastery. The place is decorated with beautiful \n\npaintings of Gods and demons, attractive designs and inscriptions.<br/><br/>\n\nThis Monastery is second only to Dharamshala Monastery when it comes to Tibetan settlements and is home to around 6000 monks, students and nuns. Not only this, Golden \n\ntemple is the largest Nyingmapa Teaching center. You should visit this place during the Tibetan new year for the best experience which lasts for 15 days in the month \n\nof February or March. The place is decorated with Lights, colors and culture lama dances are also very great to enjoy.<br/><br/>\n\n●&nbsp;<span class=\"bold\">Tadiandamol</span><br/>\nBeing the highest peak of Coorg at 5735 feet, Tadiandamol Trek is a nature's haven surrounded by scenic greenery. Nalkhand palace marks the start of trek and then \n\nprogresses into dense forest with beautiful flowers, streams, trees and exotic birds and creatures. The final destination of this trek is the Big Rock which serves as \n\na camping ground for a lot of travelers. This trek is a part of <strong>coorg tour package</strong> for many people. <br/><br/>\n\n●&nbsp;<span class=\"bold\">Coorg White Water Rafting</span><br/>\nWith waterfalls and hills in Coorg, you can expect a river rafting experience too. Coorg river rafting is one of the best amongst all <a href=\"https://www.mytravelindia.com/india-tour-packages/holidays-in-south-india\" target=\"_blank\">south Indian tour package</a>. The \n\nrafting is set in kithu Kakathu river and ends in the Arabian sea. One of the interesting aspect of Rafting is the way they name the rapids and Coorg white water \n\nrafting is not different. Morning Coffee, Wicked witch, Milky Churn, Grasshopper, Big Bang and frame head are the names of dangerous rapids that you will have to cross \n\nduring rafting session of 4kms. The rapids are rated as a class 4 rapids.<br/><br/>\n\n \n●&nbsp;<span class=\"bold\">Gaddige</span><br/>\nPopular with the name of Raja's Tomb, this place will definitely be perfect for both History ad nature lovers. This place contains the tombs of famous rulers from \n\nKodava Dynasty both Hindu and Muslims. You can easily notice the difference between a Hindu tomb and a Muslim tomb by the Shaivite Depictions. The place is built with \n\na Indo-Saracenic architectural style.<br/><br/> \n \n<span class=\"bold\">When to visit?</span><br/>\nThe climate of Coorg is calm and pleasant with average temperature ranging between the values of 15oC and 20oC. if you want to enjoy the beauty of nature and watch \n\nbeautiful flowers and vegetation in the valley, then March and April are the best months to plan your coorg holiday package while otherwise, the <a \n\nhref=\"https://www.mytravelindia.com/india-tourism/coorg/best-time-to-visit\" target=\"_blank\">best time to visit coorg</a> is between the months of October and March.<br/><br/>\n\n<span class=\"bold\">Conclusion</span><br/>\nAll in all, Coorg in one of the best destinations in South India with nice climate and majestic sceneries to enjoy. Even though, we haven't mentioned it yet but Coorg \n\nis also one of the best Indian destination for Honeymoon. Coffee and Spice plantation tours are really different places to visit than traditional sightseeing. For South Indians, this place is a great spot for a quick getaway trip while newlywed couple can definitely plan their <a href=\"https://www.mytravelindia.com/india-tour-packages/honeymoon-holidays-in-coorg\" target=\"_blank\">Coorg honeymoon package</a>. So, I hope you gather \n\nonly the happy memories during your stay in Coorg and Bon Voyage. There is wide range of good hotels & <a href=\"https://www.mytravelindia.com/hotels/resorts-in-coorg\" target=\"_blank\">resorts in Coorg</a> for visitors where tourist can feel the comfort of home.', `publish_status` = '1', `id_site` = '1' WHERE `id_category_meta` =  '86278';
      

      Processed the core dump and attached the same.

      Attachments

        Issue Links

          Activity

            People

              marko Marko Mäkelä
              Sukan Sukan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.