MariaDB Crashed on Master Server

One of our master servers crashed suddenly while checking error logs we could see below update query was found. After that we have promoted it's slave server as master but it was also crashed again.

The same query was found in error log.

MariaDB Version : 10.2.12-MariaDB
OS version : CentOS Linux 7 (Core)

Mysql Error Log :

2019-10-23 11:58:17 0x7ef5b85d4700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/storage/innobase/row/row0upd.cc line 1877
InnoDB: Failing assertion: dict_index_is_clust(index) || ind_field->prefix_len <= dfield_len
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
191023 11:58:17 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.2.12-MariaDB-log
key_buffer_size=2147483648
read_buffer_size=2097152
max_used_connections=2255
max_threads=5002
thread_count=2259
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22713427 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7ef93c023f28
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7ef5b85d3e98 thread_stack 0x49000
(my_addr_resolve failure: fork)
2019-10-23 11:58:18 139593472235264 [Warning] Aborted connection 975697007 to db: 'axolotlstore' user: 'wmaapp' host: '172.16.7.12' (Got timeout reading communication packets)
/usr/local/mariadb/bin/mysqld(my_print_stacktrace+0x2e) [0xde610e]
/usr/local/mariadb/bin/mysqld(handle_fatal_signal+0x471) [0x7de601]
/lib64/libpthread.so.0(+0xf6d0) [0x7f1c894bb6d0]
/lib64/libc.so.6(gsignal+0x37) [0x7f1c882c7277]
/lib64/libc.so.6(abort+0x148) [0x7f1c882c8968]
/usr/local/mariadb/bin/mysqld() [0xbafc13]
/usr/local/mariadb/bin/mysqld() [0xb4912c]
/usr/local/mariadb/bin/mysqld() [0xb4c07e]
/usr/local/mariadb/bin/mysqld() [0xb4d4b1]
/usr/local/mariadb/bin/mysqld() [0xb15076]
/usr/local/mariadb/bin/mysqld() [0xa4556b]
/usr/local/mariadb/bin/mysqld(handler::ha_update_row(unsigned char const*, unsigned char*)+0x16a) [0x7e845a]
/usr/local/mariadb/bin/mysqld(mysql_update(THD*, TABLE_LIST*, List<Item>&, List<Item>&, Item*, unsigned int, st_order*, unsigned long long, enum_duplicates, bool, unsigned long long*, unsigned long long*)+0x1433) [0x69d403]
/usr/local/mariadb/bin/mysqld(mysql_execute_command(THD*)+0x89eb) [0x5f1cbb]
/usr/local/mariadb/bin/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool)+0x3a2) [0x5f4272]
/usr/local/mariadb/bin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool)+0x26f0) [0x5f7190]
/usr/local/mariadb/bin/mysqld(do_command(THD*)+0x15d) [0x5f84cd]
/usr/local/mariadb/bin/mysqld(do_handle_one_connection(CONNECT*)+0x235) [0x6dc865]
/usr/local/mariadb/bin/mysqld(handle_one_connection+0x3f) [0x6dca2f]
/usr/local/mariadb/bin/mysqld() [0xa054a9]
/lib64/libpthread.so.0(+0x7e25) [0x7f1c894b3e25]
/lib64/libc.so.6(clone+0x6d) [0x7f1c8838fbad]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7ef93c0e70f0): is an invalid pointer
Connection ID (thread ID): 975738669
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

We think the query pointer is invalid, but we will try to print it anyway.
Query:
UPDATE `cat_meta` SET `url` = 'india-tour-packages/holidays-in-coorg', `meta_title` = 'default', `meta_keywords` = 'default', `meta_description` = '', `robots` = 'index,follow', `canonical` = 'default', `content` = '\n\nSet amidst the enchanting hills, deep valleys and huge coffee plantations, <a href=\"https://www.test.com/india-tourism/coorg-travel-guide\" target=\"_blank\">Coorg</a> is one of the perfect getaway places for a micro vacation to take a quick escape from \n\nhustle and bustle of urban life. Coorg can be described as a complete travel destination for a nature lover and for the ones residing in the Southern parts of India, \n\nit offers a pleasant escape from the hot and humid climate of the region which in itself is a change to enjoy. Today, we are going to provide a complete travel guide \n\nfor Coorg with tips to getting best tour package for Coorg and furthermore, we will list out some of the activities and Best <a href=\"https://www.test.com/india-\n\ntourism/places-to-visit-in-coorg\" target=\"_blank\">places to visit in Coorg</a> that you should definitely not miss out on. So, let us get right onto the main \n\ndiscussion.<br/><br/>\n\n<span class=\"bold\">What to pack?</span><br/>\nAre you wondering what to pack in your travel bag for a Coorg trip? Well, it all comes down to the fact that after all coorg is a hill station and you should pack \n\nexpecting a bit of cold but definitely no hot sun rather a pleasant and soothing weather. Your luggage will differ depending upon what kind of traveler you are, like \n\nyou will surely try to pack a lot less if you plan for a quick trip or a trip with lots of activity involved but you will pack relatively more if you are travelling \n\nwith a family or a group or might be your trip is a bit long. But don’t worry we got you covered in either of the cases and below, we have mentioned every bit of \n\ndetail for what to carry on your trip to coorg. <br/><br/>\n\nIf you want to keep your luggage light and easy to carry on the go, then apart from the essentials like tickets and itinerary and their print outs, Travel insurance \n\ndocs and relevant stuff, pack in a leather jacket (you can always look cool in a leather jacket even if it goes against the weather which will probably not be the case \n\nin Coorg), a set of thermals, boots. Comfortable cotton clothing will definitely go with the climate of coorg and will also keep the weight less. A pair of sports \n\njoggers and T-shirt will be handy if you plan to go for adventure sports.<br/><br/>\n \nFor those of you for whom the weight is not a concern but missing out on anything is, start with increasing the quantity of everything we mentioned earlier until it \n\nsatisfies your mind. Then you can add up essentials you always take along with you on a trip like a mp3 player, DVD player, a fridge etc. No, definitely you’re not \n\ngonna pack any of these things or are you? For Sure, we would recommend you to keep your luggage light and if you still want to add somethings then keep electronic \n\nessentials like a portable battery pack and some cables. Also, do not forget to keep some toiletries and a first aid kit.<br/><br/>\n\n<span class=\"bold\"><a href=\"https://www.test.com/india-tourism/mangalore/how-to-reach\" target=\"_blank\">How to Reach</a>?</span><br/>\nYou can reach Coorg via the nearest cities which are Mysore, Mangalore and Bengaluru. Below we have put up the information for how to reach coorg via different modes \n\nof transport.<br/><br/>\n\n<span class=\"bold\">Via Air</span><br/>\n<a href=\"https://www.test.com/domestic-airports/bajpe-airport_mangalore\" target=\"_blank\">Mangalore international airport</a> is the nearest domestic airport to coorg \n\nwhich is about 160km from it. If you are from abroad, then nearest international airport is located in Bengaluru and is approximately 260km from coorg. <br/><br/>\n\n<span class=\"bold\">Via Train</span><br/>\nAbout 95kms away, the Railway Station in Mysore is the nearest railway station to coorg.<br/><br/>\n\n<span class=\"bold\">Via Bus</span><br/>\nYou can take the bus route if you feel like and book <a href=\"https://www.test.com/bus-operators-india/ksrtc\" target=\"_blank\">KSRTC Online Booking</a> at test.com which has its daily trip from Bengaluru, Mangalore and Mysore.<br/><br/>\n\n<span class=\"bold\">Via Road</span><br/><br/>\nJust open google maps and get on the road in a 4 wheeler or a 2 wheeler with a helmet. <br/><br/>\nWell, in my opinion the best tour package for Coorg out of all <strong>coorg tour packages</strong> is the one done of road with best partners.<br/><br/>\n\n<span class=\"bold\">About the Place</span><br/><br/>\n\n● <span class=\"bold\">Plantations</span><br/>\nOne of the unique attraction about the <strong>Coorg holiday packages</strong> is its coffee \n\nplantation tours. You can get a complete insight right from the seed stage upto the processing of coffee that ends up in your house. These types of tours can last upto \n\n4 hours and will offer a great one of a kind experience to any person who loves their coffee. You get to learn about techniques like picking, drying and pruning which \n\nare the part of complete process. You should definitely add a visit to one coffee estate in things to do on your coorg tour package.<br/><br/>\n\n● <span class=\"bold\">Best places to Visit</span><br/>\nLike e mentioned in our introduction that coorg is a complete destination for a nature lover and we really meant it and to show it to you, below, we have mentioned a \n\nfew places, each with its own set of experiences and fun.<br/><br/>\n\n● <span class=\"bold\">Abbey Falls</span><br/>\nIn any collection of best places to visit in coorg, you will certainly come across the name of Abbey falls which is surely the best place to visit in the coorg. The \n\nwaterfall is formed by the combination of multiple small streams and heavy rainfall which finally ends up in the River Kaveri. A 30 minutes’ drive from the main town \n\nMadikeri located 10kms away will get you at the Abbey Falls. After this you will have to take down a 10minutes trek from a coffee plantation and a spice estate to \n\nfinally reach at the destination. The sound of waterfalls will provide you a calm and you can enjoy the majestic views of the falls from a hanging bridge. Cross the \n\nbridge and you will get to a kali Mata Mandir where a lot of tourists gathers.<br/><br/>\nIf you happen to travel during monsoon, then you will have a totally amazing experience on the waterfalls.<br/><br/>\n\n● <span class=\"bold\">Golden Temple</span><br/><br/>\nSituated at a distance of 34km from Madikeri, The Golden temple is a Buddhist Monastery also known as the Namdroling Monastery. The place is decorated with beautiful \n\npaintings of Gods and demons, attractive designs and inscriptions.<br/><br/>\n\nThis Monastery is second only to Dharamshala Monastery when it comes to Tibetan settlements and is home to around 6000 monks, students and nuns. Not only this, Golden \n\ntemple is the largest Nyingmapa Teaching center. You should visit this place during the Tibetan new year for the best experience which lasts for 15 days in the month \n\nof February or March. The place is decorated with Lights, colors and culture lama dances are also very great to enjoy.<br/><br/>\n\n● <span class=\"bold\">Tadiandamol</span><br/>\nBeing the highest peak of Coorg at 5735 feet, Tadiandamol Trek is a nature’s haven surrounded by scenic greenery. Nalkhand palace marks the start of trek and then \n\nprogresses into dense forest with beautiful flowers, streams, trees and exotic birds and creatures. The final destination of this trek is the Big Rock which serves as \n\na camping ground for a lot of travelers. This trek is a part of <strong>coorg tour package</strong> for many people. <br/><br/>\n\n● <span class=\"bold\">Coorg White Water Rafting</span><br/>\nWith waterfalls and hills in Coorg, you can expect a river rafting experience too. Coorg river rafting is one of the best amongst all <a href=\"https://www.test.com/india-tour-packages/holidays-in-south-india\" target=\"_blank\">south Indian tour package</a>. The \n\nrafting is set in kithu Kakathu river and ends in the Arabian sea. One of the interesting aspect of Rafting is the way they name the rapids and Coorg white water \n\nrafting is not different. Morning Coffee, Wicked witch, Milky Churn, Grasshopper, Big Bang and frame head are the names of dangerous rapids that you will have to cross \n\nduring rafting session of 4kms. The rapids are rated as a class 4 rapids.<br/><br/>\n\n \n● <span class=\"bold\">Gaddige</span><br/>\nPopular with the name of Raja’s Tomb, this place will definitely be perfect for both History ad nature lovers. This place contains the tombs of famous rulers from \n\nKodava Dynasty both Hindu and Muslims. You can easily notice the difference between a Hindu tomb and a Muslim tomb by the Shaivite Depictions. The place is built with \n\na Indo-Saracenic architectural style.<br/><br/> \n \n<span class=\"bold\">When to visit?</span><br/>\nThe climate of Coorg is calm and pleasant with average temperature ranging between the values of 15oC and 20oC. if you want to enjoy the beauty of nature and watch \n\nbeautiful flowers and vegetation in the valley, then March and April are the best months to plan your coorg holiday package while otherwise, the <a \n\nhref=\"https://www.test.com/india-tourism/coorg/best-time-to-visit\" target=\"_blank\">best time to visit coorg</a> is between the months of October and March.<br/><br/>\n\n<span class=\"bold\">Conclusion</span><br/>\nAll in all, Coorg in one of the best destinations in South India with nice climate and majestic sceneries to enjoy. Even though, we haven’t mentioned it yet but Coorg \n\nis also one of the best Indian destination for Honeymoon. Coffee and Spice plantation tours are really different places to visit than traditional sightseeing. For South Indians, this place is a great spot for a quick getaway trip while newlywed couple can definitely plan their <a href=\"https://www.test.com/india-tour-packages/honeymoon-holidays-in-coorg\" target=\"_blank\">Coorg honeymoon package</a>. So, I hope you gather \n\nonly the happy memories during your stay in Coorg and Bon Voyage. There is wide range of good hotels & <a href=\"https://www.test.com/hotels/resorts-in-coorg\" target=\"_blank\">resorts in Coorg</a> for visitors where tourist can feel the comfort of home.', `publish_status` = '1', `id_site` = '1' WHERE `id_category_meta` = '86278'

2019-10-23 11:58:38 140068900886336 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-10-23 11:58:38 140068900886336 [Note] InnoDB: Uses event mutexes
2019-10-23 11:58:38 140068900886336 [Note] InnoDB: Compressed tables use zlib 1.2.3
2019-10-23 11:58:38 140068900886336 [Note] InnoDB: Using Linux native AIO
2019-10-23 11:58:38 140068900886336 [Note] InnoDB: Number of pools: 1
2019-10-23 11:58:38 140068900886336 [Note] InnoDB: Using SSE2 crc32 instructions
2019-10-23 11:58:38 140068900886336 [Note] InnoDB: Initializing buffer pool, total size = 120G, instances = 8, chunk size = 128M
2019-10-23 11:58:44 140068900886336 [Note] InnoDB: Completed initialization of buffer pool
2019-10-23 11:58:46 139926732941056 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-10-23 11:58:46 140068900886336 [Note] InnoDB: Highest supported file format is Barracuda.
2019-10-23 11:58:47 140068900886336 [Note] InnoDB: Starting crash recovery from checkpoint LSN=210253180548824
2019-10-23 11:58:50 140068900886336 [Note] InnoDB: 3 transaction(s) which must be rolled back or cleaned up in total 3 row operations to undo
2019-10-23 11:58:50 140068900886336 [Note] InnoDB: Trx id counter is 79168903680
2019-10-23 11:58:50 140068900886336 [Note] InnoDB: Starting final batch to recover 19454 pages from redo log.
2019-10-23 11:58:54 140068900886336 [Note] InnoDB: Last binlog file '/data/binlogs1/mysql-bin.091425', position 496119900

2019-10-23 11:58:55 140068900886336 [ERROR] InnoDB: Operating system error number 2 in a file operation.
2019-10-23 11:58:55 140068900886336 [ERROR] InnoDB: The error means the system cannot find the path specified.
2019-10-23 11:58:55 140068900886336 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
2019-10-23 11:58:55 140068900886336 [ERROR] InnoDB: Cannot open datafile for read-only: './test/#sql-ib25-3173865361.ibd' OS error: 71
2019-10-23 11:58:55 140068900886336 [ERROR] InnoDB: Operating system error number 2 in a file operation.
2019-10-23 11:58:55 140068900886336 [ERROR] InnoDB: The error means the system cannot find the path specified.
2019-10-23 11:58:55 140068900886336 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
2019-10-23 11:58:55 140068900886336 [ERROR] InnoDB: Could not find a valid tablespace file for `test/#sql-ib25-3173865361`. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting-datadict.html for how to resolve the issue.
2019-10-23 11:58:55 140068900886336 [Warning] InnoDB: Ignoring tablespace for `test`.`#sql-ib25-3173865361` because it could not be opened.

2019-10-23 11:58:56 140068900886336 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-10-23 11:58:56 139926699370240 [Note] InnoDB: Starting in background the rollback of recovered transactions
2019-10-23 11:58:56 140068900886336 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2019-10-23 11:58:56 140068900886336 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-10-23 11:58:56 140068900886336 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-10-23 11:58:56 140068900886336 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2019-10-23 11:58:56 140068900886336 [Note] InnoDB: Waiting for purge to start
2019-10-23 11:58:56 139926699370240 [Note] InnoDB: Rollback of trx with id 0 completed
2019-10-23 11:58:56 139926699370240 [Note] InnoDB: Rollback of trx with id 0 completed
2019-10-23 11:58:56 0x7f433935e700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/storage/innobase/row/row0upd.cc line 1877
InnoDB: Failing assertion: dict_index_is_clust(index) || ind_field->prefix_len <= dfield_len
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
191023 11:58:56 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.2.12-MariaDB-log
key_buffer_size=2147483648
read_buffer_size=2097152
max_used_connections=0
max_threads=5002
thread_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 22692084 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
2019-10-23 11:58:56 140068900886336 [Note] InnoDB: 5.7.20 started; log sequence number 210253184959664
2019-10-23 11:58:56 139929682171648 [Note] InnoDB: Loading buffer pool(s) from /data/mariadata1/ib_buffer_pool
2019-10-23 11:58:56 140068900886336 [Note] Plugin 'FEEDBACK' is disabled.
191023 11:58:56 server_audit: MariaDB Audit Plugin version 1.4.3 STARTED.
191023 11:58:56 server_audit: logging started to the file /data/binlogs1/auditlog1/audit.log.
2019-10-23 11:58:56 140068900886336 [Note] Recovering after a crash using /data/binlogs1/mysql-bin
/usr/local/mariadb/bin/mysqld(my_print_stacktrace+0x2e)[0xde610e]
/usr/local/mariadb/bin/mysqld(handle_fatal_signal+0x471)[0x7de601]
/lib64/libpthread.so.0(+0xf5d0)[0x7f6454d2d5d0]
/lib64/libc.so.6(gsignal+0x37)[0x7f6453b39207]
/lib64/libc.so.6(abort+0x148)[0x7f6453b3a8f8]
/usr/local/mariadb/bin/mysqld[0xbafc13]
/usr/local/mariadb/bin/mysqld[0xb4912c]
/usr/local/mariadb/bin/mysqld[0xcece55]
/usr/local/mariadb/bin/mysqld[0xb44d25]
/usr/local/mariadb/bin/mysqld[0xad9076]
/usr/local/mariadb/bin/mysqld[0xb8c0cf]
/usr/local/mariadb/bin/mysqld[0xb8f5bf]
/usr/local/mariadb/bin/mysqld[0xb8feda]
/lib64/libpthread.so.0(+0x7dd5)[0x7f6454d25dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f6453c00ead]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.