[MDEV-29142] Assertion `!_current_thd()' failed in void destroy_background_thd(THD*) Created: 2022-07-20  Updated: 2023-03-10  Resolved: 2022-10-10

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 10.10
Fix Version/s: 10.4.27, 10.5.18, 10.6.11, 10.7.7, 10.8.6, 10.9.4, 10.10.2

Type: Bug Priority: Major
Reporter: Ramesh Sivaraman Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 0
Labels: not-10.3


 Description   

SET @@global.tx_read_only = ON;
SET default_storage_engine = SEQUENCE;
create table t1 (c1 int);
CREATE TABLE t1(c1 VARCHAR(20)) engine=innodb;
INSERT INTO t1  VALUES(0xA9B8);
create TABLE t1 (c1 int) engine=innodb;
create TABLE t1 (c1 int) engine=innodb;
SET GLOBAL wsrep_on=OFF;
SET GLOBAL wsrep_cluster_address='gcomm://';
INSERT INTO t1  VALUES (1);
SELECT 1;
SELECT 1;

Leads to

10.9.2 4a164364d71eecb004dc7c21ff608df996c05f0a (Optimized)

InnoDB: Failing assertion: trx_list.empty()

10.9.2 4a164364d71eecb004dc7c21ff608df996c05f0a (Optimized)

Core was generated by `/test/GAL_MD300622-mariadb-10.9.2-linux-x86_64-opt/bin/mysqld --defaults-file=/'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6)
    at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
[Current thread is 1 (Thread 0x14d26ee3f700 (LWP 3921561))]
(gdb) bt
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1  0x00005581d2a88acf in my_write_core (sig=sig@entry=6) at /test/10.9_opt/mysys/stacktrace.c:424
#2  0x00005581d25d8380 in handle_fatal_signal (sig=6) at /test/10.9_opt/sql/signal_handler.cc:355
#3  <signal handler called>
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#5  0x000014d27c6c7859 in __GI_abort () at abort.c:79
#6  0x00005581d2233588 in ut_dbg_assertion_failed (expr=expr@entry=0x5581d2ddb06e "trx_list.empty()", file=file@entry=0x5581d2ddb0d0 "/test/10.9_opt/storage/innobase/trx/trx0sys.cc", line=line@entry=374) at /test/10.9_opt/storage/innobase/ut/ut0dbg.cc:60
#7  0x00005581d2231afa in trx_sys_t::close (this=0x5581d3388100 <trx_sys>) at /test/10.9_opt/storage/innobase/trx/trx0sys.cc:374
#8  0x00005581d292f4ad in innodb_shutdown () at /test/10.9_opt/storage/innobase/srv/srv0start.cc:1755
#9  0x00005581d28494b7 in innobase_end () at /test/10.9_opt/storage/innobase/handler/ha_innodb.cc:4328
#10 innobase_end () at /test/10.9_opt/storage/innobase/handler/ha_innodb.cc:4315
#11 0x00005581d25db1ca in ha_finalize_handlerton (plugin=0x5581d4e0c8b0) at /test/10.9_opt/sql/handler.cc:596
#12 0x00005581d23af6f9 in plugin_deinitialize (plugin=0x5581d4e0c8b0, ref_check=ref_check@entry=true) at /test/10.9_opt/sql/sql_plugin.cc:1269
#13 0x00005581d23b3916 in reap_plugins () at /test/10.9_opt/sql/sql_plugin.cc:1343
#14 0x00005581d23b430d in plugin_shutdown () at /test/10.9_opt/sql/sql_plugin.cc:2051
#15 0x00005581d22949f5 in clean_up (print_message=<optimized out>) at /test/10.9_opt/sql/mysqld.cc:1987
#16 0x00005581d229654e in clean_up (print_message=<optimized out>) at /test/10.9_opt/sql/mysqld.cc:1953
#17 unireg_abort (exit_code=exit_code@entry=1) at /test/10.9_opt/sql/mysqld.cc:1899
#18 0x00005581d281bdd8 in wsrep_recover_sr_from_storage (orig_thd=<optimized out>) at /test/10.9_opt/sql/wsrep_mysqld.cc:480
#19 0x00005581d2bff15a in wsrep::server_state::recover_streaming_appliers_if_not_recovered<wsrep::high_priority_service> (this=0x5581d4d8f450, this=0x5581d4d8f450, c=@0x14d26ee3ed30: {_vptr.high_priority_service = 0x5581d3341158 <vtable for Wsrep_applier_service+16>, server_state_ = @0x5581d4d8f450, must_exit_ = false}, lock=@0x14d26ee3d8a0: {_M_device = 0x5581d4d8f738, _M_owns = false}) at /test/10.9_opt/wsrep-lib/include/wsrep/mutex.hpp:76
#20 wsrep::server_state::on_primary_view (this=0x5581d4d8f450, view=@0x14d26ee3db20: {state_id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, data_ = {buf = '\000' <repeats 15 times>}}, data_ = {buf = '\000' <repeats 15 times>}}, seqno_ = {seqno_ = -1}}, id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, data_ = {buf = '\000' <repeats 15 times>}}, data_ = {buf = "\324kж\b\006\021\355\207B*\274\264$&D"}}, seqno_ = {seqno_ = 10}}, view_seqno_ = {seqno_ = 1}, status_ = wsrep::view::primary, capabilities_ = 184703, own_index_ = 0, protocol_version_ = 4, members_ = std::vector of length 1, capacity 1 = {{id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, data_ = {buf = '\000' <repeats 15 times>}}, data_ = {buf = "\334К<\b\006\021\355\255+\233\240\371\367m\203"}}, name_ = "galapq", incoming_ = "127.0.0.1:10490"}}}, high_priority_service=0x14d26ee3ed30) at /test/10.9_opt/wsrep-lib/src/server_state.cpp:969
#21 0x00005581d2c0108a in wsrep::server_state::on_view (this=this@entry=0x5581d4d8f450, view=@0x14d26ee3db20: {state_id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, data_ = {buf = '\000' <repeats 15 times>}}, data_ = {buf = '\000' <repeats 15 times>}}, seqno_ = {seqno_ = -1}}, id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, data_ = {buf = '\000' <repeats 15 times>}}, data_ = {buf = "\324kж\b\006\021\355\207B*\274\264$&D"}}, seqno_ = {seqno_ = 10}}, view_seqno_ = {seqno_ = 1}, status_ = wsrep::view::primary, capabilities_ = 184703, own_index_ = 0, protocol_version_ = 4, members_ = std::vector of length 1, capacity 1 = {{id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, data_ = {buf = '\000' <repeats 15 times>}}, data_ = {buf = "\334К<\b\006\021\355\255+\233\240\371\367m\203"}}, name_ = "galapq", incoming_ = "127.0.0.1:10490"}}}, high_priority_service=high_priority_service@entry=0x14d26ee3ed30) at /test/10.9_opt/wsrep-lib/src/server_state.cpp:1028
#22 0x00005581d2c12982 in (anonymous namespace)::view_cb (app_ctx=0x5581d4d8f450, recv_ctx=0x14d26ee3ed30, view_info=<optimized out>) at /test/10.9_opt/wsrep-lib/src/wsrep_provider_v26.cpp:404
#23 0x000014d27c13345d in galera::ReplicatorSMM::submit_view_info (this=<optimized out>, recv_ctx=<optimized out>, view_info=<optimized out>) at /test/10.9_galera_opt/galera/src/replicator_smm.cpp:2435
#24 0x000014d27c13349d in galera::ReplicatorSMM::submit_ordered_view_info (this=0x5581d4dc51e0, recv_ctx=<optimized out>, view_info=<optimized out>) at /test/10.9_galera_opt/galera/src/replicator_smm.cpp:2825
#25 0x000014d27c143ce9 in galera::ReplicatorSMM::process_prim_conf_change (this=0x5581d4dc51e0, recv_ctx=0x14d26ee3ed30, conf=<optimized out>, my_index=<optimized out>, cc_buf=<optimized out>) at /test/10.9_galera_opt/galera/src/replicator_smm.cpp:3016
#26 0x000014d27c143f6d in galera::ReplicatorSMM::process_conf_change (this=0x5581d4dc51e0, recv_ctx=0x14d26ee3ed30, conf=@0x14d26ee3e560: {memb = std::vector of length 1, capacity 1 = {{uuid_ = {data = "\334К<\b\006\021\355\255+\233\240\371\367m\203", alignment = 17082441494039285980}, name_ = "galapq", incoming_ = "127.0.0.1:10490", cached_ = 1, state_ = GCS_NODE_STATE_SYNCED}}, uuid = {data = "\324kж\b\006\021\355\207B*\274\264$&D", alignment = 17082441496089619412}, seqno = 10, conf_id = 1, vote_seqno = -1, vote_res = 0, repl_proto_ver = 10, appl_proto_ver = 4}, cc=@0x14d26ee3e750: {seqno_g = 0, seqno_l = 17, buf = 0x14d26fd2a550, size = 136, type = GCS_ACT_CCHANGE}) at /test/10.9_galera_opt/galera/src/replicator_smm.cpp:2464
#27 0x000014d27c174c6b in galera::GcsActionSource::dispatch (this=0x5581d4dbdd80, recv_ctx=0x14d26ee3ed30, act=@0x14d26ee3e750: {seqno_g = 0, seqno_l = 17, buf = 0x14d26fd2a550, size = 136, type = GCS_ACT_CCHANGE}, exit_loop=<optimized out>) at /test/10.9_galera_opt/galera/src/gcs_action_source.cpp:127
#28 0x000014d27c175132 in galera::GcsActionSource::process (this=0x5581d4dbdd80, recv_ctx=0x14d26ee3ed30, exit_loop=@0x14d26ee3e95f: false) at /test/10.9_galera_opt/galera/src/gcs_action_source.cpp:186
#29 0x000014d27c13fa90 in galera::ReplicatorSMM::async_recv (this=0x5581d4dc51e0, recv_ctx=0x14d26ee3ed30) at /test/10.9_galera_opt/galera/src/replicator_smm.cpp:402
#30 0x000014d27c114151 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at /test/10.9_galera_opt/galera/src/wsrep_provider.cpp:263
#31 0x00005581d2c106f2 in wsrep::wsrep_provider_v26::run_applier (this=<optimized out>, applier_ctx=<optimized out>) at /test/10.9_opt/wsrep-lib/src/wsrep_provider_v26.cpp:858
#32 0x00005581d2830be7 in wsrep_replication_process (thd=0x14d260001a08, arg=<optimized out>) at /test/10.9_opt/wsrep-lib/include/wsrep/server_state.hpp:320
#33 0x00005581d2820930 in start_wsrep_THD (arg=0x14d20801d6e0) at /test/10.9_opt/sql/wsrep_mysqld.h:575
#34 0x000014d27cbd6609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#35 0x000014d27c7c4293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

10.10.0 330e5b5de41a5d519d75d9d9dcbf8f483cfb1fed

mysqld: /test/mtest/10.10/sql/sql_class.cc:5029: void destroy_background_thd(THD*): Assertion `!_current_thd()' failed.

10.10.0 330e5b5de41a5d519d75d9d9dcbf8f483cfb1fed

Core was generated by `/test/mtest/mariadb-10.10.0-linux-x86_64/bin/mysqld --defaults-file=/test/mtest'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill (threadid=<optimized out>, signo=6)
    at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
[Current thread is 1 (Thread 0x149af4109700 (LWP 3605603))]
(gdb) bt
#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1  0x000055a64bf85ef2 in my_write_core (sig=6) at /test/mtest/10.10/mysys/stacktrace.c:424
#2  0x000055a64b6da443 in handle_fatal_signal (sig=6) at /test/mtest/10.10/sql/signal_handler.cc:345
#3  <signal handler called>
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#5  0x0000149b29fc2859 in __GI_abort () at abort.c:79
#6  0x0000149b29fc2729 in __assert_fail_base (fmt=0x149b2a158588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55a64c0ac65f "!_current_thd()", file=0x55a64c0aad78 "/test/mtest/10.10/sql/sql_class.cc", line=5029, function=<optimized out>) at assert.c:92
#7  0x0000149b29fd3f36 in __GI___assert_fail (assertion=0x55a64c0ac65f "!_current_thd()", file=0x55a64c0aad78 "/test/mtest/10.10/sql/sql_class.cc", line=5029, function=0x55a64c0ac688 "void destroy_background_thd(THD*)") at assert.c:101
#8  0x000055a64b28fd8c in destroy_background_thd (thd=0x55a64f3f2f18) at /test/mtest/10.10/sql/sql_class.cc:5029
#9  0x000055a64baced1c in innobase_destroy_background_thd (thd=0x55a64f3f2f18) at /test/mtest/10.10/storage/innobase/handler/ha_innodb.cc:1779
#10 0x000055a64be9a412 in fts_optimize_shutdown () at /test/mtest/10.10/storage/innobase/fts/fts0opt.cc:2977
#11 0x000055a64bd0ed39 in srv_shutdown_bg_undo_sources () at /test/mtest/10.10/storage/innobase/srv/srv0start.cc:1641
#12 0x000055a64bd0ee00 in innodb_preshutdown () at /test/mtest/10.10/storage/innobase/srv/srv0start.cc:1670
#13 0x000055a64bd0ee48 in innodb_shutdown () at /test/mtest/10.10/storage/innobase/srv/srv0start.cc:1681
#14 0x000055a64bad41ed in innobase_end () at /test/mtest/10.10/storage/innobase/handler/ha_innodb.cc:4310
#15 0x000055a64b6ddbfc in ha_finalize_handlerton (plugin=0x55a64ef7c800) at /test/mtest/10.10/sql/handler.cc:596
#16 0x000055a64b340d0d in plugin_deinitialize (plugin=0x55a64ef7c800, ref_check=false) at /test/mtest/10.10/sql/sql_plugin.cc:1272
#17 0x000055a64b343a62 in plugin_shutdown () at /test/mtest/10.10/sql/sql_plugin.cc:2098
#18 0x000055a64b1744ae in clean_up (print_message=true) at /test/mtest/10.10/sql/mysqld.cc:1987
#19 0x000055a64b174222 in unireg_abort (exit_code=1) at /test/mtest/10.10/sql/mysqld.cc:1899
#20 0x000055a64ba8859d in wsrep_recover_sr_from_storage (orig_thd=0x149aa4000ff8) at /test/mtest/10.10/sql/wsrep_mysqld.cc:480
#21 0x000055a64bab1ba6 in Wsrep_server_service::recover_streaming_appliers (this=0x55a64eefd8d8, hs=@0x149af4108d00: {_vptr.high_priority_service = 0x55a64ca779c8 <vtable for Wsrep_applier_service+16>, server_state_ = @0x55a64eefd5d0, must_exit_ = false}) at /test/mtest/10.10/sql/wsrep_server_service.cc:299
#22 0x000055a64c04bc7b in wsrep::server_state::recover_streaming_appliers_if_not_recovered<wsrep::high_priority_service> (this=0x55a64eefd5d0, lock=@0x149af4107810: {_M_device = 0x55a64eefd8b8, _M_owns = false}, c=@0x149af4108d00: {_vptr.high_priority_service = 0x55a64ca779c8 <vtable for Wsrep_applier_service+16>, server_state_ = @0x55a64eefd5d0, must_exit_ = false}) at /test/mtest/10.10/wsrep-lib/src/server_state.cpp:1405
#23 0x000055a64c047eae in wsrep::server_state::on_primary_view (this=0x55a64eefd5d0, view=@0x149af4107ab0: {state_id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, data_ = {buf = '\000' <repeats 15 times>}}, data_ = {buf = '\000' <repeats 15 times>}}, seqno_ = {seqno_ = -1}}, id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, data_ = {buf = '\000' <repeats 15 times>}}, data_ = {buf = "$[\017\307\b\004\021\355\265u69\214>]e"}}, seqno_ = {seqno_ = 10}}, view_seqno_ = {seqno_ = 1}, status_ = wsrep::view::primary, capabilities_ = 184703, own_index_ = 0, protocol_version_ = 4, members_ = std::vector of length 1, capacity 1 = {{id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, data_ = {buf = '\000' <repeats 15 times>}}, data_ = {buf = "3$dk\b\004\021\355\254\301Fq\033\246/k"}}, name_ = "galapq", incoming_ = "127.0.0.1:11264"}}}, high_priority_service=0x149af4108d00) at /test/mtest/10.10/wsrep-lib/src/server_state.cpp:969
#24 0x000055a64c0482ff in wsrep::server_state::on_view (this=0x55a64eefd5d0, view=@0x149af4107ab0: {state_id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, data_ = {buf = '\000' <repeats 15 times>}}, data_ = {buf = '\000' <repeats 15 times>}}, seqno_ = {seqno_ = -1}}, id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, data_ = {buf = '\000' <repeats 15 times>}}, data_ = {buf = "$[\017\307\b\004\021\355\265u69\214>]e"}}, seqno_ = {seqno_ = 10}}, view_seqno_ = {seqno_ = 1}, status_ = wsrep::view::primary, capabilities_ = 184703, own_index_ = 0, protocol_version_ = 4, members_ = std::vector of length 1, capacity 1 = {{id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, data_ = {buf = '\000' <repeats 15 times>}}, data_ = {buf = "3$dk\b\004\021\355\254\301Fq\033\246/k"}}, name_ = "galapq", incoming_ = "127.0.0.1:11264"}}}, high_priority_service=0x149af4108d00) at /test/mtest/10.10/wsrep-lib/src/server_state.cpp:1028
#25 0x000055a64c062f15 in (anonymous namespace)::view_cb (app_ctx=0x55a64eefd5d0, recv_ctx=0x149af4108d00, view_info=0x149aa40224b0) at /test/mtest/10.10/wsrep-lib/src/wsrep_provider_v26.cpp:404
#26 0x0000149b29a4a3fd in galera::ReplicatorSMM::submit_view_info (this=<optimized out>, recv_ctx=<optimized out>, view_info=<optimized out>) at /test/mtest/galera/galera/src/replicator_smm.cpp:2435
#27 0x0000149b29a4a43d in galera::ReplicatorSMM::submit_ordered_view_info (this=0x55a64ef31ff0, recv_ctx=<optimized out>, view_info=<optimized out>) at /test/mtest/galera/galera/src/replicator_smm.cpp:2825
#28 0x0000149b29a5ac59 in galera::ReplicatorSMM::process_prim_conf_change (this=0x55a64ef31ff0, recv_ctx=0x149af4108d00, conf=<optimized out>, my_index=<optimized out>, cc_buf=<optimized out>) at /test/mtest/galera/galera/src/replicator_smm.cpp:3016
#29 0x0000149b29a5aedd in galera::ReplicatorSMM::process_conf_change (this=0x55a64ef31ff0, recv_ctx=0x149af4108d00, conf=@0x149af41084e0: {memb = std::vector of length 1, capacity 1 = {{uuid_ = {data = "3$dk\b\004\021\355\254\301Fq\033\246/k", alignment = 17082439295800976435}, name_ = "galapq", incoming_ = "127.0.0.1:11264", cached_ = 1, state_ = GCS_NODE_STATE_SYNCED}}, uuid = {data = "$[\017\307\b\004\021\355\265u69\214>]e", alignment = 17082439297338923812}, seqno = 10, conf_id = 1, vote_seqno = -1, vote_res = 0, repl_proto_ver = 10, appl_proto_ver = 4}, cc=@0x149af41086d0: {seqno_g = 0, seqno_l = 18, buf = 0x149b1c000598, size = 136, type = GCS_ACT_CCHANGE}) at /test/mtest/galera/galera/src/replicator_smm.cpp:2464
#30 0x0000149b29a8bbbb in galera::GcsActionSource::dispatch (this=0x55a64ef62240, recv_ctx=0x149af4108d00, act=@0x149af41086d0: {seqno_g = 0, seqno_l = 18, buf = 0x149b1c000598, size = 136, type = GCS_ACT_CCHANGE}, exit_loop=<optimized out>) at /test/mtest/galera/galera/src/gcs_action_source.cpp:127
#31 0x0000149b29a8c082 in galera::GcsActionSource::process (this=0x55a64ef62240, recv_ctx=0x149af4108d00, exit_loop=@0x149af41088df: false) at /test/mtest/galera/galera/src/gcs_action_source.cpp:186
#32 0x0000149b29a56a20 in galera::ReplicatorSMM::async_recv (this=0x55a64ef31ff0, recv_ctx=0x149af4108d00) at /test/mtest/galera/galera/src/replicator_smm.cpp:402
#33 0x0000149b29a2b161 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at /test/mtest/galera/galera/src/wsrep_provider.cpp:263
#34 0x000055a64c064af6 in wsrep::wsrep_provider_v26::run_applier (this=0x55a64eebd060, applier_ctx=0x149af4108d00) at /test/mtest/10.10/wsrep-lib/src/wsrep_provider_v26.cpp:858
#35 0x000055a64baa3236 in wsrep_replication_process (thd=0x149aa4000ff8, arg=0x149a940b80e0) at /test/mtest/10.10/sql/wsrep_thd.cc:57
#36 0x000055a64ba925ac in start_wsrep_THD (arg=0x149a940b80e0) at /test/mtest/10.10/sql/wsrep_mysqld.cc:3593
#37 0x000055a64b9fc00f in pfs_spawn_thread (arg=0x149a940b99e8) at /test/mtest/10.10/storage/perfschema/pfs.cc:2201
#38 0x0000149b2a4ec609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#39 0x0000149b2a0bf293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Bug confirmed present in:
MariaDB: 10.3.36 (opt), 10.3.36 (dbg), 10.4.26 (opt), 10.5.17 (opt), 10.6.9 (opt), 10.7.5 (opt), 10.8.4 (opt), 10.9.2 (opt), 10.10.0 (opt), 10.4.26 (dbg), 10.5.17 (dbg), 10.6.9 (dbg), 10.7.5 (dbg), 10.8.4 (dbg), 10.9.2 (dbg), 10.10.0 (dbg)



 Comments   
Comment by Marko Mäkelä [ 2022-10-12 ]

The added test galera.MDEV-29142 fails on 10.5 with WSREP: Inconsistency detected. Please fix.

Generated at Thu Feb 08 10:06:13 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.