The function rtr_estimate_n_rows_in_range() looks like dead and buggy code.
If it were called, it should trigger rec_offs_validate() assertion failure in debug builds,
because it is not updating 'offsets' in sync with 'rec'.
Spatial indexes define a partial order, not a total order. A range cannot really exist without total order. Therefore, the virtual member function handler::records_in_range() probably should not executed on a SPATIAL INDEX at all.
We should probably remove the code from MariaDB Server 10.2.