Details
-
New Feature
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Currently the diff router uses a sample buffer based on which a histogram is built. The bin count for the histogram depends on the data itself and extreme outlier values may cause massive bin counts to appear. This is because the bin count comes from either the Freedman-Diaconis rule or Sturges's formula and the bin count is (max - min) / bin_width which can be very large.
Using DDSketch for the histogram implementation has a number of benefits:
- guaranteed relative error values that are known in advance
- configurable error (e.g. 0.5%, 1% or 2%)
- fixed size
- supports a larger range of values with less memory
- outliers clamped to predefined maximums
This also would make it possible to have rotation of the histograms which currently is not possible as the sampling is done at the start and at the end of the comparison, the whole workload might've drifted out of the sampled window.