E · Benchmark and metrics explorer
Metrics
Latency is wall-clock per frame, pooled over repeats. The dashed line is the sensor period. A run's hardware is in its label.
| Run | Frames | P50 | P90 | P95 | P99 | Max | Violations | Accepted | ATE RMSE | Peak RSS | Evidence |
|---|---|---|---|---|---|---|---|---|---|---|---|
| baseline · laptop-full newer_college_bag9 · phase3 · 0862f4f · uncommitted changes | 270 | 749.9 ms | 1275.7 ms | 1547.6 ms | 2111.6 ms | 2684.2 ms | 100.0 % | 100.0 % | 0.51 m | 77 MB | REAL |
| baseline · laptop-4c newer_college_bag9 · phase3 · 0862f4f · uncommitted changes | 270 | 676.2 ms | 1104.0 ms | 1309.6 ms | 1816.2 ms | 2328.2 ms | 100.0 % | 100.0 % | 0.51 m | 78 MB | REAL |
| baseline · laptop-1c newer_college_bag9 · phase3 · 0862f4f · uncommitted changes | 270 | 671.5 ms | 1108.8 ms | 1367.2 ms | 1864.3 ms | 2318.7 ms | 100.0 % | 100.0 % | 0.51 m | 75 MB | REAL |
| p2plane · laptop-full newer_college_bag9 · phase5 · 5539c47 · uncommitted changes | 270 | 734.2 ms | 921.7 ms | 996.8 ms | 1232.5 ms | 1770.1 ms | 100.0 % | 100.0 % | 0.15 m | 97 MB | REAL |
| baseline · laptop-full oxford_spires · phase5 · 5539c47 · uncommitted changes | 300 | 522.3 ms | 806.1 ms | 921.5 ms | 1294.8 ms | 2433.4 ms | 100.0 % | 100.0 % | 1.32 m | 75 MB | REAL |
| baseline · laptop-4c oxford_spires · phase5 · 5539c47 · uncommitted changes | 300 | 437.8 ms | 644.1 ms | 756.7 ms | 1067.9 ms | 1720.4 ms | 100.0 % | 100.0 % | 1.32 m | 74 MB | REAL |
| baseline · laptop-1c oxford_spires · phase5 · 5539c47 · uncommitted changes | 300 | 779.5 ms | 1466.3 ms | 2103.4 ms | 2936.4 ms | 4978.1 ms | 100.0 % | 100.0 % | 1.32 m | 74 MB | REAL |
| p2plane · laptop-full oxford_spires · phase5 · 5539c47 · uncommitted changes | 300 | 517.6 ms | 619.4 ms | 657.8 ms | 794.0 ms | 2876.8 ms | 100.0 % | 100.0 % | 0.15 m | 86 MB | REAL |
| optimized · laptop-full newer_college_bag9 · phase5-v2 · 2917b44 | 270 | 323.5 ms | 765.8 ms | 1017.9 ms | 1520.8 ms | 3190.3 ms | 97.2 % | 100.0 % | 0.51 m | 80 MB | REAL |
| optimized · laptop-4c newer_college_bag9 · phase5-v2 · 2917b44 | 270 | 611.4 ms | 1349.4 ms | 1746.0 ms | 2693.6 ms | 6835.5 ms | 99.9 % | 100.0 % | 0.51 m | 76 MB | REAL |
| optimized · laptop-1c newer_college_bag9 · phase5-v2 · 2917b44 | 270 | 869.8 ms | 1633.4 ms | 2090.3 ms | 2808.4 ms | 4922.3 ms | 99.9 % | 100.0 % | 0.51 m | 76 MB | REAL |
| optimized · laptop-full oxford_spires · phase5-v2 · 2917b44 · uncommitted changes | 300 | 247.5 ms | 375.2 ms | 456.7 ms | 1055.6 ms | 3694.6 ms | 98.8 % | 100.0 % | 1.32 m | 76 MB | REAL |
| baseline · laptop-full newer_college_bag9 · final · e9036cd | 270 | 887.5 ms | 1602.3 ms | 2069.0 ms | 4124.3 ms | 8970.7 ms | 100.0 % | 100.0 % | 0.51 m | 78 MB | REAL |
| optimized · laptop-full newer_college_bag9 · final · e9036cd | 270 | 505.8 ms | 1803.3 ms | 2301.0 ms | 3642.5 ms | 10755.3 ms | 98.8 % | 100.0 % | 0.51 m | 79 MB | REAL |
Single-frame preprocessing (NON-CONSECUTIVE frames)
Median 143 ms, P95 186 ms per frame for invalid-point removal, range filter and voxel downsample; voxel stage alone 133 ms; 29918 points after voxel filter (median). No odometry or accuracy claim.
Scaling across hardware profiles
newer_college_bag9 · baseline
- laptop-full vs laptop-4c: per-repeat median ranges overlap (674-839 ms vs 649-739 ms); no difference is claimed.
- laptop-4c vs laptop-1c: per-repeat median ranges overlap (649-739 ms vs 650-676 ms); no difference is claimed.
- Profiles were measured at different times on a shared, non-isolated laptop; a non-monotonic ordering across profiles reflects machine drift as much as core count.
- [ASSUMED, one-way inference, not measured] It misses the deadline on 4 laptop cores, so a Raspberry Pi 5 very likely misses it too.
oxford_spires · baseline
- laptop-4c is faster than laptop-full beyond run-to-run spread (median 438 ms vs 522 ms).
- laptop-4c is faster than laptop-1c beyond run-to-run spread (median 438 ms vs 779 ms).
- Profiles were measured at different times on a shared, non-isolated laptop; a non-monotonic ordering across profiles reflects machine drift as much as core count.
- [ASSUMED, one-way inference, not measured] It misses the deadline on 4 laptop cores, so a Raspberry Pi 5 very likely misses it too.
newer_college_bag9 · optimized
- laptop-full is faster than laptop-4c beyond run-to-run spread (median 324 ms vs 611 ms).
- laptop-4c is faster than laptop-1c beyond run-to-run spread (median 611 ms vs 870 ms).
- Profiles were measured at different times on a shared, non-isolated laptop; a non-monotonic ordering across profiles reflects machine drift as much as core count.
- [ASSUMED, one-way inference, not measured] It misses the deadline on 4 laptop cores, so a Raspberry Pi 5 very likely misses it too.
Verdicts
baseline · laptop-full · newer_college_bag9
FAIL
No. 100.0 % of 798 frames missed 100 ms. Median latency 750 ms is 7.5x the deadline; P99 is 2112 ms.
Rule and evidence
pass: 0 violations; warn: 0 < violations <= 5 %; fail: violations > 5 %. Violation = per-frame pipeline latency (preprocess + register) strictly greater than the deadline.
{
"deadline_ms": 100,
"p50_ms": 749.8913,
"p95_ms": 1547.597945,
"p99_ms": 2111.641545999998,
"max_ms": 2684.1567,
"violation_percent": 100,
"frames_measured": 798
}PASS
269 of 269 registrations passed the validity gate (100.0 %).
Rule and evidence
pass: >= 99 % accepted; warn: >= 90 %; fail: < 90 %. Accepted = passes the registration validity gate.
{
"attempts": 269,
"accepted": 269,
"rejected": 0,
"rejected_by_reason": {},
"icp_iterations_median": 10,
"inlier_ratio_median": 0.99936,
"residual_median_m": 0.12463
}INFO
ATE RMSE 0.51 m (max 1.06 m) over 22.5 m of ground-truth path after rigid alignment; position RPE is reported per window in the summary. Both include any unmodelled sensor-to-body offset (lever arm) between the estimate and the ground-truth body frame, so they mix drift with extrinsic bias. Scan-to-scan odometry has no loop closure, so error accumulates with distance.
Rule and evidence
descriptive; no pass/fail threshold is applied to accuracy
{
"ate_rmse_m": 0.5068686215458409,
"ate_max_m": 1.0565121346857196,
"gt_path_length_m": 22.472984119767467,
"ate_rmse_percent_of_path": 2.2554575700518296,
"final_error_m": 0.19201671534112752,
"rpe_mode": "world_position_after_umeyama_alignment",
"rpe": {
"time_1s": 28.56793327591613,
"time_5s": 14.567642505063228,
"distance_5m": 10.553605423564546,
"distance_10m": 7.834970613283673
}
}baseline · laptop-4c · newer_college_bag9
FAIL
No. 100.0 % of 798 frames missed 100 ms. Median latency 676 ms is 6.8x the deadline; P99 is 1816 ms.
Rule and evidence
pass: 0 violations; warn: 0 < violations <= 5 %; fail: violations > 5 %. Violation = per-frame pipeline latency (preprocess + register) strictly greater than the deadline.
{
"deadline_ms": 100,
"p50_ms": 676.2093500000001,
"p95_ms": 1309.6312149999999,
"p99_ms": 1816.2018479999997,
"max_ms": 2328.1782,
"violation_percent": 100,
"frames_measured": 798
}PASS
269 of 269 registrations passed the validity gate (100.0 %).
Rule and evidence
pass: >= 99 % accepted; warn: >= 90 %; fail: < 90 %. Accepted = passes the registration validity gate.
{
"attempts": 269,
"accepted": 269,
"rejected": 0,
"rejected_by_reason": {},
"icp_iterations_median": 10,
"inlier_ratio_median": 0.99936,
"residual_median_m": 0.12463
}INFO
ATE RMSE 0.51 m (max 1.06 m) over 22.5 m of ground-truth path after rigid alignment; position RPE is reported per window in the summary. Both include any unmodelled sensor-to-body offset (lever arm) between the estimate and the ground-truth body frame, so they mix drift with extrinsic bias. Scan-to-scan odometry has no loop closure, so error accumulates with distance.
Rule and evidence
descriptive; no pass/fail threshold is applied to accuracy
{
"ate_rmse_m": 0.5068686215458409,
"ate_max_m": 1.0565121346857196,
"gt_path_length_m": 22.472984119767467,
"ate_rmse_percent_of_path": 2.2554575700518296,
"final_error_m": 0.19201671534112752,
"rpe_mode": "world_position_after_umeyama_alignment",
"rpe": {
"time_1s": 28.56793327591613,
"time_5s": 14.567642505063228,
"distance_5m": 10.553605423564546,
"distance_10m": 7.834970613283673
}
}baseline · laptop-1c · newer_college_bag9
FAIL
No. 100.0 % of 798 frames missed 100 ms. Median latency 672 ms is 6.7x the deadline; P99 is 1864 ms.
Rule and evidence
pass: 0 violations; warn: 0 < violations <= 5 %; fail: violations > 5 %. Violation = per-frame pipeline latency (preprocess + register) strictly greater than the deadline.
{
"deadline_ms": 100,
"p50_ms": 671.51435,
"p95_ms": 1367.2325399999995,
"p99_ms": 1864.297877,
"max_ms": 2318.7079,
"violation_percent": 100,
"frames_measured": 798
}PASS
269 of 269 registrations passed the validity gate (100.0 %).
Rule and evidence
pass: >= 99 % accepted; warn: >= 90 %; fail: < 90 %. Accepted = passes the registration validity gate.
{
"attempts": 269,
"accepted": 269,
"rejected": 0,
"rejected_by_reason": {},
"icp_iterations_median": 10,
"inlier_ratio_median": 0.99936,
"residual_median_m": 0.12463
}INFO
ATE RMSE 0.51 m (max 1.06 m) over 22.5 m of ground-truth path after rigid alignment; position RPE is reported per window in the summary. Both include any unmodelled sensor-to-body offset (lever arm) between the estimate and the ground-truth body frame, so they mix drift with extrinsic bias. Scan-to-scan odometry has no loop closure, so error accumulates with distance.
Rule and evidence
descriptive; no pass/fail threshold is applied to accuracy
{
"ate_rmse_m": 0.5068686215458409,
"ate_max_m": 1.0565121346857196,
"gt_path_length_m": 22.472984119767467,
"ate_rmse_percent_of_path": 2.2554575700518296,
"final_error_m": 0.19201671534112752,
"rpe_mode": "world_position_after_umeyama_alignment",
"rpe": {
"time_1s": 28.56793327591613,
"time_5s": 14.567642505063228,
"distance_5m": 10.553605423564546,
"distance_10m": 7.834970613283673
}
}p2plane · laptop-full · newer_college_bag9
FAIL
No. 100.0 % of 798 frames missed 100 ms. Median latency 734 ms is 7.3x the deadline; P99 is 1232 ms.
Rule and evidence
pass: 0 violations; warn: 0 < violations <= 5 %; fail: violations > 5 %. Violation = per-frame pipeline latency (preprocess + register) strictly greater than the deadline.
{
"deadline_ms": 100,
"p50_ms": 734.24235,
"p95_ms": 996.8155599999999,
"p99_ms": 1232.468592,
"max_ms": 1770.1241,
"violation_percent": 100,
"frames_measured": 798
}PASS
269 of 269 registrations passed the validity gate (100.0 %).
Rule and evidence
pass: >= 99 % accepted; warn: >= 90 %; fail: < 90 %. Accepted = passes the registration validity gate.
{
"attempts": 269,
"accepted": 269,
"rejected": 0,
"rejected_by_reason": {},
"icp_iterations_median": 5,
"inlier_ratio_median": 0.99932,
"residual_median_m": 0.1277
}INFO
ATE RMSE 0.15 m (max 0.35 m) over 22.5 m of ground-truth path after rigid alignment; position RPE is reported per window in the summary. Both include any unmodelled sensor-to-body offset (lever arm) between the estimate and the ground-truth body frame, so they mix drift with extrinsic bias. Scan-to-scan odometry has no loop closure, so error accumulates with distance.
Rule and evidence
descriptive; no pass/fail threshold is applied to accuracy
{
"ate_rmse_m": 0.15113568120206913,
"ate_max_m": 0.3455220968441755,
"gt_path_length_m": 22.472984119767467,
"ate_rmse_percent_of_path": 0.6725216393008022,
"final_error_m": 0.13946928656653032,
"rpe_mode": "world_position_after_umeyama_alignment",
"rpe": {
"time_1s": 5.694847694836872,
"time_5s": 3.0709409587105023,
"distance_5m": 2.9858696453402276,
"distance_10m": 2.528365538024447
}
}baseline · laptop-full · oxford_spires
FAIL
No. 100.0 % of 888 frames missed 100 ms. Median latency 522 ms is 5.2x the deadline; P99 is 1295 ms.
Rule and evidence
pass: 0 violations; warn: 0 < violations <= 5 %; fail: violations > 5 %. Violation = per-frame pipeline latency (preprocess + register) strictly greater than the deadline.
{
"deadline_ms": 100,
"p50_ms": 522.26365,
"p95_ms": 921.46106,
"p99_ms": 1294.765847,
"max_ms": 2433.4405,
"violation_percent": 100,
"frames_measured": 888
}PASS
299 of 299 registrations passed the validity gate (100.0 %).
Rule and evidence
pass: >= 99 % accepted; warn: >= 90 %; fail: < 90 %. Accepted = passes the registration validity gate.
{
"attempts": 299,
"accepted": 299,
"rejected": 0,
"rejected_by_reason": {},
"icp_iterations_median": 10,
"inlier_ratio_median": 0.99795,
"residual_median_m": 0.15885
}INFO
ATE RMSE 1.32 m (max 2.79 m) over 45.6 m of ground-truth path after rigid alignment; position RPE is reported per window in the summary. Both include any unmodelled sensor-to-body offset (lever arm) between the estimate and the ground-truth body frame, so they mix drift with extrinsic bias. Scan-to-scan odometry has no loop closure, so error accumulates with distance.
Rule and evidence
descriptive; no pass/fail threshold is applied to accuracy
{
"ate_rmse_m": 1.3217205014603846,
"ate_max_m": 2.791091479410538,
"gt_path_length_m": 45.563618073800534,
"ate_rmse_percent_of_path": 2.900824292134924,
"final_error_m": 1.2982540736844619,
"rpe_mode": "world_position_after_umeyama_alignment",
"rpe": {
"time_1s": 15.048729850924103,
"time_5s": 11.731982600999217,
"distance_5m": 12.273713131830242,
"distance_10m": 11.389612695757071
}
}WARN
No. This dataset uses a different sensor (Hesai QT64 (64 beams) - DIFFERENT from OS1-64 (beam layout, FOV, range)); do not read it as an OS1-64 result.
Rule and evidence
dataset.same_sensor_as_target is false
{}baseline · laptop-4c · oxford_spires
FAIL
No. 100.0 % of 888 frames missed 100 ms. Median latency 438 ms is 4.4x the deadline; P99 is 1068 ms.
Rule and evidence
pass: 0 violations; warn: 0 < violations <= 5 %; fail: violations > 5 %. Violation = per-frame pipeline latency (preprocess + register) strictly greater than the deadline.
{
"deadline_ms": 100,
"p50_ms": 437.8044,
"p95_ms": 756.6902749999999,
"p99_ms": 1067.862499,
"max_ms": 1720.3739,
"violation_percent": 100,
"frames_measured": 888
}PASS
299 of 299 registrations passed the validity gate (100.0 %).
Rule and evidence
pass: >= 99 % accepted; warn: >= 90 %; fail: < 90 %. Accepted = passes the registration validity gate.
{
"attempts": 299,
"accepted": 299,
"rejected": 0,
"rejected_by_reason": {},
"icp_iterations_median": 10,
"inlier_ratio_median": 0.99795,
"residual_median_m": 0.15885
}INFO
ATE RMSE 1.32 m (max 2.79 m) over 45.6 m of ground-truth path after rigid alignment; position RPE is reported per window in the summary. Both include any unmodelled sensor-to-body offset (lever arm) between the estimate and the ground-truth body frame, so they mix drift with extrinsic bias. Scan-to-scan odometry has no loop closure, so error accumulates with distance.
Rule and evidence
descriptive; no pass/fail threshold is applied to accuracy
{
"ate_rmse_m": 1.3217205014603846,
"ate_max_m": 2.791091479410538,
"gt_path_length_m": 45.563618073800534,
"ate_rmse_percent_of_path": 2.900824292134924,
"final_error_m": 1.2982540736844619,
"rpe_mode": "world_position_after_umeyama_alignment",
"rpe": {
"time_1s": 15.048729850924103,
"time_5s": 11.731982600999217,
"distance_5m": 12.273713131830242,
"distance_10m": 11.389612695757071
}
}WARN
No. This dataset uses a different sensor (Hesai QT64 (64 beams) - DIFFERENT from OS1-64 (beam layout, FOV, range)); do not read it as an OS1-64 result.
Rule and evidence
dataset.same_sensor_as_target is false
{}baseline · laptop-1c · oxford_spires
FAIL
No. 100.0 % of 888 frames missed 100 ms. Median latency 779 ms is 7.8x the deadline; P99 is 2936 ms.
Rule and evidence
pass: 0 violations; warn: 0 < violations <= 5 %; fail: violations > 5 %. Violation = per-frame pipeline latency (preprocess + register) strictly greater than the deadline.
{
"deadline_ms": 100,
"p50_ms": 779.4844,
"p95_ms": 2103.4184499999997,
"p99_ms": 2936.4477319999996,
"max_ms": 4978.1011,
"violation_percent": 100,
"frames_measured": 888
}PASS
299 of 299 registrations passed the validity gate (100.0 %).
Rule and evidence
pass: >= 99 % accepted; warn: >= 90 %; fail: < 90 %. Accepted = passes the registration validity gate.
{
"attempts": 299,
"accepted": 299,
"rejected": 0,
"rejected_by_reason": {},
"icp_iterations_median": 10,
"inlier_ratio_median": 0.99795,
"residual_median_m": 0.15885
}INFO
ATE RMSE 1.32 m (max 2.79 m) over 45.6 m of ground-truth path after rigid alignment; position RPE is reported per window in the summary. Both include any unmodelled sensor-to-body offset (lever arm) between the estimate and the ground-truth body frame, so they mix drift with extrinsic bias. Scan-to-scan odometry has no loop closure, so error accumulates with distance.
Rule and evidence
descriptive; no pass/fail threshold is applied to accuracy
{
"ate_rmse_m": 1.3217205014603846,
"ate_max_m": 2.791091479410538,
"gt_path_length_m": 45.563618073800534,
"ate_rmse_percent_of_path": 2.900824292134924,
"final_error_m": 1.2982540736844619,
"rpe_mode": "world_position_after_umeyama_alignment",
"rpe": {
"time_1s": 15.048729850924103,
"time_5s": 11.731982600999217,
"distance_5m": 12.273713131830242,
"distance_10m": 11.389612695757071
}
}WARN
No. This dataset uses a different sensor (Hesai QT64 (64 beams) - DIFFERENT from OS1-64 (beam layout, FOV, range)); do not read it as an OS1-64 result.
Rule and evidence
dataset.same_sensor_as_target is false
{}p2plane · laptop-full · oxford_spires
FAIL
No. 100.0 % of 888 frames missed 100 ms. Median latency 518 ms is 5.2x the deadline; P99 is 794 ms.
Rule and evidence
pass: 0 violations; warn: 0 < violations <= 5 %; fail: violations > 5 %. Violation = per-frame pipeline latency (preprocess + register) strictly greater than the deadline.
{
"deadline_ms": 100,
"p50_ms": 517.58795,
"p95_ms": 657.842765,
"p99_ms": 794.00649,
"max_ms": 2876.7601,
"violation_percent": 100,
"frames_measured": 888
}PASS
299 of 299 registrations passed the validity gate (100.0 %).
Rule and evidence
pass: >= 99 % accepted; warn: >= 90 %; fail: < 90 %. Accepted = passes the registration validity gate.
{
"attempts": 299,
"accepted": 299,
"rejected": 0,
"rejected_by_reason": {},
"icp_iterations_median": 6,
"inlier_ratio_median": 0.99794,
"residual_median_m": 0.16729
}INFO
ATE RMSE 0.15 m (max 0.42 m) over 45.6 m of ground-truth path after rigid alignment; position RPE is reported per window in the summary. Both include any unmodelled sensor-to-body offset (lever arm) between the estimate and the ground-truth body frame, so they mix drift with extrinsic bias. Scan-to-scan odometry has no loop closure, so error accumulates with distance.
Rule and evidence
descriptive; no pass/fail threshold is applied to accuracy
{
"ate_rmse_m": 0.15251217145307253,
"ate_max_m": 0.4168300997306193,
"gt_path_length_m": 45.563618073800534,
"ate_rmse_percent_of_path": 0.3347235753009007,
"final_error_m": 0.15346740111930274,
"rpe_mode": "world_position_after_umeyama_alignment",
"rpe": {
"time_1s": 3.924105717352643,
"time_5s": 1.5048470621703773,
"distance_5m": 1.7766911665036935,
"distance_10m": 1.3573782967742307
}
}WARN
No. This dataset uses a different sensor (Hesai QT64 (64 beams) - DIFFERENT from OS1-64 (beam layout, FOV, range)); do not read it as an OS1-64 result.
Rule and evidence
dataset.same_sensor_as_target is false
{}optimized · laptop-full · newer_college_bag9
FAIL
No. 97.2 % of 798 frames missed 100 ms. Median latency 324 ms is 3.2x the deadline; P99 is 1521 ms.
Rule and evidence
pass: 0 violations; warn: 0 < violations <= 5 %; fail: violations > 5 %. Violation = per-frame pipeline latency (preprocess + register) strictly greater than the deadline.
{
"deadline_ms": 100,
"p50_ms": 323.54755,
"p95_ms": 1017.8753149999999,
"p99_ms": 1520.7788489999996,
"max_ms": 3190.3493,
"violation_percent": 97.24310776942356,
"frames_measured": 798
}PASS
269 of 269 registrations passed the validity gate (100.0 %).
Rule and evidence
pass: >= 99 % accepted; warn: >= 90 %; fail: < 90 %. Accepted = passes the registration validity gate.
{
"attempts": 269,
"accepted": 269,
"rejected": 0,
"rejected_by_reason": {},
"icp_iterations_median": 10,
"inlier_ratio_median": 0.99936,
"residual_median_m": 0.12463
}INFO
ATE RMSE 0.51 m (max 1.06 m) over 22.5 m of ground-truth path after rigid alignment; position RPE is reported per window in the summary. Both include any unmodelled sensor-to-body offset (lever arm) between the estimate and the ground-truth body frame, so they mix drift with extrinsic bias. Scan-to-scan odometry has no loop closure, so error accumulates with distance.
Rule and evidence
descriptive; no pass/fail threshold is applied to accuracy
{
"ate_rmse_m": 0.5068686215458409,
"ate_max_m": 1.0565121346857196,
"gt_path_length_m": 22.472984119767467,
"ate_rmse_percent_of_path": 2.2554575700518296,
"final_error_m": 0.19201671534112752,
"rpe_mode": "world_position_after_umeyama_alignment",
"rpe": {
"time_1s": 28.56793327591613,
"time_5s": 14.567642505063228,
"distance_5m": 10.553605423564546,
"distance_10m": 7.834970613283673
}
}optimized · laptop-4c · newer_college_bag9
FAIL
No. 99.9 % of 798 frames missed 100 ms. Median latency 611 ms is 6.1x the deadline; P99 is 2694 ms.
Rule and evidence
pass: 0 violations; warn: 0 < violations <= 5 %; fail: violations > 5 %. Violation = per-frame pipeline latency (preprocess + register) strictly greater than the deadline.
{
"deadline_ms": 100,
"p50_ms": 611.3552999999999,
"p95_ms": 1746.0251549999998,
"p99_ms": 2693.608919,
"max_ms": 6835.5067,
"violation_percent": 99.87468671679198,
"frames_measured": 798
}PASS
269 of 269 registrations passed the validity gate (100.0 %).
Rule and evidence
pass: >= 99 % accepted; warn: >= 90 %; fail: < 90 %. Accepted = passes the registration validity gate.
{
"attempts": 269,
"accepted": 269,
"rejected": 0,
"rejected_by_reason": {},
"icp_iterations_median": 10,
"inlier_ratio_median": 0.99936,
"residual_median_m": 0.12463
}INFO
ATE RMSE 0.51 m (max 1.06 m) over 22.5 m of ground-truth path after rigid alignment; position RPE is reported per window in the summary. Both include any unmodelled sensor-to-body offset (lever arm) between the estimate and the ground-truth body frame, so they mix drift with extrinsic bias. Scan-to-scan odometry has no loop closure, so error accumulates with distance.
Rule and evidence
descriptive; no pass/fail threshold is applied to accuracy
{
"ate_rmse_m": 0.5068686215458409,
"ate_max_m": 1.0565121346857196,
"gt_path_length_m": 22.472984119767467,
"ate_rmse_percent_of_path": 2.2554575700518296,
"final_error_m": 0.19201671534112752,
"rpe_mode": "world_position_after_umeyama_alignment",
"rpe": {
"time_1s": 28.56793327591613,
"time_5s": 14.567642505063228,
"distance_5m": 10.553605423564546,
"distance_10m": 7.834970613283673
}
}optimized · laptop-1c · newer_college_bag9
FAIL
No. 99.9 % of 798 frames missed 100 ms. Median latency 870 ms is 8.7x the deadline; P99 is 2808 ms.
Rule and evidence
pass: 0 violations; warn: 0 < violations <= 5 %; fail: violations > 5 %. Violation = per-frame pipeline latency (preprocess + register) strictly greater than the deadline.
{
"deadline_ms": 100,
"p50_ms": 869.7582,
"p95_ms": 2090.321995,
"p99_ms": 2808.401579999999,
"max_ms": 4922.3288,
"violation_percent": 99.87468671679198,
"frames_measured": 798
}PASS
269 of 269 registrations passed the validity gate (100.0 %).
Rule and evidence
pass: >= 99 % accepted; warn: >= 90 %; fail: < 90 %. Accepted = passes the registration validity gate.
{
"attempts": 269,
"accepted": 269,
"rejected": 0,
"rejected_by_reason": {},
"icp_iterations_median": 10,
"inlier_ratio_median": 0.99936,
"residual_median_m": 0.12463
}INFO
ATE RMSE 0.51 m (max 1.06 m) over 22.5 m of ground-truth path after rigid alignment; position RPE is reported per window in the summary. Both include any unmodelled sensor-to-body offset (lever arm) between the estimate and the ground-truth body frame, so they mix drift with extrinsic bias. Scan-to-scan odometry has no loop closure, so error accumulates with distance.
Rule and evidence
descriptive; no pass/fail threshold is applied to accuracy
{
"ate_rmse_m": 0.5068686215458409,
"ate_max_m": 1.0565121346857196,
"gt_path_length_m": 22.472984119767467,
"ate_rmse_percent_of_path": 2.2554575700518296,
"final_error_m": 0.19201671534112752,
"rpe_mode": "world_position_after_umeyama_alignment",
"rpe": {
"time_1s": 28.56793327591613,
"time_5s": 14.567642505063228,
"distance_5m": 10.553605423564546,
"distance_10m": 7.834970613283673
}
}optimized · laptop-full · oxford_spires
FAIL
No. 98.8 % of 888 frames missed 100 ms. Median latency 248 ms is 2.5x the deadline; P99 is 1056 ms.
Rule and evidence
pass: 0 violations; warn: 0 < violations <= 5 %; fail: violations > 5 %. Violation = per-frame pipeline latency (preprocess + register) strictly greater than the deadline.
{
"deadline_ms": 100,
"p50_ms": 247.53379999999999,
"p95_ms": 456.6521599999999,
"p99_ms": 1055.5644659999998,
"max_ms": 3694.6012,
"violation_percent": 98.76126126126125,
"frames_measured": 888
}PASS
299 of 299 registrations passed the validity gate (100.0 %).
Rule and evidence
pass: >= 99 % accepted; warn: >= 90 %; fail: < 90 %. Accepted = passes the registration validity gate.
{
"attempts": 299,
"accepted": 299,
"rejected": 0,
"rejected_by_reason": {},
"icp_iterations_median": 10,
"inlier_ratio_median": 0.99795,
"residual_median_m": 0.15885
}INFO
ATE RMSE 1.32 m (max 2.79 m) over 45.6 m of ground-truth path after rigid alignment; position RPE is reported per window in the summary. Both include any unmodelled sensor-to-body offset (lever arm) between the estimate and the ground-truth body frame, so they mix drift with extrinsic bias. Scan-to-scan odometry has no loop closure, so error accumulates with distance.
Rule and evidence
descriptive; no pass/fail threshold is applied to accuracy
{
"ate_rmse_m": 1.3217205014603846,
"ate_max_m": 2.791091479410538,
"gt_path_length_m": 45.563618073800534,
"ate_rmse_percent_of_path": 2.900824292134924,
"final_error_m": 1.2982540736844619,
"rpe_mode": "world_position_after_umeyama_alignment",
"rpe": {
"time_1s": 15.048729850924103,
"time_5s": 11.731982600999217,
"distance_5m": 12.273713131830242,
"distance_10m": 11.389612695757071
}
}WARN
No. This dataset uses a different sensor (Hesai QT64 (64 beams) - DIFFERENT from OS1-64 (beam layout, FOV, range)); do not read it as an OS1-64 result.
Rule and evidence
dataset.same_sensor_as_target is false
{}baseline · laptop-full · newer_college_bag9
FAIL
No. 100.0 % of 1330 frames missed 100 ms. Median latency 887 ms is 8.9x the deadline; P99 is 4124 ms.
Rule and evidence
pass: 0 violations; warn: 0 < violations <= 5 %; fail: violations > 5 %. Violation = per-frame pipeline latency (preprocess + register) strictly greater than the deadline.
{
"deadline_ms": 100,
"p50_ms": 887.4585,
"p95_ms": 2069.0244699999994,
"p99_ms": 4124.290643000001,
"max_ms": 8970.7056,
"violation_percent": 100,
"frames_measured": 1330
}PASS
269 of 269 registrations passed the validity gate (100.0 %).
Rule and evidence
pass: >= 99 % accepted; warn: >= 90 %; fail: < 90 %. Accepted = passes the registration validity gate.
{
"attempts": 269,
"accepted": 269,
"rejected": 0,
"rejected_by_reason": {},
"icp_iterations_median": 10,
"inlier_ratio_median": 0.99936,
"residual_median_m": 0.12463
}INFO
ATE RMSE 0.51 m (max 1.06 m) over 22.5 m of ground-truth path after rigid alignment; position RPE is reported per window in the summary. Both include any unmodelled sensor-to-body offset (lever arm) between the estimate and the ground-truth body frame, so they mix drift with extrinsic bias. Scan-to-scan odometry has no loop closure, so error accumulates with distance.
Rule and evidence
descriptive; no pass/fail threshold is applied to accuracy
{
"ate_rmse_m": 0.5068686215458409,
"ate_max_m": 1.0565121346857196,
"gt_path_length_m": 22.472984119767467,
"ate_rmse_percent_of_path": 2.2554575700518296,
"final_error_m": 0.19201671534112752,
"rpe_mode": "world_position_after_umeyama_alignment",
"rpe": {
"time_1s": 28.56793327591613,
"time_5s": 14.567642505063228,
"distance_5m": 10.553605423564546,
"distance_10m": 7.834970613283673
}
}optimized · laptop-full · newer_college_bag9
FAIL
No. 98.8 % of 1330 frames missed 100 ms. Median latency 506 ms is 5.1x the deadline; P99 is 3642 ms.
Rule and evidence
pass: 0 violations; warn: 0 < violations <= 5 %; fail: violations > 5 %. Violation = per-frame pipeline latency (preprocess + register) strictly greater than the deadline.
{
"deadline_ms": 100,
"p50_ms": 505.83180000000004,
"p95_ms": 2301.0204099999996,
"p99_ms": 3642.4675620000053,
"max_ms": 10755.3086,
"violation_percent": 98.796992481203,
"frames_measured": 1330
}PASS
269 of 269 registrations passed the validity gate (100.0 %).
Rule and evidence
pass: >= 99 % accepted; warn: >= 90 %; fail: < 90 %. Accepted = passes the registration validity gate.
{
"attempts": 269,
"accepted": 269,
"rejected": 0,
"rejected_by_reason": {},
"icp_iterations_median": 10,
"inlier_ratio_median": 0.99936,
"residual_median_m": 0.12463
}INFO
ATE RMSE 0.51 m (max 1.06 m) over 22.5 m of ground-truth path after rigid alignment; position RPE is reported per window in the summary. Both include any unmodelled sensor-to-body offset (lever arm) between the estimate and the ground-truth body frame, so they mix drift with extrinsic bias. Scan-to-scan odometry has no loop closure, so error accumulates with distance.
Rule and evidence
descriptive; no pass/fail threshold is applied to accuracy
{
"ate_rmse_m": 0.5068686215458409,
"ate_max_m": 1.0565121346857196,
"gt_path_length_m": 22.472984119767467,
"ate_rmse_percent_of_path": 2.2554575700518296,
"final_error_m": 0.19201671534112752,
"rpe_mode": "world_position_after_umeyama_alignment",
"rpe": {
"time_1s": 28.56793327591613,
"time_5s": 14.567642505063228,
"distance_5m": 10.553605423564546,
"distance_10m": 7.834970613283673
}
}single-frame · newer_college_sparse
FAIL
Preprocessing (invalid-point removal, range filter, voxel downsample) takes a median 143 ms (P95 186 ms); the voxel stage alone is 133 ms. That already exceeds the deadline before any registration. Non-consecutive frames: no odometry or accuracy claim.
Rule and evidence
fail if the median preprocessing latency exceeds the deadline
{
"deadline_ms": 100,
"preprocess_p50_ms": 142.6959,
"preprocess_p95_ms": 186.46445,
"voxel_p50_ms": 133.3462,
"frames": 591
}Metric definitions
| Metric | Unit | Evidence | Definition | Calculation | Interpretation |
|---|---|---|---|---|---|
| Pipeline latency | ms | MEASURED | Wall-clock time to process one frame that is already in memory: invalid-point removal, range filter, voxel downsample, ICP registration, validity gate and pose update. | time.perf_counter around each stage in the worker process; total = preprocess + register. Warm-up frames and the initial reference frame are excluded. | Compare with the sensor period (100 ms at 10 Hz). Single-machine software timing; not a hard real-time guarantee. |
| End-to-end latency | ms | MEASURED | Pipeline latency plus the time to read the frame from disk. | load_ms + total_ms. | A live sensor driver delivers frames in memory, so total_ms is the primary number; e2e shows the file-I/O cost of offline replay. |
| P50 / P90 / P95 / P99 / max | ms | DERIVED | Order statistics of per-frame latency pooled over all repeats. | numpy.percentile, linear interpolation. | P99 and max describe the tail that causes missed frames. |
| Deadline violation rate | % of frames | DERIVED | Share of measured frames whose pipeline latency is strictly greater than the sensor period. | count(total_ms > deadline) / frames measured. | The deadline statistic. A P95 below the deadline still allows 5 % of frames to miss it. |
| Sustained throughput | frames/s | MEASURED | Frames processed per second of wall-clock time over the whole loop, file loading included. | frames / loop wall time per repeat (min, median, max over repeats). | Must exceed the sensor rate (10 fps) to keep up in steady state. |
| Peak memory | MB | MEASURED | Peak resident (working-set) memory of the worker process. | psutil peak_wset (Windows) or ru_maxrss. | Process peak; machine RAM is a separate specification, not a measurement. |
| Accepted registrations | % | DERIVED | Registrations that pass the validity gate (inlier ratio, residual, plausible motion) divided by attempts. | Frame 0 is excluded (it has no registration). Thresholds are in run.json. | Rejections are counted by reason, never silently accepted. |
| Absolute trajectory error (RMSE) | m | DERIVED | RMS position error between the estimated and ground-truth trajectories after a rigid SE(3) Umeyama alignment (no scale), using timestamp-associated poses. | Association within 20 ms, one-to-one. | Needs ground truth. Without loop closure error grows with distance. It also includes any unmodelled sensor-to-body offset (lever arm) between the estimated sensor positions and the ground-truth body positions, so it mixes drift with extrinsic bias. |
| Relative pose error (position) | m, % of segment | DERIVED | Error of the displacement over fixed time or path-length windows, in the aligned world frame. | Rotation RPE is UNAVAILABLE because the ground-truth body frame differs from the sensor frame (DEC-012). Position RPE removes the step-direction (yaw) offset but not a lever arm, and the global alignment is fitted over the whole window, so it is mildly optimistic. | Drift per unit motion; less sensitive to a single bad alignment than ATE. |
| ICP iterations | count | MEASURED | Iterations before the convergence rule stopped (cap 30). | Per registration. | Drives register_ms; hitting the cap flags poor initial alignment. |
| Inlier ratio | fraction | MEASURED | Source points with a target neighbour within 1.5 m, divided by source points. | Last ICP iteration. | Low values mean little scan overlap or a wrong match. |
| Registration residual | m | MEASURED | Mean distance of inlier correspondences at the start of the last iteration. | Point-to-point distance for all variants. | Included in the validity gate. |