SQLite Metric Parity Map

This table is intentionally compact: one metric per row.

Legend:

• [x] = metric is stored in both test modes
• [ ] = metric is missing in at least one mode

Notes:

• Capability rows normalize quantization to an uppercase label such as Q4_K_M; classic rows keep the classic benchmark format such as q4_k_m.

• Capability lmstudio_version stores a parsed version or pkg_version (commit:<sha>), not the raw lms version banner output.

• Capability REST runs forward the exact model variant key, including the @quantization suffix, to LM Studio load/chat/unload requests.

• Classic rows intentionally leave capability-only fields such as quality_score, raw_output, reference_output, capability, and test_id empty.

• Historical rows created before recent schema/runtime fixes may still contain NULL values in parity columns. New rows should populate them.

Historical Validation Queries

Use these queries to find older rows that predate parity fixes.

-- Classic rows that still miss parity fields introduced later.
SELECT id, model_name, timestamp,
         quantization, lmstudio_version, app_version, success
FROM benchmark_results
WHERE quantization IS NULL
    OR lmstudio_version IS NULL
    OR app_version IS NULL
    OR success IS NULL
ORDER BY id DESC;

-- Compatibility rows that still miss core parity fields.
SELECT id, model_name, capability, test_id,
         quantization, lmstudio_version, app_version,
         prompt_hash, params_hash
FROM benchmark_results
WHERE source = 'compatibility'
        AND (
            quantization IS NULL
            OR lmstudio_version IS NULL
            OR app_version IS NULL
            OR prompt_hash IS NULL
            OR params_hash IS NULL
        )
ORDER BY id DESC;

-- Compatibility summary directly from benchmark_results.
SELECT model_name,
             capability,
             COUNT(*) AS total_tests,
             SUM(CASE WHEN success = 1 THEN 1 ELSE 0 END) AS successful_tests,
             SUM(CASE WHEN success = 1 THEN 0 ELSE 1 END) AS failed_tests,
             AVG(avg_gen_time) AS avg_latency_ms,
             AVG(throughput_tokens_per_sec) AS avg_throughput,
             AVG(quality_score) AS avg_quality_score,
             AVG(rouge_score) AS avg_rouge,
             AVG(f1_score) AS avg_f1,
             AVG(exact_match_score) AS avg_exact_match,
             AVG(accuracy_score) AS avg_accuracy
FROM benchmark_results
WHERE source = 'compatibility'
GROUP BY model_name, capability
ORDER BY MAX(id) DESC;