Lesson 18 / 28
Comparing Engines: A Practical Checklist
Ask the same questions of every candidate.
By Rahul Agarwal · 15 years in web development
Questions beat feature tables
Feature lists on vendor sites are similar and change monthly, so compare with questions about your workload. Scale: how many vectors and what dimension now and in two years; does it fit in memory; what does it do beyond RAM (disk indexes, quantisation)? Filtering: are filters evaluated during search or after; are payload indexes supported; how does it behave with a 1% filter? Search types: dense only, or sparse/keyword and hybrid with fusion; multi-vector; reranking hooks? Updates: how quickly does a write become searchable; how are deletes and re-indexing handled? Consistency and durability: transactions, write-ahead log, snapshots, replication? Operations: deployment (Kubernetes, managed), backups, monitoring, upgrades. Security: authentication, role-based access, encryption, tenant isolation, audit. Ecosystem and cost: client libraries, framework integrations, licence, support, total price. Score each candidate against your answers.
A scoring sheet
Fill it in with measured results from your own tests, not from marketing pages.
Criterion (weight) Candidate A Candidate B Candidate C how measured
recall@10 at p95 < 50 ms (5) __/5 __/5 __/5 your golden set + load test
filter at 1% selectivity (4) __/5 __/5 __/5 returns 10 rows? latency?
write-to-searchable delay (3) __/5 __/5 __/5 insert, poll until found
ops effort (3) __/5 __/5 __/5 deploy, backup, upgrade drill
security features (4) __/5 __/5 __/5 RBAC, encryption, tenant tests
cost at 2-year scale (4) __/5 __/5 __/5 price sheet + memory estimateQuick check: Why compare engines with questions about your workload rather than feature tables?
- Feature lists look alike and change often; behaviour on your data decides
- Features never matter
- Vendors forbid comparison
- Tables are always wrong
Answer
Feature lists look alike and change often; behaviour on your data decides — Test with your vectors, filters and load before choosing.