Tune hybrid dense and sparse retrieval instead of guessing at the weighting

You are tuning a hybrid retrieval system that combines dense vector similarity with sparse/BM25 keyword scoring. Your job is to propose a fusion method and weighting, and justify it against the real failures below, not to recommend "reciprocal rank fusion, weighted evenly" as a default with no diagnosis behind it.

CURRENT RETRIEVAL SETUP
Pinecone dense retrieval (cosine similarity, top 20) run independently from an Elasticsearch BM25 query (top 20), currently merged by taking the top 10 from each with no deduplication

QUERY FAILURES OBSERVED
Query 'error code E4021' returns zero relevant results from dense retrieval despite the code appearing verbatim in three chunks; query 'how do I cancel a subscription' misses a chunk titled 'Ending your plan' that never uses the word cancel

RERANKING STAGE AVAILABLE
Cohere Rerank 3.5 available, currently unused, budget for one reranking call per query on up to 40 candidates

TOP-K AND LATENCY BUDGET
p95 retrieval latency budget of 400ms total, currently at 180ms for the two independent calls

FUSION METHODS TO WEIGH
1. Weighted linear combination of normalized dense + sparse scores — simple, but normalization across two different score distributions is the actual hard part, not the weight itself.
2. Reciprocal Rank Fusion (RRF) — rank-based, avoids the normalization problem entirely, but discards score magnitude, so it can't distinguish a near-perfect match from a mediocre one at the same rank.
3. Sparse as a pre-filter, dense for final ranking (or vice versa) — cheaper, but only correct if one signal is reliably a superset of the other's relevant results, which needs evidence, not an assumption.

DECISION RULES
- For each failure in the observed failures, classify it: did dense retrieval miss an exact keyword, code, or identifier match that sparse would have caught, or did sparse miss a paraphrase or synonym match that dense would have caught? The weighting decision must be justified against which failure type dominates, not a generic "combine both to be safe."
- If a reranker is available, state explicitly whether the fusion stage's job is now just cheap high-recall candidate generation, in which case fusion can lean permissive with a wider top-k and less precise weighting, versus fusion being the final ranking, in which case weighting precision matters far more since there's no second stage to correct it.
- State the actual score ranges each signal produces on 3-5 real queries from the observed failures, and show what a naive average would have done wrong before proposing the fix.
- Respect the stated latency budget — a fusion method that requires calling both retrievers at k=100 and then reranking is only viable if the budget allows two round trips plus a reranking pass.

OUTPUT FORMAT
1. Chosen fusion method with concrete parameters (RRF constant, or linear weights, or filter threshold).
2. For each observed failure, a one-line prediction of whether the new method fixes it, with the reasoning.
3. The one failure type this fix does NOT address, stated honestly, and what would be needed to fix that one too.