ABLATION_SYSTEM_VERIFICATION.md

Ablation Study System - Complete Wiring Verification ✅

Executive Summary

The ablation study system is fully wired and operational. All feature flags flow correctly from the Streamlit UI through the FastAPI backend to the RAG pipeline with proper cache isolation per phase.

---

1. UI Layer (ui.py)

Phase Selector Creation

✅ Line 278-285: Phase selectbox created in sidebar

phase = st.selectbox(
    "Select experimental phase",
    options=["V1", "V2", "V3", "V4", "V5", "V6"],
    index=5,
    help="V1: Basic RAG | V2: +Query Rewriter | V3: +History | V4: +Hybrid | V5: +Reranker | V6: +Guardrails",
)

Query Payload

✅ Line 345: Phase included in every query request

payload = {
    "question": question,
    "top_k": top_k,
    "use_reranker": use_reranker,
    "use_hybrid": use_hybrid,
    "phase": phase,  # ← SENT WITH EVERY QUERY
    "history": [...]
}

---

2. Backend Request Model (api.py)

QueryRequest Definition

✅ Line 175-180: Phase field properly defined

class QueryRequest(BaseModel):
    question: str
    top_k: int = TOP_K
    use_reranker: bool = True
    use_hybrid: bool = True
    phase: Optional[str] = None        # ← ACCEPTS PHASE FROM UI
    history: Optional[List[ChatMessage]] = None

---

3. Phase Resolution Logic (api.py)

resolve_phase_flags()

✅ Line 184-194: Converts phase string to feature flags

def resolve_phase_flags(phase: Optional[str]) -> Dict[str, bool]:
    if not phase:
        return {}
    
    phase_key = phase.strip().upper()  # "v6" → "V6"
    if phase_key not in PHASE_FLAGS:
        logger.warning("Unknown phase '%s', falling back to runtime toggles.", phase_key)
        return {}
    
    return PHASE_FLAGS[phase_key].copy()  # ← RETURNS DICT OF FLAGS

get_query_settings()

✅ Line 196-209: Routes to correct phase or fallback

def get_query_settings(req: QueryRequest) -> Dict[str, bool]:
    phase_settings = resolve_phase_flags(req.phase)
    if phase_settings:
        return phase_settings  # ← USE PHASE FLAGS IF PROVIDED
    
    # Fallback to individual toggles if no phase specified
    return {
        "use_query_rewriter": True,
        "use_hybrid": req.use_hybrid,
        "use_reranker": req.use_reranker,
        "use_chat_history": True,
        "use_guardrails": False,
    }

---

4. Phase Definitions (config.py)

PHASE_FLAGS Dictionary

✅ config.py lines 27-57: All 6 phases defined

PHASE_FLAGS = {
    "V1": {  # Baseline: No features
        "use_query_rewriter": False,
        "use_hybrid": False,
        "use_chat_history": False,
        "use_reranker": False,
        "use_guardrails": False,
    },
    "V2": {  # +Query Rewriter
        "use_query_rewriter": True,
        "use_hybrid": False,
        "use_chat_history": False,
        "use_reranker": False,
        "use_guardrails": False,
    },
    "V3": {  # +Chat History
        "use_query_rewriter": True,
        "use_hybrid": False,
        "use_chat_history": True,
        "use_reranker": False,
        "use_guardrails": False,
    },
    "V4": {  # +Hybrid Search
        "use_query_rewriter": True,
        "use_hybrid": True,
        "use_chat_history": True,
        "use_reranker": False,
        "use_guardrails": False,
    },
    "V5": {  # +Reranker
        "use_query_rewriter": True,
        "use_hybrid": True,
        "use_chat_history": True,
        "use_reranker": True,
        "use_guardrails": False,
    },
    "V6": {  # +Guardrails
        "use_query_rewriter": True,
        "use_hybrid": True,
        "use_chat_history": True,
        "use_reranker": True,
        "use_guardrails": True,
    },
}

---

5. Query Execution Pipeline (/query endpoint, api.py)

Step 1: Extract Phase Flags

✅ Line 607: Get flags from selected phase

flags = get_query_settings(req)
# flags = {"use_query_rewriter": True, "use_hybrid": True, ...}

Step 2: Chat History Based on Flags

✅ Line 608: Include history only if phase enables it

history = normalize_chat_history(req.history) if flags["use_chat_history"] else []

Step 3: Query Rewriting Based on Flags

✅ Line 609-611: Use query rewriter only if phase enables it

query_text, query_vector = rewrite_and_embed_query(
    question, use_query_rewriter=flags["use_query_rewriter"]  # ← FLAG PASSED
)

• If use_query_rewriter=False → question used as-is
• If use_query_rewriter=True → question rewritten via Gemini

Step 4: Document Search with Flags

✅ Line 612-618: Pass hybrid & reranker flags to search

retrieved = search_documents(
    store,
    query_text,
    query_vector,
    top_k=req.top_k,
    use_hybrid=flags["use_hybrid"],          # ← Semantic + BM25
    use_reranker=flags["use_reranker"],      # ← Sets retrieve_k
)

• If use_reranker=True → retrieve 12 chunks, keep top 6 after reranking
• If use_reranker=False → retrieve exactly top_k chunks

Step 5: Answer Generation with Flags

✅ Line 629-636: Pass reranker & guardrail flags to LLM pipeline

result = generate_answer_payload(question, retrieved, req, flags, history=history)

Where in generate_answer_payload:

def generate_answer_payload(question: str, retrieved: list, req: QueryRequest, flags: Dict[str, bool], ...):
    out = rag_answer(
        question=question,
        retrieved=retrieved,
        min_score=MIN_SCORE,
        use_reranker=flags["use_reranker"],         # ← RERANKING FLAG
        final_top_k=req.top_k,
        use_guardrails=flags["use_guardrails"],     # ← GUARDRAIL FLAG
        history=history,
    )
    # ...

---

6. RAG Pipeline Logic (src/rag_pipeline.py)

Reranking Based on Flag

✅ Lines 75-98: Reranking logic respects flag

if use_reranker and retrieved:
    # Cross-encoder reranks the candidates
    reranker = get_reranker()
    retrieved = reranker.rerank(...)
else:
    logger.info("⛔ Re-ranking DISABLED by user")
    for r in retrieved:
        r["final_score"] = r.get("score", 0.0)

Guardrails Based on Flag

✅ Lines 107-121: Guardrail abstention respects flag

if use_guardrails and retrieved:
    max_score = max((r.get("score", 0.0) for r in retrieved), default=0.0)
    if max_score < min_score:  # min_score = 0.5
        logger.warning("Guardrail triggered: max retrieved score %.3f...", max_score, min_score)
        return {
            "answer": "I don't have enough high-confidence information from the documents...",
            "sources": [],
        }

---

7. Cache Isolation by Phase (src/cache.py)

Cache Key Includes Phase

✅ Line 47: Function signature accepts phase

def get_cache_key(question: str, use_hybrid=True, use_reranker=True, top_k=6, phase: str | None = None):

Phase Baked into Hash

✅ Lines 60-65: Phase included in cache key computation

data = {
    "q": question.strip().lower(),
    "hybrid": bool(use_hybrid),
    "rerank": bool(use_reranker),
    "k": int(top_k),
    "phase": phase.strip().upper() if phase else None,  # ← PHASE IN KEY
}
return hashlib.md5(json.dumps(data, sort_keys=True).encode()).hexdigest()

Cache Key Called with Phase

✅ api.py line 590-596: Cache key generation includes phase

cache_key = get_cache_key(
    question=question,
    use_hybrid=req.use_hybrid,
    use_reranker=req.use_reranker,
    top_k=req.top_k,
    phase=req.phase,  # ← PHASE PASSED
)

Result: Same question returns different cache keys per phase

• cache_key_v1("What is ML?") ≠ cache_key_v6("What is ML?")
• Prevents cross-phase cache contamination ✅

---

8. Feature Disable Verification

Feature	V1	V2	V3	V4	V5	V6	Disabled When
Query Rewriter	❌	✅	✅	✅	✅	✅	Line 609: rewrite_and_embed_query(..., use_query_rewriter=False)
Hybrid Search	❌	❌	❌	✅	✅	✅	Line 617: search_documents(..., use_hybrid=False) → only semantic
Chat History	❌	❌	✅	✅	✅	✅	Line 608: history = [] if flag is False
Reranker	❌	❌	❌	❌	✅	✅	Line 612: use_hybrid=False sets retrieve_k = top_k; Line 85 skips reranking
Guardrails	❌	❌	❌	❌	❌	✅	Line 632: use_guardrails=False → no score threshold check

---

9. Tested Scenarios

All verified in tests/test_phase_flags.py:

✅ Test 1: Phase keys V1-V6 present in PHASE_FLAGS ✅ Test 2: resolve_phase_flags() returns correct dictionaries for each phase ✅ Test 3: get_query_settings() properly resolves phases ✅ Test 4: Cache keys differ per phase (prevents cross-phase hits) ✅ Test 5: Guardrail abstention triggers at score 0.2 (< 0.5 threshold) when enabled ✅ Test 6: Normal LLM flow occurs at score 0.95 even when guardrails enabled ✅ Test 7: Guardrails don't trigger in V5 (guardrails disabled)

---

10. Complete Request Flow Diagram

┌─────────────────────────────────────────────────────────────┐
│ STREAMLIT UI (ui.py)                                         │
│ User selects phase (V1-V6) and asks a question             │
│ Sends: {"question": "...", "phase": "V6", ...}             │
└──────────────────────┬──────────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────────┐
│ FASTAPI /query ENDPOINT (api.py:583)                       │
│ 1. Extract phase from request → "V6"                       │
│ 2. Call get_query_settings(req)                            │
│    → resolve_phase_flags("V6")                             │
│    → Returns PHASE_FLAGS["V6"]                             │
│    → {use_query_rewriter: True, use_hybrid: True, ...}    │
└──────────────────────┬──────────────────────────────────────┘
                       │
         ┌─────────────┼─────────────┐
         ▼             ▼             ▼
    ┌─────────┐  ┌─────────┐  ┌──────────────┐
    │ History │  │ Query   │  │ Search &     │
    │ Filter  │  │ Rewrite │  │ Rerank       │
    │         │  │         │  │              │
    │ use_... │  │ use_... │  │ use_hybrid   │
    │ history │  │ rewriter│  │ use_reranker │
    └────┬────┘  └────┬────┘  └──────┬───────┘
         │             │              │
         ▼             ▼              ▼
   [filtered or ] [rewritten or]  [retrieve+rerank
    full history] original query   or just semantic]
         │             │              │
         └─────────────┼──────────────┘
                       ▼
         ┌──────────────────────────────┐
         │ generate_answer_payload()    │
         │ Calls rag_answer() with:     │
         │ - use_reranker               │
         │ - use_guardrails             │
         └────────┬─────────────────────┘
                  ▼
    ┌──────────────────────────────────┐
    │ rag_pipeline.rag_answer()        │
    │                                  │
    │ If use_reranker:                 │
    │   Cross-encode score candidates  │
    │ If use_guardrails:               │
    │   Check max_score >= 0.5        │
    │   If not → abstain               │
    │                                  │
    │ Generate answer + evaluate       │
    └────────┬─────────────────────────┘
             ▼
  ┌─────────────────────────────┐
  │ Cache with phase in key     │
  │ (prevent cross-phase hits)  │
  └────────┬────────────────────┘
           ▼
  ┌─────────────────────────────┐
  │ Return answer to UI         │
  └─────────────────────────────┘

---

11. Conclusion

✅ System Status: FULLY OPERATIONAL

All wiring verified:

• UI phase selector → sends phase string
• QueryRequest accepts phase
• Phase resolution → correctly maps to PHASE_FLAGS
• Feature flags → properly extracted and passed through pipeline
• Each flag → correctly disables/enables the corresponding feature
• Cache isolation → prevents cross-phase contamination
• Tests → all passing and verified

You can now:

1. Start the backend: python -m uvicorn src.api:app --reload
2. Start the UI: streamlit run ui.py
3. Select phase V1-V6 in the dropdown
4. Submit queries and observe feature behavior change per phase
5. Run full ablation study with high confidence in feature isolation