FaustDerLegende/RAGVERIFIER

ClaimLens: Multilingual RAG Hallucination Verifier

ClaimLens is a fine-tuned multilingual (English, Hindi, Kannada) claim-level hallucination verifier for Retrieval-Augmented Generation (RAG) pipelines based on xlm-roberta-base.

It evaluates generated answers claim-by-claim against retrieved context passages and classifies each into one of three factual grounding states:

• Correct (Label 0): Retrieved evidence entails the claim.
• Incorrect (Label 1): Retrieved evidence contradicts the claim.
• Unverified (Label 2): Evidence neither confirms nor denies the claim (unsupported / baseless).

---

Model Architecture & Training Protocol

• Base Model: xlm-roberta-base
• Task: Sentence-Pair Classification (<s> claim </s></s> evidence </s>)
• Two-Stage Supervised Fine-Tuning:
  1. Stage A (NLI Warm-up): Trained on 60,000 NLI pairs (20k each for English, Hindi, and Kannada from facebook/xnli and Divyanshu/indicxnli).
  2. Stage B (RAG Fine-Tuning): Fine-tuned on RAGTruth claim-level data (English) + machine-translated Hindi and Kannada subsets, using Class-Weighted Cross-Entropy to mitigate the ~90% majority class skew.

---

Evaluation Results

Two decision rules are reported:

• argmax: the most likely of the three labels.
• tuned T=0.10: predict Correct if P(Correct) ≥ 0.10, otherwise the more likely of Incorrect / Unverified. T was tuned on held-out validation data with RAGTruth's natural class mix (91% Correct) and is stored in config.verifier_threshold. Recommended for real RAG outputs: on the natural English test set it raises macro-F1 from 0.489 to 0.577.

Test set	Rule	macro-F1	F1 Correct	F1 Incorrect	F1 Unverified	hallucination F1	accuracy
en (natural, ~92% Correct)	majority baseline	0.320	0.959	0.000	0.000	0.000	0.922
en (natural, ~92% Correct)	argmax	0.489	0.865	0.285	0.316	0.367	0.766
en (natural, ~92% Correct)	tuned T=0.10	0.577	0.939	0.365	0.427	0.466	0.884
en (balanced 100/class)	majority baseline	0.167	0.500	0.000	0.000	0.000	0.333
en (balanced 100/class)	argmax	0.693	0.700	0.688	0.691	0.822	0.693
en (balanced 100/class)	tuned T=0.10	0.653	0.672	0.663	0.625	0.729	0.657
hi (balanced, translated)	majority baseline	0.167	0.500	0.000	0.000	0.000	0.333
hi (balanced, translated)	argmax	0.609	0.575	0.647	0.604	0.816	0.610
hi (balanced, translated)	tuned T=0.10	0.614	0.643	0.652	0.547	0.731	0.620
kn (balanced, translated)	majority baseline	0.167	0.500	0.000	0.000	0.000	0.333
kn (balanced, translated)	argmax	0.569	0.539	0.614	0.555	0.806	0.570
kn (balanced, translated)	tuned T=0.10	0.550	0.568	0.608	0.473	0.691	0.553

• Compare languages using the balanced rows with argmax (100 claims per class each): English 0.693, Hindi 0.609, Kannada 0.569 macro-F1, vs 0.167 for the majority baseline.
• The natural English test set is about 92% Correct, so its accuracy and hallucination F1 are not comparable with the balanced sets.
• Hindi and Kannada test sets are machine-translated from the RAGTruth test split (NLLB-200), labels copied from English.

Quickstart & Usage

Using Hugging Face Transformers

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "FaustDerLegende/RAGVERIFIER"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

claim = "Antigravity is an AI development platform."
evidence = "Antigravity is an AI development platform created by Deepmind engineers."

# Tokenize sentence-pair (claim, evidence)
inputs = tokenizer(
    claim,
    evidence,
    truncation="only_second",
    max_length=384,
    return_tensors="pt"
)

with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)[0]

id2label = model.config.id2label
for i, prob in enumerate(probs):
    print(f"{id2label[i]}: {prob.item():.4f}")

predicted_class = id2label[torch.argmax(probs).item()]
print(f"
Final Verdict: {predicted_class}")

Using Hugging Face Pipeline

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="FaustDerLegende/RAGVERIFIER",
    tokenizer="FaustDerLegende/RAGVERIFIER"
)

result = classifier({
    "text": "Antigravity is an AI development platform.",
    "text_pair": "Antigravity is an AI development platform created by Deepmind engineers."
})
print(result)

---

Citation & Acknowledgment

Developed as part of the ClaimLens Multilingual Hallucination Detection project, adapting datasets from RAGTruth (Niu et al.), IndicXNLI (Aggarwal et al.), and IndicGLUE.