plan_done.md

Similarity Report Feature — Work Done (Phases 0–4)

This document records everything implemented for the Corpus Similarity Report feature on branch report-ui-updated, per plan.md + opencode-prompt.md. The feature is complete through Phase 4 (API). Phase 5 (PDF highlight mapper integration) was not started.

Superseded 2026-08-02: the remote merge described under "Remaining" is DONE (merge commit 506ad88). All 9 conflicts resolved; suite is 175 passed / 2 failed (both frontend-app/dist env failures). See the post-merge box at the top of plan.md for what the merge changed. Resume at Phase 5.

All 5 commits are local-only (based on 33cb4a2) and have not been pushed. The remote report-ui-updated branch has diverged (20+ commits of Groq-key UI, citation-relevance/novelty agents, frontend work, LF line endings) and contains zero similarity code.

---

Commits

Commit	Description
12cfcb6	feat(similarity): Phase 0 scoring core — normalize, exclusions, winnowing, matcher, aggregate
69c39cd	feat(similarity): Phase 1 internal corpus — write path, read path, privacy
658a727	feat(similarity): Phase 2 external providers — CORE, arXiv, OpenAlex, Crossref
ecc8207	feat(similarity): Phase 3 pipeline + agent
328d8c2	feat(similarity): Phase 4 API — report, recheck, status

Working tree currently contains 212 files of pure CRLF-vs-LF noise (git diff -w collapses them to 0) plus one dirty LFS binary models/kbir_semeval2017/training_args.bin. Plan agreed with user: normalize line endings to LF in a housekeeping commit, then merge the remote.

---

Files added

Similarity package (similarity/)

• __init__.py — package export
• config.py — settings + environment toggles
• schema.py — SimilarityReport, SourceMatch, Coverage, providers
• normalize.py — text normalization
• exclusions.py — stopword/boilerplate exclusion
• fingerprint.py — winnowing fingerprints (k=5, w=4)
• matcher.py — matching + attribution
• aggregate.py — aggregation into report
• index_writer.py — corpus fingerprint write path
• selector.py — internal-corpus read path
• pipeline.py — SimilarityPipeline orchestration
• semantic.py — semantic paraphrase pass
• corpus/base.py — provider protocol
• corpus/internal.py — internal corpus provider
• corpus/core_api.py — CORE provider
• corpus/arxiv.py — arXiv provider (https base)
• corpus/openalex.py — OpenAlex provider
• corpus/crossref.py — Crossref provider

Agent (agents/similarity_agent.py)

SimilarityAgent (name="similarity", wave=3, critical=False) — reads paragraph/section text from Neo4j, builds providers per SIM_ENABLE_* toggles, persists SimilaritySource nodes via ctx.neo4j.run_write.

Phase 4 API

• backend/routers/similarity.py — POST /similarity/{job_id}/recheck (202, 409 in-flight guard, BackgroundTasks), GET /similarity/status
• backend/schemas/similarity.py — SimilarityStatus, RecheckAccepted
• backend/tests/test_similarity.py — 7 tests (report 200/202/404, recheck accepted/409/404, status shape)

Files modified

• backend/routers/reports.py — added GET /reports/{job_id}/similarity (200 report / 202 not-ready / 404 unknown), response_model=SimilarityReport
• backend/main.py — router registration
• orchestrators/custom_orchestrator.py — get_agents() + similarity read-off, upsert_job, save_result (3 spots)
• agents/__init__.py — export
• configs/pipeline_config.yaml — similarity: wave 3, depends_on: [parser, keyword]
• tests/test_agents.py — +1 SimilarityAgent short-text/unavailable test
• New tests: tests/similarity/test_pipeline.py (8), test_self_match_guard.py (9), test_semantic.py (3)

---

Key algorithm facts (stable)

• Winnowing k=5, w=4; coverage bitmap covered[i] = source_index|None.
• Attribution longest-match, ties by trust (internal > CORE > publisher).
• Bucket percents computed independently; status complete|partial|unavailable (< SIM_MIN_DOCUMENT_WORDS=300 → unavailable).
• Headline is verbatim-only; semantic pass never touches overall_percent.
• Preprint self-match guard: title token-overlap ≥ 0.9, DOI match, author overlap ≥ 50% at similarity > 60%.
• Budget timeout → status="partial" + coverage.budget_exhausted=True.
• Provider exceptions caught → last_error + coverage.providers_failed.
• Privacy boundary (plan §7.4): internal matches via build_source_match() (neutral label, no URL/authors, stripped source_excerpt).
• Recheck reuses stored Neo4j text (no PDF re-parse); save_result uses $set merge so refresh is non-breaking.

---

Test status (last full run)

tests/ backend/tests/ tests/similarity/ → 212 passed, 7 failed. All 7 failures are pre-existing and environment-related (not caused by the similarity work) — verified they also fail at Phase 3 commit ecc8207:

• tests/test_agents.py::test_scoring_agent_returns_result
• tests/test_text_pipeline.py::test_no_detectors_enabled_returns_uncertain
• backend/tests/test_health.py::test_root (404 — no frontend-app/dist)
• backend/tests/test_reports.py::test_download_pdf (500)
• backend/tests/test_text_detection.py (3 tests — model returns REAL not AI_GENERATED)

tests/similarity/ alone: 112/112 green. backend/tests/test_similarity.py: 7/7.

Test command:

cd /mnt/c/citation_edge/IIT-Patna && timeout N /tmp/opencode/venv/bin/python -m pytest tests backend/tests tests/similarity -q

---

Environment notes

• venv /tmp/opencode/venv (Py 3.12); missing torch/transformers installed (CPU torch). Also installed python-dotenv, aiofiles, pdfplumber, reportlab, pyyaml, tenacity, structlog, motor, pymongo, uvicorn, pymupdf, matplotlib, scikit-learn, keybert, scipy, fastapi/alice httpx, pydantic 2.13.4.
• Git auth: pull via askpass script /tmp/opencode/git-askpass.sh (Username x-access-token), token supplied by user; never written to repo.
• git user.name/user.email (per-repo): Vansh Patil / vanshpatil@Vansh.localdomain.
• Commits carry a "Git LFS" hook warning (noise only).

---

Remaining (Phase 5+)

• Phase 5: PDF highlight mapper integration — combined AI+similarity span list, kind field on spans, single mapper call (plan §8).
• Merge remote report-ui-updated DONE (506ad88) — resolved 9 overlapping files (backend/main.py, backend/routers/reports.py, orchestrators/custom_orchestrator.py, configs/pipeline_config.yaml, agents/__init__.py, backend/routers/jobs.py, backend/tests/conftest.py, backend/tests/test_jobs.py, tests/test_agents.py).
• Push local commits DONE.