plan_done.md

# Similarity Report Feature — Work Done (Phases 0–4)

This document records everything implemented for the **Corpus Similarity Report**
feature on branch `report-ui-updated`, per `plan.md` + `opencode-prompt.md`.
The feature is complete through Phase 4 (API). Phase 5 (PDF highlight mapper
integration) was **not** started.

> **Superseded 2026-08-02:** the remote merge described under "Remaining" is DONE
> (merge commit `506ad88`). All 9 conflicts resolved; suite is 175 passed / 2 failed
> (both `frontend-app/dist` env failures). See the post-merge box at the top of
> `plan.md` for what the merge changed. Resume at Phase 5.

All 5 commits are local-only (based on `33cb4a2`) and have **not** been pushed.
The remote `report-ui-updated` branch has diverged (20+ commits of Groq-key UI,
citation-relevance/novelty agents, frontend work, LF line endings) and contains
**zero** similarity code.

---

## Commits

| Commit | Description |
|---|---|
| `12cfcb6` | `feat(similarity): Phase 0 scoring core — normalize, exclusions, winnowing, matcher, aggregate` |
| `69c39cd` | `feat(similarity): Phase 1 internal corpus — write path, read path, privacy` |
| `658a727` | `feat(similarity): Phase 2 external providers — CORE, arXiv, OpenAlex, Crossref` |
| `ecc8207` | `feat(similarity): Phase 3 pipeline + agent` |
| `328d8c2` | `feat(similarity): Phase 4 API — report, recheck, status` |

Working tree currently contains 212 files of pure CRLF-vs-LF noise
(`git diff -w` collapses them to 0) plus one dirty LFS binary
`models/kbir_semeval2017/training_args.bin`. Plan agreed with user:
normalize line endings to LF in a housekeeping commit, then merge the remote.

---

## Files added

### Similarity package (`similarity/`)
- `__init__.py` — package export
- `config.py` — settings + environment toggles
- `schema.py` — `SimilarityReport`, `SourceMatch`, `Coverage`, providers
- `normalize.py` — text normalization
- `exclusions.py` — stopword/boilerplate exclusion
- `fingerprint.py` — winnowing fingerprints (k=5, w=4)
- `matcher.py` — matching + attribution
- `aggregate.py` — aggregation into report
- `index_writer.py` — corpus fingerprint write path
- `selector.py` — internal-corpus read path
- `pipeline.py` — `SimilarityPipeline` orchestration
- `semantic.py` — semantic paraphrase pass
- `corpus/base.py` — provider protocol
- `corpus/internal.py` — internal corpus provider
- `corpus/core_api.py` — CORE provider
- `corpus/arxiv.py` — arXiv provider (https base)
- `corpus/openalex.py` — OpenAlex provider
- `corpus/crossref.py` — Crossref provider

### Agent (`agents/similarity_agent.py`)
`SimilarityAgent` (`name="similarity"`, `wave=3`, `critical=False`) — reads
paragraph/section text from Neo4j, builds providers per `SIM_ENABLE_*` toggles,
persists `SimilaritySource` nodes via `ctx.neo4j.run_write`.

### Phase 4 API
- `backend/routers/similarity.py` — `POST /similarity/{job_id}/recheck` (202,
  409 in-flight guard, `BackgroundTasks`), `GET /similarity/status`
- `backend/schemas/similarity.py` — `SimilarityStatus`, `RecheckAccepted`
- `backend/tests/test_similarity.py` — 7 tests (report 200/202/404, recheck
  accepted/409/404, status shape)

## Files modified
- `backend/routers/reports.py` — added `GET /reports/{job_id}/similarity`
  (200 report / 202 not-ready / 404 unknown), `response_model=SimilarityReport`
- `backend/main.py` — router registration
- `orchestrators/custom_orchestrator.py` — `get_agents()` + similarity read-off,
  `upsert_job`, `save_result` (3 spots)
- `agents/__init__.py` — export
- `configs/pipeline_config.yaml` — `similarity: wave 3, depends_on: [parser, keyword]`
- `tests/test_agents.py` — +1 SimilarityAgent short-text/`unavailable` test
- New tests: `tests/similarity/test_pipeline.py` (8), `test_self_match_guard.py`
  (9), `test_semantic.py` (3)

---

## Key algorithm facts (stable)
- Winnowing k=5, w=4; coverage bitmap `covered[i] = source_index|None`.
- Attribution longest-match, ties by trust (internal > CORE > publisher).
- Bucket percents computed independently; status `complete|partial|unavailable`
  (< `SIM_MIN_DOCUMENT_WORDS`=300 → unavailable).
- Headline is verbatim-only; semantic pass never touches `overall_percent`.
- Preprint self-match guard: title token-overlap ≥ 0.9, DOI match, author overlap
  ≥ 50% at similarity > 60%.
- Budget timeout → `status="partial"` + `coverage.budget_exhausted=True`.
- Provider exceptions caught → `last_error` + `coverage.providers_failed`.
- Privacy boundary (plan §7.4): internal matches via `build_source_match()`
  (neutral label, no URL/authors, stripped `source_excerpt`).
- Recheck reuses stored Neo4j text (no PDF re-parse); `save_result` uses `$set`
  merge so refresh is non-breaking.

---

## Test status (last full run)
`tests/ backend/tests/ tests/similarity/` → **212 passed, 7 failed**. All 7
failures are pre-existing and environment-related (not caused by the similarity
work) — verified they also fail at Phase 3 commit `ecc8207`:

- `tests/test_agents.py::test_scoring_agent_returns_result`
- `tests/test_text_pipeline.py::test_no_detectors_enabled_returns_uncertain`
- `backend/tests/test_health.py::test_root` (404 — no `frontend-app/dist`)
- `backend/tests/test_reports.py::test_download_pdf` (500)
- `backend/tests/test_text_detection.py` (3 tests — model returns REAL not
  AI_GENERATED)

`tests/similarity/` alone: 112/112 green. `backend/tests/test_similarity.py`: 7/7.

Test command:
```
cd /mnt/c/citation_edge/IIT-Patna && timeout N /tmp/opencode/venv/bin/python -m pytest tests backend/tests tests/similarity -q
```

---

## Environment notes
- venv `/tmp/opencode/venv` (Py 3.12); missing torch/transformers installed
  (CPU torch). Also installed python-dotenv, aiofiles, pdfplumber, reportlab,
  pyyaml, tenacity, structlog, motor, pymongo, uvicorn, pymupdf, matplotlib,
  scikit-learn, keybert, scipy, fastapi/alice httpx, pydantic 2.13.4.
- Git auth: pull via askpass script `/tmp/opencode/git-askpass.sh`
  (Username `x-access-token`), token supplied by user; never written to repo.
- git user.name/user.email (per-repo): `Vansh Patil` / `vanshpatil@Vansh.localdomain`.
- Commits carry a "Git LFS" hook warning (noise only).

---

## Remaining (Phase 5+)
- Phase 5: PDF highlight mapper integration — combined AI+similarity span list,
  `kind` field on spans, single mapper call (plan §8).
- ~~Merge remote `report-ui-updated`~~ **DONE** (`506ad88`) — resolved 9 overlapping files
  (`backend/main.py`, `backend/routers/reports.py`,
  `orchestrators/custom_orchestrator.py`, `configs/pipeline_config.yaml`,
  `agents/__init__.py`, `backend/routers/jobs.py`, `backend/tests/conftest.py`,
  `backend/tests/test_jobs.py`, `tests/test_agents.py`).
- ~~Push local commits~~ **DONE**.