# Repository — Phase 4 outcome & verification report

**Date:** 2026-06-30
**Branch:** `fix/repo-test-issues-2026-06-29`
**Scope:** Phase 4 of `setup/repository_test_issues_mitigation.md` — Search &
retrieval completeness (SRS REPO-SP3, FRREP-SR001/004/005/007/008, items
4.1–4.5; tester issues #32/#27/#28/#29/#33). Builds on Phase 0
(`repository_phase0_verification_2026_06_29.md`), Phase 1/2/3
(`repository_phase1_/phase2_/phase3_…md`).

All testing in the Docker stack (`docker-compose.yml`), one pytest invocation at
a time. New view tests use `force_login` (never `client.login()`). No login wall
was added to any public read surface (search / advanced / detail / download /
reader stay public; `Document.objects.for_user(anon)` scopes the visible set).

---

## A. Changes shipped

| # | Change | Files |
| --- | --- | --- |
| 4.1 | **Inline document reader (#32, FRREP-SR007).** "Read online" was a raw `<a href="{{ document.file.url }}">` — a direct hit on `/uploads/…` that bypassed permissions and rendered blank for non-PDFs or any media-serving gap. New `read/<uuid>/` → `DocumentReadView` renders an in-app viewer: **PDF.js** (already vendored at `static/js/vendor/pdf.min.js`, lazy-loaded as an ES module exactly like the assignment annotator) for PDFs; a graceful **"Preview isn't available for this format — download instead"** panel for other types; and a **"This document is restricted → Request access"** panel for a signed-in user who can see the record but not its bytes. The viewer fetches the PDF from a new `raw/<uuid>/` → `DocumentRawView` that streams the file *through Django* (same `can_download` gate + watermark path as download, `Content-Disposition: inline`) — so the reader works even where nginx `/uploads/` is mis-mapped (see §D). Records an access event, mirroring the detail view. | `documents/views.py` (`DocumentReadView`, `DocumentRawView`, `_serve_document_file`), `documents/urls.py`, `documents/templates/documents/document_read.html`, `document_detail.html` (repointed "Read online") |
| 4.2 | **Author facet + corpus reindex (#27, FRREP-SR001/004).** `search_documents` gains an explicit `author` filter — case-insensitive substring across first/last name (mirrors `advanced_search`), `.distinct()` so a multi-author match returns one row. Surfaced as a **"Filter by author"** input on the search page that round-trips as `?author=` (HTMX-included into the live results, with a Clear link). New management command **`reindex_repository_search`** loops `update_search_vector` over the corpus (`--collection <slug>`, `--since <iso-date>`, `--missing-only`, `--batch-size`, progress output, reindexed count) — the imported Drupal corpus was never vectorised, so author/facet FTS missed those rows until this is run. | `search/services.py` (`search_documents`), `search/views.py` (`_search_filters`), `documents/templates/search/search.html`, `documents/management/commands/reindex_repository_search.py` |
| 4.3 | **Advanced search repro & fix (#28, FRREP-SR005).** `advanced.html` rendered `doi` and `license` inputs that `advanced_search()` silently ignored, and `int(params["year_from"])` 500'd on non-numeric input. Wired **doi** (`doi__icontains`) and **license** (`license_type=`) into `advanced_search`; guarded both year casts with a `_safe_int` helper (junk → ignored, no 500); made `has_query` robust (only fires on a non-blank value in a real field — `ADVANCED_FIELDS`, ignoring `page`/`sort`/empties); added a **"No results — broaden your search"** empty-state; extended the audit filter dict to the full field set. | `search/services.py` (`advanced_search`), `search/views.py` (`AdvancedSearchView`, `ADVANCED_FIELDS`, `_safe_int`), `documents/templates/search/advanced.html` |
| 4.4 | **Public discoverability (#29, FRREP-SR008).** The corpus was reachable but not discoverable for anonymous visitors (nav hides Repository, landing offered no entry point). Added a **Repository** entry to the public marketing header (desktop + mobile) and a fourth **"Repository — Search Open Access … no account needed"** pillar on the landing hero, both pointing at `repo_search:search`. Confirmed `search.html` / `document_detail.html` render for `AnonymousUser`; restricted docs still surface "Request Access" (FRREP-SR009, unchanged). | `templates/components/public_header.html`, `templates/core/landing.html` |
| 4.5 | **Download streaming + media-pipeline verification (#33).** `DocumentDownloadView` read the whole file into memory (`payload = fh.read()`). Refactored into a shared `_serve_document_file` helper: the open-access / non-watermarked branch now streams from disk with **`FileResponse`** (large-file safe); the restricted-PDF branch still buffers (watermarking needs the full payload) and watermarks exactly as before. Verified the prod nginx media pipeline and found a real demo-environment misconfiguration (see §D). | `documents/views.py` (`DocumentDownloadView`, `_serve_document_file`) |

### Key decisions
- **Read/download both stream through Django, not the raw media URL.** The old "Read online" linked straight to `document.file.url` (`/uploads/…`), which (a) bypassed `can_download` + watermarking and (b) depended entirely on nginx serving `MEDIA_ROOT`. Routing the reader's bytes through `DocumentRawView` fixes the permission leak *and* makes the reader/ download immune to the nginx `/uploads/` mapping — the most likely true cause of "download/read gives nothing" in the test environment (§D).
- **Three reader modes, gated by the existing permission helpers.** `can_access` False → 403 (same as the detail page — no metadata/bytes leak). `can_access` True + `can_download` False → `request_access` panel (e.g. an AUTHENTICATED-visibility doc seen by a signed-in non-owner). `can_download` True → `pdf` (inline viewer) or `fallback` (download panel) by extension. No new permission logic.
- **Author match is substring, not FTS.** The FTS vector already folds author names (weight B), but the imported corpus has stale/NULL vectors, so an FTS-only author facet would miss them until a reindex. A `first/last name icontains` filter works regardless of vector freshness and matches the advanced-search semantics; the reindex command then makes the *full-text* `q=` path findable too.
- **Reindex asserts on findability, not raw vectors.** Tests prove the command's effect via `search_documents(query=…)` returning the doc, not by reading the `SearchVectorField` back (which is brittle) — this is the behaviour users actually depend on.

---

## B. Test results

New Phase 4 tests (all green):

- **`search/tests/test_author_filter.py` (4)** — `search_documents(filters={"author": …})` matches surname / first name, dedupes a co-authored doc, and the search page exposes the `author` field + round-trips `?author=` returning only the match.
- **`documents/tests/test_reindex_command.py` (3)** — a doc with a NULL vector isn't FTS-findable by author until `reindex_repository_search`, then is; `--collection` scopes to one collection; `--missing-only` reindexes only NULL-vector rows into findability.
- **`search/tests/test_advanced_search.py` (8)** — each field (title/abstract/author/author_orcid/doi/license) filters to the seeded doc; year range filters; a non-numeric year doesn't 500; doi/license round-trip through the view; an empty result renders the "No results" broaden-state; a blank-only submit runs no search.
- **`search/tests/test_public_discoverability.py` (4)** — the landing page links to repo search; an anonymous search returns only public docs; anon can open a public detail (200); an anon restricted detail is blocked (302/403, no leak).
- **`documents/tests/test_inline_reader.py` (6)** — public PDF opens the in-app viewer for anon + authed (PDF.js + the raw stream URL present, not the media URL); a non-PDF shows the download fallback; a restricted doc blocks anon bytes on both `read` and `raw`; an AUTHENTICATED-only doc shows the request-access panel to a signed-in non-owner; the reader records an access event.
- **`documents/tests/test_download_stream.py` (3)** — open-access download streams a valid PDF (`resp.streaming is True`, body starts `%PDF`, attachment); a restricted PDF download is watermarked-and-readable for an internal user; the raw endpoint serves `inline`.

Updated existing test: `documents/tests/test_phase0_verification.py::test_download_view_delivers_file_bytes` now reads `streaming_content` (the open-access download intentionally streams via `FileResponse`). This is the only pre-existing test touched, and it is a direct consequence of the 4.5 streaming change.

**Suite:** `docker compose exec -T web pytest apps/repository/ -p no:cacheprovider -q`
→ **236 passed, 13 failed** (was 205 passed after Phase 3; +28 new Phase 4 tests + the FileResponse-streaming update to the one Phase 0 download test, all green). The 13 failures are the unchanged pre-existing baseline (2 `test_ai_insights_live.py` needing a live `OPENAI_API_KEY`; 11 django-axes `client.login()` tests: `test_views.py` ×6 — `test_qa_queue_requires_role`, `test_manager_can_view_qa_queue`, `test_staged_upload_returns_token_and_persists_file`, `test_submit_wizard_consumes_staged_token`, `test_submit_form_rejects_when_no_file_or_token`, `test_staged_upload_delete_clears_token`; `test_followups.py` ×4 — `test_qa008_queue_view_marks_overdue`, `test_mcl007_keyword_list_requires_role`, `test_int009_dashboard_renders_per_target_metrics`, `test_dci002_bulk_submit_creates_one_document_per_file`; `test_integration_endpoints.py` ×1 — `test_rims_ingest_requires_permission`). No new failures.

---

## C. SRS / issue coverage

| Issue | SRS | Verdict before | Status |
| --- | --- | --- | --- |
| #32 Read-online blank | FRREP-SR007 | CONFIRMED | **Fixed** — in-app PDF.js reader + graceful fallbacks, bytes streamed through Django. |
| #27 Author search empties | FRREP-SR001/004 | PARTIAL | **Fixed** — explicit author facet + UI; `reindex_repository_search` revectorises the imported corpus. |
| #28 Advanced search no results | FRREP-SR005 | CONFIRMED | **Fixed** — doi/license wired, year-cast guarded, robust `has_query`, empty-state. |
| #29 Public can't discover search | FRREP-SR008 | UX-GAP | **Fixed** — landing pillar + public header entry; anon search/detail verified. |
| #33 Download delivers nothing | — | NOT REPRODUCED (verify prod) | **Hardened + root-caused** — streaming `FileResponse`; reader/download no longer depend on nginx `/uploads/`; demo nginx mapping bug documented (§D). |

---

## D. Production media-pipeline findings (#33)

**Settings:** `MEDIA_URL = "/uploads/"`, `MEDIA_ROOT = BASE_DIR / "uploads"` (=
`/app/uploads`). WhiteNoise serves `/static/` only — it does **not** serve
`MEDIA_ROOT`, so media must come from nginx or the app.

**nginx (`docker/nginx.conf`) does have a media block** for both server blocks:

```
location /uploads/ { alias /app/uploads/; access_log off; }
```

So the *prod* server (port 80, upstream `web`) serves media correctly — both
mount the **same** named volume `uploads_data:/app/uploads` (web rw, nginx ro).

**Real bug found — the demo environment (port 8001).** `web_demo` writes uploads
to a **separate** volume `demo_uploads:/app/uploads`, but the nginx **8001**
server block serves `/uploads/` from the shared `uploads_data` alias (nginx only
mounts `uploads_data`, not `demo_uploads`). So any file uploaded on the demo site
lands in `demo_uploads` while nginx looks in `uploads_data` → **404 / "download
gives nothing"** on the demo site. Testers exercising the seeded demo environment
would hit exactly the reported symptom.

**Two fixes, both now shipped:**
1. *(Shipped, code)* The reader's "Read online" and the Download button now stream
   bytes **through the Django app** (`DocumentRawView` / `DocumentDownloadView`,
   proxied by `location /` → web), not via the `/uploads/` alias. This makes
   reading + downloading work **regardless** of the nginx media mapping, and
   closes the prior permission/watermark bypass.
2. *(Shipped, infra)* The demo nginx **8001** block now serves `/uploads/` from
   the demo's own files: `nginx` mounts `demo_uploads:/app/demo_uploads:ro`
   (`docker-compose.prod.yml`) and the 8001 block's `location /uploads/` aliases
   `/app/demo_uploads/` (`docker/nginx.conf`). `nginx -t` passes. Takes effect on
   the next `make redeploy` (or a manual `docker compose -f docker-compose.prod.yml
   up -d nginx` to recreate the container with the new volume). Direct
   `/uploads/…` deep links on the demo site now resolve.

---

## E. Follow-ups / not in scope

- Running `reindex_repository_search` over the live corpus is an **operational
  step**. Now wired as make targets: `make reindex-repository-search` (dev) and
  `make deploy-reindex-repository-search` (remote, via `scripts/deploy.sh
  reindex-repository-search`); both accept `ARGS="--missing-only"` etc. Run on
  prod after deploy to make the imported backlog FTS-searchable by author/facets.
- The demo media-serving fix (§D.2) is committed; it activates on the next
  `make redeploy` (or an `nginx` container recreate).