# Repository — Phase 2 outcome & verification report

**Date:** 2026-06-29
**Branch:** `fix/repo-test-issues-2026-06-29`
**Scope:** Phase 2 of `setup/repository_test_issues_mitigation.md` — author registry,
classification & RIMS linkage (the SRS REPO-SP2 build, items 2.1–2.6). Builds on
Phase 0 (`repository_phase0_verification_2026_06_29.md`) and Phase 1
(`repository_phase1_2026_06_29.md`).

All testing in the Docker stack (`docker-compose.yml`), one pytest invocation at a
time. New tests use `force_login`.

---

## A. Changes shipped

| # | Change | Files |
| --- | --- | --- |
| 2.1 | **Registry-first author entry (FRREP-MCL001/002/003).** The Authors step is now a typeahead over the Author Registry (search by name / ORCID / email / institution) plus an inline add-new form (name, email, affiliation, ORCID). The structured selection posts as a new hidden `authors_json` field; the view resolves it via `services.resolve_authors_payload`, which attaches existing authors by id (no re-dedup) and runs new authors through `register_author` (ORCID/email/name dedupe + the ORCID verification hook + `affiliation_text` capture). The legacy `authors_text` block survives as the no-JS/edit-form fallback. The ≥1-author rule moved into `clean()` and spans both channels. New JSON endpoint `AuthorSearchView` (`submit/authors/search/`). | `documents/services.py` (`resolve_authors_payload`, `count_payload_authors`, `_iter_author_payload`, `_split_author_name` refactor), `documents/views.py` (`AuthorSearchView`, `form_valid`), `documents/forms.py`, `documents/urls.py`, `documents/templates/documents/document_submit.html` |
| 2.2 | **Auto metadata extraction (FRREP-DCI009).** New `services.extract_file_metadata(path)` reads embedded title/author/creation-year from PDF (pypdf `PdfReader.metadata`) and DOCX (python-docx `core_properties`), plus filename + size. `StagedUploadView` runs it on the staged file and returns `metadata` in its JSON response; the wizard pre-fills title/year (when empty) as flagged "suggested from your file" editable values and offers a one-click "add suggested author". Never raises — a metadata-less or corrupt file yields just the filesystem facts. | `documents/services.py`, `documents/views.py` (`StagedUploadView`), `document_submit.html` |
| 2.3 | **Controlled-vocabulary tag autocomplete (FRREP-MCL007).** The Tags field is now a combobox over APPROVED `Keyword` rows (new `KeywordSearchView` at `submit/keywords/search/`); free-typed tags are accepted, shown with a "pending review" chip, and continue into the existing PENDING moderation queue via `apply_keywords`. The combobox writes a comma-joined label list back into the (now hidden) `keywords_text` field — the server path is unchanged. | `documents/views.py` (`KeywordSearchView`), `documents/forms.py`, `documents/urls.py`, `document_submit.html` |
| 2.4 | **RIMS grant linkage picker (FRREP-MCL009).** New `apps/repository/integration/grant_link.py` with `linkable_grants_qs` / `lookup_grant` / `is_linkable_grant` (only `APPROVED` + `CLOSED` `FundingRecord`s) and `GrantSearchView` (`api/grants/search/`, searches `public_grant_id` + `title`). The submit form's `grant_reference` is now a hidden field the combobox writes the chosen `public_grant_id` into; `clean_grant_reference` rejects an id that doesn't resolve to a linkable grant. The document detail page resolves the linkage and shows the grant title — deep-linked to `rims_grants:funding_detail` for users who can read RIMS funding, plain text otherwise. `apply_grant_closure_flag` still keys off the stored string (unchanged). | `integration/grant_link.py`, `integration/urls.py`, `documents/forms.py`, `documents/views.py` (`DocumentDetailView`), `documents/templates/documents/document_detail.html`, `document_submit.html` |
| 2.5 | **Auto-classification rule engine (FRREP-MCL006 / SRS A3).** New `services.evaluate_auto_classification(keywords, title, abstract)` treats every `MetadataSchema.auto_classification_rules` entry (`{if_keyword, then_collection_slug}`) as one global ruleset — a rule fires on an exact keyword match or a title/abstract substring, first firing rule with an existing target wins. New `ClassificationSuggestView` (`submit/classify/suggest/`) surfaces a suggestion on the Review step only when it differs from the current pick. Accepting it switches the collection select; keeping a different one records `suggested_collection_slug`, and the view writes an override audit row via `services.log_classification_override`. Advisory only — never silently moves the document. | `documents/services.py` (`evaluate_auto_classification`, `log_classification_override`), `documents/views.py` (`ClassificationSuggestView`, `form_valid`), `documents/forms.py`, `documents/urls.py`, `document_submit.html` |
| 2.6 | **Metadata-step completeness + review summary (#13).** A sixth wizard step, **Review**, renders a live summary of the full record (title, abstract, type, authors, collection, licence, grant, tags) plus the classification suggestion before submit. The submit button now lives on the Review step. | `document_submit.html` |

### Key decisions
- **`authors_json` is the new author channel; `authors_text` is the fallback.** Existing authors serialize with full display fields (id + name + orcid + institution + affiliation) so chips rebuild on an error re-render; the server only reads `id` for existing entries and `name/orcid/email/affiliation` for new ones. Validation (`clean`) never writes Author rows — it only *counts* resolvable entries (`count_payload_authors`).
- **Tags & grant reuse the existing server contracts.** The tag combobox writes `keywords_text` (comma list → `apply_keywords`), and the grant picker writes `grant_reference` (the `public_grant_id` string). No ingest-path or schema changes — only the widgets and validation changed.
- **Classification is advisory and audited.** The engine never moves a document; it suggests. An override (user kept a collection different from the one suggested) is written to the audit trail per SRS A3.
- **Custom Alpine comboboxes, not Select2-ajax.** The three typeaheads need richer rows (ORCID/affiliation, "pending review" chips, an inline add-new form) and free-text add, so they're hand-rolled Alpine comboboxes hitting GET JSON endpoints — no CSRF needed, and the existing Select2 include still serves `document_type` / `license_type` / `collection`.

---

## B. Test results

New Phase 2 tests (all green):
- `documents/tests/test_author_registry_picker.py` — author-search endpoint (login gate, name/ORCID match, returned metadata); `resolve_authors_payload` (attach-by-id without re-create, new-author + affiliation, dedupe, junk tolerance); form ≥1-author rule; view submit attaches without duplicating / creates one new author.
- `documents/tests/test_metadata_extraction.py` — PDF + DOCX embedded title/author extraction, metadata-less + corrupt files are safe, `StagedUploadView` returns the metadata.
- `documents/tests/test_tag_autocomplete.py` — keyword-search endpoint returns approved-only; approved tag attaches immediately; new tag queues PENDING and does not attach.
- `documents/tests/test_grant_picker.py` — form accepts linkable / rejects unknown + draft / allows blank; detail page deep-links for privileged users, shows title-only otherwise.
- `integration/tests/test_grant_link.py` — grant-search endpoint login gate, linkable-statuses-only, title match.
- `documents/tests/test_auto_classification.py` — rule match (keyword + title substring), no-match, dangling-collection guard, suggest endpoint, override audited / no-audit-on-accept (on-commit captured).
- `documents/tests/test_review_summary.py` — Review step + all labelled summary rows render; full record submits.

Modified Phase 0/1 tests (to track the Phase 2 refactor, ≥1-author guarantee preserved):
- `test_submit_form_fields.py` — grant widget is now a picker-driven `HiddenInput` (was the interim `TextInput`); the visibility test no longer posts a bogus grant id.
- `test_mandatory_fields.py` — the no-author case now reports on `authors_json`; added a registry-payload happy-path.

**Suite:** `docker compose exec -T web pytest apps/repository/` → **195 passed / 13 failed**
(was 157 passed / 13 failed after Phase 1 — +38 new passes, baseline failures
unchanged). The 13 are the pre-existing baseline, identical test names before and
after: 2 `test_ai_insights_live.py` (need a live `OPENAI_API_KEY`) + 11 django-axes
`client.login()` tests (`feedback_axes_client_login.md`) — `test_views.py` ×6
(qa-queue / staged-upload), `test_followups.py` ×4, `test_integration_endpoints.py`
×1. No new test fails and nothing regressed.

> Operational note: run the repository suite as a **single** pytest invocation — a
> second concurrent run collides on `test_iilmp` and produces ~200 setup-time
> ERRORs (not code failures). If that happens, terminate stale connections and
> `DROP DATABASE test_iilmp` via `psql -U iilmp` in the postgres container, then
> re-run (see `feedback_test_db_concurrent_external.md`).

---

## C. Not done in Phase 2 (by design)

Deferred per the handoff: inline reader (4.1), full per-file bulk progress/metadata
(3.x), author search facet / corpus reindex (4.2), advanced-search field wiring
(4.3). The new-author form captures `affiliation_text` (free text); linking a new
author to a structured `Institution` FK remains a registry/merge-tool action, not a
submit-time picker. No login wall was added to public read surfaces.