# Repository — Test-Issue Investigation & Phased Mitigation Plan

**Source:** `REPOSITORY TEST ISSUES.docx` (tester report, 33 issues)
**Scope:** `apps/repository/*` (documents, search, analytics, integration) + global nav
**Authored:** 2026-06-29
**Ground truth:** SRS §5.2 REPO-SP1 (Table 23 FRREP-DCI001–017), REPO-SP2 (Table 25 FRREP-MCL001–013), REPO-SP3 (Table 27 FRREP-SR001–011)

---

## 1. How to read this document

Every reported issue was checked against the current code. Each is tagged with a **verdict**:

| Verdict | Meaning |
| --- | --- |
| **CONFIRMED** | The defect is real and reproducible from the code. Needs a build. |
| **PARTIAL** | Capability exists but is incomplete or does not match the SRS flow. Needs extension. |
| **UX-GAP** | The backend/data is correct, but the user-facing surface (placement, widget, discoverability) is wrong. Needs a front-end change. |
| **NOT REPRODUCED** | The code already implements this correctly. The tester likely hit a stale deploy, an un-seeded environment, or a misconfiguration — **verify in the live environment and add a regression test**, do not rebuild. |

The upload surface is a **single 5-step Alpine wizard** (`document_submit.html`): **1 File → 2 Bibliographic metadata → 3 Authors → 4 Collection & licence → 5 Funding & tags**. Several tester complaints about "the metadata page" / "the collection page" / "the tags page" are really about *which step* a field lives on.

---

## 2. Issue register (verdict + root cause)

### Navigation

| # | Issue | Verdict | Root cause (file:line) |
| --- | --- | --- | --- |
| 1 | Repository unreachable from the small-screen / mobile menu | **CONFIRMED** | `templates/components/topnav.html` — the desktop nav has a Repository dropdown (`{% if nav_show_repository %}`, line ~1253) but the **mobile menu panel (lines 2457–2670) has no Repository section at all** (it lists RIMS, REP/Courses, MEL, etc., never `repo_documents`/`repo_search`). The hamburger menu is a *separate* markup block from the desktop dropdown, and Repository was never mirrored into it. |

### REPO-SP1 — Document Capture & Ingestion (the upload wizard)

| # | Issue | Verdict | Root cause (file:line) |
| --- | --- | --- | --- |
| 2 | Upload accepts disallowed formats (e.g. **MP4**) | **CONFIRMED** | `apps/repository/documents/services.py:75-79` — `_validate_repository_upload` passes `allowed_categories=["document","data","image","archive","media"]`. `apps/core/storage/registry.py:72` maps `.mp4 → "media"`, so audio/video pass. Resumable path only checks "extension exists in registry" with no category gate (`services.py:1682`). |
| 3 | No real-time per-file progress during **bulk** uploads | **PARTIAL** | The *primary* file (stage 1) has full XHR progress + speed/ETA (`document_submit.html:83-111`). `additional_files` upload synchronously inside `form_valid` with **no per-file UI** (`views.py:832-872`). |
| 4 | No "replace/update file" control after a mistake | **CONFIRMED** | Wizard offers cancel-then-restart only; no swap-file affordance once a file is staged (`document_submit.html` stage 1; no replace handler). |
| 5 | No cancel option during upload | **NOT REPRODUCED** | `cancelUpload()` exists (`document_submit.html:~582`) and aborts the XHR + DELETEs the staged blob via `StagedUploadView` (`views.py:614-622`). Verify + lock with a test. |
| 6 | No duplicate detection on bulk upload | **PARTIAL** | Detection *exists* — SHA-256 hash + `detect_duplicate` raise `DuplicateDocumentError` per file (`services.py:82-97, 255-261`; bulk loop catches at `views.py:862-864`). But it only **skips + warns after the fact**; it does **not** match the SRS interactive alert (FRREP-DCI011: cancel / new version / separate document). |
| 7 | No automatic metadata extraction | **PARTIAL** | Only *full-text body* extraction for search exists (`tasks.py:19-53`, PDF via pypdf / DOCX via python-docx). No filename/size/creation-date or **embedded author/title** extraction to pre-populate the form (FRREP-DCI009). |
| 8 | Bulk: only one document gets manual metadata | **CONFIRMED** | All files in a bulk submit share one metadata set (`views.py:841-861`; template note "same metadata"). No per-file metadata step. |
| 9 | Mandatory vs non-mandatory fields not indicated | **PARTIAL** | A schema exists (`MetadataSchema.mandatory_fields`, `models.py:139-142`) and is checked *post-submission* in QA pre-checks (`services.py:335-348`), but the form shows **no required markers** and does not validate against it. |
| 10 | Upload proceeds with only one field filled | **CONFIRMED** | `DocumentSubmitForm.clean` only requires *a file* (`forms.py:123-127`). Title/abstract/document_type/collection/licence/author are not enforced at form level (FRREP-DCI010 / MCL010). |
| 11 | Email & API ingestion not implemented | **NOT REPRODUCED** | Both are implemented: email (`integration/mail_ingest.py` `poll_inbox`/`process_message`, models `EmailIngestMailbox`/`EmailIngestEvent`) and API (`integration/rims_api.py` `RimsIngestView`, route `integration/urls.py:10`). Likely **not configured/enabled** in the test env — verify mailbox + API token config. |
| 25 | No unique human Document ID | **NOT REPRODUCED** | `public_document_id` is generated as `REPO-{year}-{hex}` (`services.py:100-108`, assigned at `services.py:281`), unique-checked, and shown in the success message. Verify it renders on the detail page. |

### REPO-SP2 — Metadata, Classification & Author Management

| # | Issue | Verdict | Root cause (file:line) |
| --- | --- | --- | --- |
| 13 | SRS fields missing from the metadata form (subject keywords, grant linkage, authors) | **UX-GAP** | All three exist but live on *other* wizard steps, not step 2: authors → step 3 (`authors_text`), keywords/subject → step 5 (`keywords_text`), grant → step 5 (`grant_reference`). None are on the "metadata" page the tester inspected. (Grant field also renders empty — see #22.) |
| 14–16 | Author capture is error-prone; no registry-first selection; no auto-create with ORCID/affiliation/institution | **PARTIAL** | The Author Registry, ORCID/alias matching, and select-or-create logic all exist server-side (`register_author` `services.py:114-213`, `parse_authors_text` `services.py:1352`). But the UI is a **free-text textarea** (`forms.py:45-49`) — no typeahead against the registry, and the form never captures **affiliation / institution** (model fields exist but are not solicited). |
| 17 | Visibility field should be removed from upload | **CONFIRMED** | `visibility` is in `DocumentSubmitForm.Meta.fields` (`forms.py:79`) and rendered at `document_submit.html:229-230`. Per SRS, visibility is the KM officer's call, not the uploader's. |
| 18 | No document-type dropdown on the collection page | **UX-GAP** | `document_type` *is* a searchable dropdown, but on step 2 (Bibliographic), not step 4 (Collection) (`document_submit.html:158-159`). Placement mismatch. |
| 19 | Target-collection dropdown is empty | **CONFIRMED (data/UX)** | Queryset excludes `INTERNAL` collections for non-staff (`forms.py:98-101`). Empty when no public collection is seeded, or the submitter's collections are all internal, or `show_internal` is unset. Needs collection seeding + an empty-state. |
| 20 | Auto-classification rules not implemented | **CONFIRMED** | `MetadataSchema.auto_classification_rules` is defined (`models.py:149-152`) but read **nowhere** in the codebase. No rule engine (FRREP-MCL006). |
| 21 | Licence should be a dropdown of OA licences | **NOT REPRODUCED** | `license_type` is already a `Select` of Creative Commons / OA choices (`forms.py:87`, `models.py` `License` enum). The tester may have seen the free-text `license_notes`. Verify. |
| 22 | "Grant reference" duplicated on the tags page | **CONFIRMED (different defect)** | There is only **one** grant field, so the "duplicate" claim is stale — **but** `grant_reference` is a `CharField` (`models.py:616`) rendered with a `forms.Select` widget that has **no choices/queryset** (`forms.py:89`), so it renders as an **empty dropdown**. SRS wants a searchable RIMS grant picker (FRREP-MCL009). |
| 23 | Tags must autocomplete from a controlled vocabulary | **PARTIAL** | Free-text comma input only (`forms.py:50-57`). The moderation queue exists (`apply_keywords`/`suggest_or_get_keyword`, new tags → PENDING), and a `Keyword` controlled vocab model exists, but there is **no typeahead** surfacing approved terms (FRREP-MCL007). |
| 24 | No success confirmation after submit | **NOT REPRODUCED** | Single submit → `messages.success("Submission received: REPO-…")` + redirect to detail (`views.py:899-904`); bulk → count message + redirect to browse. Verify the messages partial renders on the destination page. |

### REPO-SP3 — Search & Retrieval

| # | Issue | Verdict | Root cause (file:line) |
| --- | --- | --- | --- |
| 27 | Search bar returns nothing for **author name** | **PARTIAL** | Author names *are* folded into `search_vector` (weight B, `search/indexes.py:26-29,55-57`), so FTS *should* match — **if the document was (re)indexed after authors were attached.** The bulk Drupal-imported corpus likely has stale/empty vectors. Also there is **no explicit author filter/facet** in `search_documents` (`search/services.py:16-87`). |
| 28 | Advanced search returns no results for any input | **CONFIRMED (investigate)** | Form field names match the view contract (`advanced.html` ↔ `AdvancedSearchView` params), so it is **not** a naming mismatch. Likely causes: (a) corpus `for_user`/status filtering returns nothing for the test account (docs in DRAFT/QA or not visible); (b) the form's `doi` & `licence` fields are silently ignored by `advanced_search` (`services.py:113-140`). Needs runtime repro. |
| 29 | Public (unauthenticated) users cannot search the repository | **UX-GAP** | No hard block — `SearchView`/`DocumentDetailView` have **no** `LoginRequiredMixin`, and `Document.objects.for_user(anon)` returns `PUBLIC_OPEN` docs (`managers.py`). The real problem: nav hides Repository from anonymous users (`nav_show_repository=False`, `apps/core/nav_context.py:86`) and the **landing page offers no public search entry point** → not discoverable (FRREP-SR008). |
| 30 | View/download events not recorded | **NOT REPRODUCED** | `record_access_event` (`views.py:521-524`) and `record_download_event` (`views.py:1001`) both fire and write `AccessEvent`/`DownloadEvent`. Verify rows land in the test DB. |
| 31 | Search queries / filters / saved searches not in the audit trail | **CONFIRMED** | `track_search_event` + `record_search_event` + `SearchEvent` model all exist but `track_search_event` is **never called** from `SearchView` (FRREP-SR + DCI017 analytics). Disconnected. |
| 32 | "Read online" opens a blank page | **CONFIRMED** | "Read online" is a raw `<a href="{{ document.file.url }}">` (`document_detail.html:90-93`). There is **no inline viewer route**; non-PDFs and any prod media-serving gap render blank. |
| 33 | Download does not deliver a readable file | **NOT REPRODUCED (verify prod)** | `DocumentDownloadView` serves the bytes with guessed content-type + `Content-Disposition: attachment` + watermark for restricted PDFs (`views.py:993-1018`). Code is correct; the field report points at a **deployment/media-serving** issue (e.g. WhiteNoise does not serve `MEDIA_ROOT`; nginx/EC2 media path). Verify the live media pipeline; also note it reads the whole file into memory (stream large files). |

**Summary:** of 33 reported items — **13 CONFIRMED**, **8 PARTIAL**, **4 UX-GAP**, **8 NOT REPRODUCED** (already implemented; verify + regression-test rather than rebuild).

---

## 3. Phased mitigation plan

Phases are ordered by **risk-to-users × effort**. Each item lists the SRS requirement it satisfies. Run the Docker stack (`docker-compose.yml`) for all testing; one pytest invocation at a time (test DB contention).

### Phase 0 — Verify-and-confirm + one-line safety fixes (≈1–2 days)

*Goal: close the data-integrity hole, restore access, and convert the 8 "NOT REPRODUCED" claims into either a confirmed prod bug or a green regression test — before building anything new.*

0.1 **Restrict upload formats** (#2, FRREP-DCI005). In `services.py:75-79` change `allowed_categories` to `["document","data","image"]` (drop `media`, and `archive` unless ZIP bundles are intended). Add the **same category check to the resumable start path** (`start_resumable_upload`, `services.py:1661-1685`) — currently it only checks registry membership. Reject with the allowed-format list in the message.

0.2 **Repository in the mobile menu** (#1, discoverability). Add a Repository `<div x-data="{open:…}">` block to the topnav mobile panel (`topnav.html` 2457–2670), gated by `{% if nav_show_repository %}`, mirroring the desktop links (Browse / Search / Advanced search / Collections / Submit / Saved searches, plus manager links behind the same role checks as the desktop block).

0.3 **Remove the visibility field from upload** (#17, SP2 KM-officer rule). Drop `visibility` from `DocumentSubmitForm.Meta.fields` (`forms.py:79`) and from `document_submit.html:229-231`; default new docs to `Visibility.INHERIT` in `ingest_document` (already the default). Visibility stays editable on the KM/edit + QA surfaces.

0.4 **Wire search audit logging** (#31, FRREP-SR/DCI017). Call `search_services.track_search_event(...)` from `SearchView` (after the queryset count is known) and from `AdvancedSearchView`. Persist query + filters + result_count + user.

0.5 **Fix the empty grant dropdown now** (#22, interim). Until the RIMS picker (Phase 2) lands, render `grant_reference` as a **text input** (or a Select2 with an actual queryset) so it is usable instead of an empty `<select>`.

0.6 **Verification sweep** (turn field reports into tests): confirm in the running stack and add a regression test for each of #5 (cancel), #11 (email + API ingestion config), #21 (licence dropdown), #24 (submit confirmation message), #25 (public Document ID on detail), #30 (access/download events written). For #33 (download) and #32 prerequisite, **verify the live media-serving pipeline** (EC2/nginx serves `MEDIA_ROOT`; WhiteNoise does not) — this is likely the true cause of "download/read gives nothing" in production.

### Phase 1 — Upload integrity & submission correctness (≈3–5 days)

*Goal: a submission can no longer be created with junk/empty metadata, and the collection picker works.*

1.1 **Mandatory-field enforcement** (#9, #10, #11; FRREP-DCI010 / MCL010). Enforce required at the form level: `title`, `abstract`, `document_type`, `collection`, `license_type`, and ≥1 author. Drive "required" from `collection.metadata_schema.mandatory_fields` where present, else the SRS default set. Surface required markers (`*`) and inline errors per step in the wizard; block "Submit for review" until satisfied (mirror the existing QA pre-check logic into form validation so it fails *before* ingest, not after).

1.2 **Collection dropdown population + empty-state** (#19). Seed at least one public collection in the environment seeders; confirm the submitter's eligible collections appear (audit the `show_internal` initial + the `exclude(INTERNAL)` rule for the uploader's role). If the queryset is empty, render an explicit empty-state ("No collection is available to you yet — contact a Repository Administrator") instead of a blank `<select>`.

1.3 **SRS-compliant duplicate handling** (#6, FRREP-DCI011). On detecting a hash match, return an interactive choice — **cancel / upload as new version / continue as separate document** — instead of silently skipping. The hash + `detect_duplicate` already exist; this is wizard UX + an `allow_duplicate` / `as_version` branch into `ingest_document` (which already accepts `allow_duplicate`).

1.4 **Document-type placement** (#18). Decide: keep `document_type` on step 2 (it is a sensible bibliographic field) and add a clarifying caption, OR move it onto step 4 next to Collection if testers expect it there. Low effort; align with the KM team's mental model.

### Phase 2 — Author registry, classification & RIMS linkage (the SP2 build) (≈1.5–2 weeks)

*Goal: bring author capture, tags, grant linkage, and classification up to the SRS flow.*

2.1 **Registry-first author entry** (#14–16, FRREP-MCL001/002/003). Replace the free-text author textarea with a typeahead that queries the Author Registry (name/ORCID/email/institution) and shows ORCID + affiliation + institution for selection; "not found" opens an inline add form (name, email, affiliation, ORCID) that calls the existing `register_author` and links the new entry. Capture `affiliation_text` / `institution` (model fields already exist, just unsolicited). Keep ORCID-verification hook for FRREP-MCL002.

2.2 **Auto metadata extraction** (#7, FRREP-DCI009). On stage-1 finalize, extract filename, size, creation date, and **embedded PDF/DOCX author + title** (pypdf `document_info`, python-docx core-properties) and pre-populate the metadata form. Mark prefilled fields as editable suggestions.

2.3 **Controlled-vocabulary tag autocomplete** (#23, FRREP-MCL007). Add typeahead over approved `Keyword` rows; free-text entries continue into the existing PENDING moderation queue. Surface "this tag will be reviewed before it appears" inline.

2.4 **RIMS grant linkage picker** (#22, FRREP-MCL009). Replace the interim grant text input with a searchable dropdown of active/completed RIMS grant/project records (reuse the RIMS integration layer), storing the `public_grant_id` and rendering the linkage on both the document and the RIMS record.

2.5 **Auto-classification rule engine** (#20, FRREP-MCL006). Implement evaluation of `MetadataSchema.auto_classification_rules` (`if_keyword`/`then_collection_slug` etc.) at ingest, with a user-visible suggested classification and an override (log overrides per SRS A3).

2.6 **Metadata-step completeness** (#13). Ensure every SRS metadata field is reachable and labelled; if the wizard keeps fields across steps, add a step-5 review summary so the uploader sees the full record before submit.

### Phase 3 — Bulk upload experience (≈1–1.5 weeks)

*Goal: bulk upload matches FRREP-DCI002/003.*

3.1 **Per-file real-time progress** (#3, FRREP-DCI003) — route each additional file through the existing staged/resumable mechanism with its own progress bar + speed/ETA, instead of a single synchronous form POST.

3.2 **Batch summary report** (#2, FRREP-DCI002) — after a bulk run, show a per-file table of success / failure (with reason) and a **retry** action for failed items.

3.3 **Per-file metadata in bulk** (#8) — allow editing metadata per file (default = shared template, override per item) before final submit.

3.4 **Replace/swap file** (#4) — add a "replace file" control on the staged file so a mistaken upload can be swapped without restarting the wizard.

### Phase 4 — Search & retrieval completeness (≈1–1.5 weeks)

*Goal: full SP3 parity.*

4.1 **Inline document reader** (#32, FRREP-SR007) — add a `read/<uid>/` view that renders an embedded viewer (PDF.js for PDFs; graceful "preview unavailable, download instead" for other types), respecting visibility/watermark rules. Repoint the "Read online" button at it.

4.2 **Author search + reindex** (#27, FRREP-SR001/004) — add an explicit `author` filter to `search_documents` and the filter panel; run a one-off `update_search_vector` reindex management command over the imported corpus so author names are actually in the vectors.

4.3 **Advanced search repro & fix** (#28, FRREP-SR005) — reproduce with a known-visible doc; wire the form's `doi`/`licence` fields into `advanced_search` (or remove them), guard the `int(year_*)` casts, and add a "no results — broaden your search" empty-state. Cover with tests for each field.

4.4 **Public discoverability** (#29, FRREP-SR008) — surface a public repository search entry point on the landing page and in the anonymous nav; confirm `search.html` / `document_detail.html` render for `AnonymousUser` (no template assumptions that require `request.user`). Restricted docs show "Request Access" per FRREP-SR009 (already present).

4.5 **Verify download/media in production** (#33) — confirm `MEDIA_ROOT` is served on the EC2 host (nginx location block), switch the download view to `FileResponse`/streaming for large files, and add a smoke test that a downloaded PDF opens.

---

## 4. Out-of-scope / already-shipped (do not rebuild)

These SRS capabilities are already implemented and were *miscategorised* as missing in the report — Phase 0 only **verifies** them: file cancel (#5), email + API ingestion (#11), unique Document ID (#25), licence dropdown (#21), submit confirmation (#24), view/download analytics (#30), and the working download view (#33 — investigate as a prod media-serving issue, not a code rebuild).

---

## 5. Suggested sequencing

Phase 0 first (small, ships immediately, and de-risks the rest by confirming what is actually broken). Phases 1 and 4 are independent and can run in parallel by two contributors. Phase 2 depends on Phase 1's form refactor. Phase 3 depends on Phase 1's staged-upload/validation work.