# Repository Migration Plan — Drupal 7 (AgriDrupal/AGRIS AP) → `apps/repository`

**Status:** Draft for sign-off · **Date:** 2026-06-08 · **Source dump:** `rufoadm_drpl1.sql` (1.5 GB, MySQL 5.7)
**Live source:** `repository.ruforum.org` (cPanel account `rufoadm`, `/home/rufoadm/public_html`)

> Plan-first per agreement: this document is for review **before** the importer is built.
> Decisions already taken: **pull PDFs from the server**, **extend the Django model** for
> first-class bibliographic metadata, **build a management-command importer** after sign-off.

---

## 1. What the source is

The dump is a **Drupal 7 site running FAO's AgriDrupal / AGRIS Application Profile** distribution
— the platform behind `repository.ruforum.org`. Repository content is the `agdlios`
("Ag Document-Like-Information-Object") content type, described by ~80 `field_ag_*` metadata fields.

### Content inventory (verified against the dump)

| Drupal entity (`node.type`) | Rows | Migration target |
|---|---:|---|
| `agdlios` (documents) | **2,897** (2,890 published, 7 unpublished) | `Document` |
| `agauthor` (author nodes) | **5,052** | `Author` |
| `agcorporate_body` | 125 | `Author` (corporate) / affiliation text |
| `agperiodical` (journals) | 118 | new `Periodical` lookup (see §4) |
| `agconference` | 23 | bibliographic fields on `Document` |
| `agprofile` / `aginstitution` / others | — | out of scope (expert directory, not repository docs) |

### Files

| Metric | Value |
|---|---:|
| Files attached to documents (`file_usage` → node) | **3,219** |
| — of which PDF | 3,163 |
| — DOC/DOCX/PPT/XLSX | ~56 |
| Total managed-file payload | **~2.57 GB** |
| Storage scheme: `public://` → `sites/default/files/` | 3,979 |
| Storage scheme: `private://` (restricted docs) | 51 |

**Authoritative document↔file join:** `file_usage` (`module='file'`, `type='node'`, `id=nid` → `fid`),
then `file_managed.fid → uri/filename/filesize/filemime`. The Field-API table
`field_data_field_additional_uploads` holds only **11** rows and must **not** be used as the link.
`public://X` resolves to `sites/default/files/X`; `private://X` to the account's private files dir.

---

## 2. Source data model (how the pieces connect)

```
node (nid, type='agdlios', title, status, created, changed, uid)
 ├─ field_data_field_ag_abstract      (entity_id=nid) → abstract HTML
 ├─ field_data_field_ag_author        (entity_id=nid, delta=order) → target nid of agauthor node
 ├─ field_data_field_ag_datepub/year  → publication date / year
 ├─ field_data_field_ag_language      → language
 ├─ field_data_field_ag_access        → 0=open(2437) | 1(38) | 2(11)  → visibility
 ├─ field_data_field_ag_agrovoc       → AGROVOC subject terms (13,692 links)
 ├─ field_additional_keywords         → free-text keywords
 ├─ field_data_field_ag_publisher / publishplace / pub_title
 ├─ field_data_field_ag_journal / journal_issue / volume / number / issn / eissn / pagination
 ├─ field_data_field_ag_isbn / edition / book / booknumber
 ├─ field_data_field_ag_confname / confdate / conflocation
 ├─ field_data_field_ag_degree / degree_institution   (theses)
 ├─ field_data_field_ag_type          → document-type hint
 ├─ field_data_field_ag_resource_url  (140) → external resource URL
 ├─ field_data_field_ag_oai_identifier / agris / arn   → external IDs
 └─ file_usage(fid,node=nid) → file_managed(uri,filesize,filemime)

node (type='agauthor')
 ├─ field_data_field_ag_lastname  → last_name
 ├─ field_data_field_ag_name      → first_name
 └─ field_data_field_ag_email     → email
```

Authors are **separate nodes** referenced by documents; `delta` gives author order →
maps directly to `DocumentAuthor.order`. Corporate authors (`agcorporate_body`) referenced via
`field_ag_corauthor`.

---

## 3. Field mapping → `Document`

| Django `Document` field | Drupal source | Coverage | Transform |
|---|---|---:|---|
| `title` | `node.title` | 100% | trim |
| `abstract` | `field_ag_abstract` | 3,977 | strip HTML → text (keep paragraphs) |
| `file` | `file_usage`→`file_managed.uri` (primary = lowest fid / largest PDF) | 3,219 | re-home into media (§6) |
| `file_hash` | computed on import | — | SHA-256 of bytes |
| `file_size_bytes` | `file_managed.filesize` | ✓ | int |
| `document_type` | `field_ag_type` + heuristics | partial | classification rules (§5) |
| `year` | `field_ag_year` / parsed `field_ag_datepub` | 2,842 | int, validate 1900–current |
| `language` | `field_ag_language` | 2,680 | ISO-639-1 normalise, default `en` |
| `license_type` | none in source | 0% | default `CC_BY`, flag for review |
| `visibility` | `field_ag_access` (0→`public_open`, 1/2→`restricted`) + `private://` | 2,467 | + `node.status=0` → `internal` |
| `authors` (M2M via `DocumentAuthor`) | `field_ag_author` delta-ordered node refs | 9,161 links | resolve author nid → `Author` |
| `keywords` (M2M) | `field_ag_agrovoc` + `field_additional_keywords` | 13,692 | upsert `Keyword` (status=`approved`), AGROVOC URI |
| `doi` | (none populated) | 0% | — |
| `source_system` | constant | — | `"drupal_import"` |
| `submitted_at` / `published_at` | `node.created` / `node.changed` | ✓ | UNIX ts → tz-aware |
| `status` (FSM) | `node.status` | ✓ | published→`PUBLISHED`, else `DRAFT` |

### Author mapping → `Author`

| `Author` field | Source | Notes |
|---|---|---|
| `last_name` | `field_ag_lastname` | |
| `first_name` | `field_ag_name` | |
| `email` | `field_ag_email` | |
| `affiliation_text` | `field_ag_institution` (corp) | |
| `orcid` | — | none in source → null |

Dedup authors on `(last_name, first_name, email)`; keep a `drupal_nid → Author.id` map for ref resolution.

---

## 4. Proposed model extensions (decision: extend the model)

New optional fields on `Document` (all nullable/blank — backwards compatible, one migration):

```python
# Bibliographic block (AGRIS AP)
publisher        = CharField(max_length=255, blank=True)
publication_place= CharField(max_length=160, blank=True)
journal_title    = CharField(max_length=255, blank=True)   # field_ag_journal / pub_title
volume           = CharField(max_length=32,  blank=True)
issue            = CharField(max_length=32,  blank=True)    # field_ag_journal_issue / number
pages            = CharField(max_length=32,  blank=True)    # field_ag_pagination
issn             = CharField(max_length=18,  blank=True, db_index=True)
eissn            = CharField(max_length=18,  blank=True)
isbn             = CharField(max_length=20,  blank=True, db_index=True)
edition          = CharField(max_length=64,  blank=True)
conference_name  = CharField(max_length=255, blank=True)
conference_date  = CharField(max_length=64,  blank=True)
conference_place = CharField(max_length=160, blank=True)
degree           = CharField(max_length=120, blank=True)    # theses
degree_institution = CharField(max_length=255, blank=True)
external_resource_url = URLField(max_length=500, blank=True) # field_ag_resource_url
# External identifiers / provenance
agris_identifier = CharField(max_length=64, blank=True, db_index=True)
oai_identifier   = CharField(max_length=255, blank=True)
legacy_drupal_nid = PositiveIntegerField(null=True, blank=True, unique=True, db_index=True)  # idempotency anchor
```

On `Keyword`: add `agrovoc_uri = URLField(blank=True)` (carry `field_agrovoc_uri` so terms link to AGROVOC).

> `legacy_drupal_nid` is the **idempotency key** — re-running the importer updates rather than duplicates.

Alternative considered: stash everything in `ai_metadata` JSON. Rejected per decision — bibliographic
fields must be queryable/searchable (e.g. browse by journal, filter by ISSN, search by conference).

---

## 4b. Browse-facet & search parity (legacy "Browse by …")

The legacy site browses by **Authors · Subjects · Institutions · Regions · Countries ·
Conferences**. Audit of our model vs. those facets, and what was added:

| Legacy facet | Drupal vocab (vid) | Doc coverage | Status |
|---|---|---:|---|
| Authors | `agauthor` nodes | — | already had `Author` |
| Subjects — broad | vid 10 *Agris Subject Categories* | — | **added** `Keyword.kind=agris_category` |
| Subjects — specific | vid 16 *Agrovoc terms* | 13,692 | **added** `Keyword.kind=agrovoc` + `agrovoc_uri` |
| Regions | vid 22 *Region Focus* (multi) | 2,940 | **added** `Region` model + `Document.regions` M2M (12 seeded) |
| Countries | vid 19 *Country* (single) | 2,495 | **added** `Document.country` (canonical full name) |
| Institutions | vid 25/21 (Publisher / University) | — | **added** `Document.publisher`; author `Institution` FK already existed |
| Conferences | vid 11 + `agconference` | — | **added** `conference_name/date/place` |

**Search improvements shipped (this change):**
- `search_documents` gained filters: `region`, `country`, `subject` (keyword slug),
  `publisher`, `conference`.
- `BrowseView` gained `region_facets`, `country_facets`, `subject_facets` (+ counts &
  multi-select selected-state) alongside the existing type/collection facets.
- `update_search_vector` now folds **keywords, regions, country, journal, publisher,
  conference** into the tsvector (weight C) so a free-text query matches on subject/geo too.
- Migrations `0010` (schema) + `0011` (seed the 12 controlled regions).

**Browse UI shipped (this change):**
- `documents/partials/browse_nav.html` — the legacy "Browse by" rail (All · Collections ·
  Subjects · Regions · Countries · Advanced search) with active-pill state; included on
  the browse page and every browse-index page.
- `documents/browse_index.html` — generic facet-index page (grouped, counted rows linking
  into the pre-filtered browse surface), driven by `BrowseByRegionView` /
  `BrowseByCountryView` / `BrowseBySubjectView` at `/repository/browse/{regions,countries,subjects}/`.
- `documents/partials/_browse_facets_geo.html` — Region / Country / Subject filter
  fieldsets added to both the desktop filter rail and the mobile drawer on `browse.html`,
  plus active-filter chips. All inline utility classes (ruforum-* tokens) — no CSS rebuild.
- Tests: browse-index render + facet-surface tests in `search/tests/test_facets.py`.

## 5. `document_type` classification rules (applied in order)

1. `field_ag_degree` present → `THESIS`
2. `field_ag_confname` present → `CONFERENCE_PAPER`
3. `field_ag_journal`/`field_ag_issn` present → `JOURNAL_ARTICLE`
4. `field_ag_isbn`/`field_ag_book` present → `PUBLICATION`
5. `field_ag_type` keyword match (`report`→`TECHNICAL_REPORT`, `policy`→`POLICY_BRIEF`, `dataset`→`DATASET`)
6. fallback → `OTHER`

A per-document classification report is emitted (`--report`) so a curator can spot-correct before publish.

---

## 6. File re-homing

1. Pull `sites/default/files/` (public) + private files dir from the server (§8) into a local staging dir,
   preserving filenames.
2. For each `agdlios` node, resolve its `fid`s via `file_usage`; pick the **primary** file
   (single PDF, or largest PDF if multiple), attach remaining as `DocumentVersion`/supplementary.
3. Copy bytes into Django storage via the model's `upload_to_repository` path; compute `file_hash`
   (SHA-256) and `file_size_bytes`; set `preview_supported` for PDFs.
4. URL-decode/normalise filenames (source has spaces, commas, unicode — e.g. `Green Book, Part 1.pdf`).
5. Missing-file handling: if a referenced `uri` has no byte on disk, create the `Document` with
   `file` empty + `ai_metadata.import_warning="missing_file"`; log to the run report (don't abort).

---

## 7. Importer design (`manage.py import_drupal_repository`)

- **Input:** parsed dump. Convert the relevant tables to a working store first — recommended:
  load the dump into a throwaway **MySQL/MariaDB container** (`docker-compose`), then read via a
  read-only connection. (Parsing 1.5 GB of SQL text directly is brittle; a real DB makes the joins trivial.)
  Only ~25 tables are needed: `node`, `node_revision`, `file_managed`, `file_usage`,
  `field_data_field_ag_*`, `field_data_field_additional_keywords`, `field_data_field_agrovoc_uri`.
- **Stages (idempotent, resumable):**
  1. `authors` — upsert `Author` from `agauthor` nodes; build `nid→Author` map.
  2. `keywords` — upsert `Keyword` from AGROVOC + free-text; build label/uri map.
  3. `documents` — upsert `Document` keyed on `legacy_drupal_nid`; map metadata; set FSM status.
  4. `authorship` — wire `DocumentAuthor` (delta order, role=primary for delta 0).
  5. `files` — re-home bytes, attach, hash (§6).
  6. `search` — backfill `search_vector` (the model already has the GIN index).
- **Flags:** `--limit N` (sample), `--stage <name>`, `--dry-run`, `--report path`, `--media-staging path`.
- **Safety:** wrap each document in a transaction; collect a CSV run-report
  (nid, title, type, files, warnings); never delete existing IILMP data.
- **Tests:** run under `docker-compose`; fixtures = ~20 representative nodes (thesis, journal article,
  conference paper, multi-author, multi-file, restricted, missing-file).

---

## 8. File transfer — RESOLVED: public HTTP download (no SSH needed)

**No server access is required.** Every document file is served publicly through Drupal's
force-download handler. The URL is fully reconstructable from the dump:

```
https://repository.ruforum.org/system/tdf/<urlencode(file_managed.filename)>?file=1&type=node&id=<nid>&force=
```

- `filename` = `file_managed.filename` (URL-encode; handles spaces, commas, unicode)
- `nid` = `file_usage.id` for that `fid` (`module='file'`, `type='node'`)
- `file=1` and `force=` are constant flags

**Verified (2026-06-08):** constructed-from-dump URLs return `200 OK`, `Content-Type: application/pdf`,
and **byte-exact** payloads (e.g. fid 25 → 211,192 B, fid 29 → 9,820,953 B — both match
`file_managed.filesize`). Public, no auth, no cookies.

**Manifest:** join `file_usage`(node) ⋈ `file_managed`(PDF) → **3,179 document-attached PDFs**
(plus the `private://` set — see risks). The importer downloads each over HTTP, keyed by `fid`
(to avoid filename collisions), verifies size against `file_managed.filesize`, then re-homes (§6).

> Root SSH on the box is key-only and WHM API works with the root password, but **neither is needed**
> now that public download is confirmed. The earlier SSH/key approach is abandoned;
> `/tmp/ruforum_migrate.pub` can be discarded (never installed).

**Caveats:**
- The dump is a point-in-time snapshot; the live site has newer nodes (e.g. nid 41101 in the wild,
  above the dump's range). Files deleted/renamed since the dump will 404 → handle as missing-file (§6).
- `private://` files (51) are *not* served by `system/tdf` to anonymous users — if they must migrate,
  they still need authenticated/WHM access. Confirm whether legacy restricted docs are in scope.

---

## 9. Risks & open questions

- **Licensing** — source has no license field; everything defaults to `CC_BY` pending curator review.
  Confirm the correct default for legacy RUFORUM content.
- **Abstract HTML** — stored as HTML; we strip to text. Confirm no rich formatting must survive.
- **Multi-file documents** — confirm primary-file heuristic (largest PDF) vs. explicit curation.
- **Restricted docs (49 + 51 private)** — confirm target visibility (`RESTRICTED` vs `AUTHENTICATED`)
  and whether private files migrate at all.
- **Collections** — Drupal has no direct equivalent to the IILMP `Collection`. Need a default landing
  collection (e.g. "RUFORUM Legacy Repository") or a mapping rule from `field_ag_type`/domain.
- **Periodicals/conferences as entities** — modelled as flat fields here; promote to lookups only if
  browse-by-journal is required at launch.

---

## 10. Sequence

1. ☑ Model extensions (§4 + §4b) + migrations `0010`/`0011`.
2. ☑ Importer built — `manage.py import_drupal_repository` parses the dump in a single
   streaming pass (no MySQL container needed) and downloads PDFs over the §8 HTTP path.
3. ☑ **Full metadata import run locally** — 2,897 documents + 5,052 authors + 14 collections
   + AGRIS/AGROVOC keywords + regions/countries; 2,897 search vectors built. Coverage:
   abstract 95% · country 86% · year 98% · publisher 82% · ≥1 author 92% · ≥1 region 94%.
4. ☐ Full PDF pull (~3,000 files / 2.5 GB) — `--files-only` (sample of 150 validated).
5. ☐ Curator pass — spot-fix the 1,261 `document_type=other` (collections without a
   type keyword) + confirm default license/visibility.
6. ☐ Production import.

### Importer reference

| Command | Effect |
|---|---|
| `import_drupal_repository --dump /tmp/dump.sql --flush` | Wipe + full metadata import (all docs) |
| `… --flush --limit 25` | Same, first 25 docs (smoke test) |
| `… --files-only --max-files 150` | Download PDFs for the 150 newest docs only |
| `… --files-only` | Download **all** PDFs (~2.5 GB) |
| `… --download` | Metadata + files in one pass |

Idempotent on `Document.legacy_drupal_nid` (re-run updates, never duplicates).
Node-ref gotcha: `publisher` / `journal` / `conference` Drupal values are node nids — the
importer resolves them to the referenced node's title.