# IILMP Monolith (Django + HTMX + Tailwind)

This repository now follows the architecture defined in `setup/iilmp_django_architecture.md`, scaffolded in the project root as a production-oriented Django monolith with:

- Domain modules under `apps/` (`core`, `rims`, `rep`, `repository`, `mel`, `alumni`, `smehub`)
  - REP sub-apps: `courses`, `content`, `assessments`, `forums`, `analytics`, `career`, `personalisation`, `networking`, `surveys`, `wiki`, `live`
- Versioned API layer at `api/v1/`
- Environment-specific settings at `config/settings/`
- Global templates/components/partials in `templates/`
- Tailwind pipeline and static assets in `static/`
- Docker, requirements, scripts, docs, and tests directories
- Storage utility in `apps/core/storage/` (registry + validators + upload handler + path generators)

## Quick Start (Container-first)

All `make` commands run inside Docker Compose containers.

### 0) Environment

Copy `.env.example` to `.env` and set `SECRET_KEY`, optional email, and superuser vars as needed. **Development Compose** overrides `DB_HOST`, `REDIS_URL`, and `CELERY_BROKER_URL` for `web`, Celery, and Flower so the same `.env` can keep `localhost` for running Django on the host while containers talk to `postgres` and `redis` on the Compose network.

The dev `web` image is built with `requirements/development.txt` so `make test`, `make lint`, and `make format` work without a separate `pip install` into the running container.

### 1) Bootstrap and start dev stack

```bash
cp .env.example .env
make install-js
make dev-start
```

Optional: `make install-dev` only if you added Python packages to `requirements/` and need a one-off install into a throwaway container; routine dev deps are already in the `web` image.

### 2) Migrate + create superuser

```bash
make migrate
make superuser
```

### 3) Check services

```bash
make compose-dev-ps
make compose-dev-logs
```

Application URL: `http://127.0.0.1:8000` by default, or `http://127.0.0.1:<WEB_HOST_PORT>` if you set `WEB_HOST_PORT` in `.env` (e.g. when 8000 is already taken).

Flower URL: `http://127.0.0.1:5555` by default, or the port from `FLOWER_HOST_PORT` in `.env`.

Mailhog (dev SMTP sink) URL: `http://127.0.0.1:8025` by default, or the port from `MAILHOG_UI_PORT` in `.env`. App containers send outbound mail to `mailhog:1025`; open the UI to inspect every captured message instead of relying on the console backend.

Redis in the dev stack is **not** published to the host by default (avoids clashing with other local Redis on 6379). Containers reach it as `redis:6379` on the Compose network.

If you override `WEB_HOST_PORT` / `FLOWER_HOST_PORT`, add the same origins to `CSRF_TRUSTED_ORIGINS` and `CORS_ALLOWED_ORIGINS` in `.env`.

**Celery:** Do not set `CELERY_RESULT_BACKEND` in `.env` unless you mean it. Celery reads `CELERY_*` from the OS environment and overrides Django; a leftover `redis://127.0.0.1/...` breaks the web container (audit tasks use `.delay()`). The dev Compose file sets `CELERY_RESULT_BACKEND=django-db` to match `config/settings/base.py`.

`celery_worker`, `celery_beat`, and `flower` use `DJANGO_SETTINGS_MODULE=config.settings.base` in Compose so they do not load dev-only apps (the Celery image installs `requirements/base.txt` only, not `django-browser-reload`).

## Docker Compose (Dev + Prod)

Both Compose files run required infrastructure:

- `web` (Django)
- `postgres` (primary DB)
- `redis` (cache + Celery broker/result backend)
- `celery_worker`
- `celery_beat`
- `flower` (dev stack only; Celery monitoring UI)
- `tailwind` (dev stack only; continuous CSS watch/build)
- `mailhog` (dev stack only; SMTP sink + web inbox at `http://127.0.0.1:8025`)

### Development stack

```bash
cp .env.example .env
make dev-start
make compose-dev-ps
make compose-dev-logs
```

### Smoke-check in the browser

After `make dev-start`, wait a few seconds for migrations, seeds, and the Tailwind watcher to emit CSS. Use your `WEB_HOST_PORT` (default 8000) and `FLOWER_HOST_PORT` (default 5555) in URLs if you overrode them in `.env`.

- App: `/` on the web port (redirects to login or dashboard)
- Admin: `/admin/`
- API docs: `/api/v1/docs/`
- Notifications (authenticated): `/notifications/`
- Flower (Celery): Flower host port (default 5555)
- Mailhog inbox: `http://127.0.0.1:8025` (or `MAILHOG_UI_PORT`)

```bash
make dev-stop
```

### Production-style stack

```bash
cp .env.example .env
make prod-deploy
make compose-prod-ps
make compose-prod-logs
```

```bash
make prod-stop
```

## Remote deployment (local-driven over SSH)

Production deploys are driven from your laptop via Make targets that SSH into the server and orchestrate `docker-compose.prod.yml` there. See `scripts/deploy.sh` for the underlying script.

### One-time setup (run in this order)

1. **Configure SSH + GitLab credentials** on your laptop:

   ```bash
   cp .env.deploy.example .env.deploy
   chmod 600 .env.deploy
   # Edit .env.deploy: SSH_HOST, SSH_USER, SSH_PORT, SSH_KEY (optional),
   # DEPLOY_PATH, GITLAB_USERNAME, GITLAB_TOKEN, GITLAB_REPO, LOCAL_ENV_FILE.
   # The legacy-migration dump paths are only needed for those one-time imports:
   #   REPOSITORY_DUMP_FILE                      (see "Migrating the legacy Repository")
   #   MOODLE_DUMP_FILE, MOODLE_FILEDIR_ARCHIVE  (see "Migrating the legacy Moodle + SME-Hub data")
   #   EHC_DUMP_FILE, POS_DUMP_FILE              (same section)
   ```

2. **Author the runtime production env** on your laptop:

   ```bash
   cp .env.example .env.production
   chmod 600 .env.production
   # Edit .env.production: set DEBUG=False, real SECRET_KEY, ALLOWED_HOSTS,
   # CSRF_TRUSTED_ORIGINS, DB_PASSWORD, EMAIL_*, OPENAI_API_KEY, BBB_*,
   # DJANGO_SETTINGS_MODULE=config.settings.production, etc.
   ```

3. **Ensure your SSH key reaches the server** (`ssh-add ~/.ssh/<key>` or set `SSH_KEY` in `.env.deploy`). Confirm Docker Engine + `docker compose` are installed on the server.

4. **Bootstrap the server** (clones the repo + scp's `.env.production` → `${DEPLOY_PATH}/.env`):

   ```bash
   make deploy-init
   ```

### Daily deploy workflow (run in this order)

```bash
git add -A && git commit -m "..."          # 1. commit your changes
git push                                   # 2. push current branch to GitLab
make redeploy                              # 3. SSH in, pull, rebuild prod stack
```

`make redeploy` will:

- Resolve the current local branch (`git rev-parse --abbrev-ref HEAD`).
- Refuse if the branch is not on `origin` (run `git push -u origin <branch>` first).
- Prompt for confirmation when the branch is not `main` (skip with `CI=1 make redeploy`).
- scp `.env.production` → `${DEPLOY_PATH}/.env` on the server.
- On the server: `git fetch` → `git reset --hard origin/<branch>` → **rebuild Tailwind CSS in a one-off `node:20-alpine` container** (so any utility class added to a template since the last `output.css` commit lands in the bundle — Tailwind JIT only ships classes it saw at build time) → `docker compose -f docker-compose.prod.yml up -d --build` → `migrate --noinput` → `collectstatic --noinput` → restart `web` / `celery_worker` / `celery_beat` / `nginx`.

The Tailwind build uses a named `iilmp_node_modules` volume so `npm ci` is cached between deploys (first run ~30s, subsequent runs ~3s).

If `MIGRATE_REPOSITORY=TRUE` in `.env.production`, `make redeploy` also runs the legacy Repository migration as a final step once the site is verified healthy — see [Migrating the legacy Repository](#migrating-the-legacy-repository-one-time-flag-gated) below. The same gate exists for `MIGRATE_MOODLE` and `MIGRATE_SMEHUB` — see [Migrating the legacy Moodle + SME-Hub data](#migrating-the-legacy-moodle--sme-hub-data-one-time-flag-gated).

### Ops helpers (run any time from your laptop)

```bash
make deploy-status      # docker compose ps on the server
make deploy-logs        # tail prod docker compose logs over SSH
make deploy-shell       # interactive bash inside the remote web container
make deploy-css-build   # rebuild Tailwind CSS only (no full redeploy)
make deploy-seed        # run seed_all --volume demo on the server (y/N prompt)
make deploy-seed-heavy  # run seed_all --volume heavy on the server (y/N prompt)
make deploy-migrate-repository  # wipe + re-import the legacy Repository (y/N prompt)
make deploy-migrate-moodle      # import legacy Moodle courses + media (y/N prompt)
make deploy-migrate-smehub      # import legacy SME-Hub EHC + UltimatePOS (y/N prompt)
```

`make deploy-css-build` is for the case where only templates changed (no Python / migration / dependency changes) and the redeploy churn is overkill. It runs the same Tailwind container step as `make redeploy`, then `collectstatic --noinput`, then restarts `web` + `nginx` so the new manifest-hashed CSS is served.

The seed commands invoke `apps/core`'s `seed_all` orchestrator inside a one-off
prod-web container. Both ask for confirmation before writing to the production
DB; set `YES=1` (e.g. `YES=1 make deploy-seed`) to skip the prompt in CI. For
finer control, call the script directly to pass through `--only` / `--skip`:

```bash
bash scripts/deploy.sh seed demo --only repository alumni
bash scripts/deploy.sh seed heavy --skip rep_live
```

### Migrating the legacy Repository (one-time, flag-gated)

The legacy Drupal repository (`repository.ruforum.org`) is imported by a
flag-gated step in the deploy. When enabled it **WIPES** the Repository tables
(documents, authors, keywords, collections) and re-imports ~2,900 documents from
the Drupal SQL dump, downloading ~2.5 GB of PDFs from the live site (**60–100 min**;
the Repository is empty until it repopulates). Every other module is untouched.

1. **Point `.env.deploy` at the local dump** (the 1.5 GB `rufoadm_drpl1.sql`):

   ```bash
   REPOSITORY_DUMP_FILE=/absolute/path/to/rufoadm_drpl1.sql
   ```

2. **Turn the flag on in `.env.production`:**

   ```bash
   MIGRATE_REPOSITORY=TRUE
   ```

3. **Run it** — folded into a normal redeploy, or standalone:

   ```bash
   make redeploy                   # deploys, then migrates AFTER the site is healthy
   # or, without a full redeploy:
   make deploy-migrate-repository  # y/N prompt; YES=1 make deploy-migrate-repository to skip
   ```

   `deploy.sh` uploads the dump to the server (skipped if an identical copy is
   already there), then runs `import_drupal_repository --dump … --flush --download`
   in a one-off container with the dump mounted read-only. Downloaded PDFs land in
   the persistent `uploads_data` volume.

4. **Set `MIGRATE_REPOSITORY=FALSE` again** so the next `make redeploy` does not
   re-wipe + re-migrate. The deploy prints this reminder on success.

> Run during a maintenance window. The server needs outbound HTTPS to
> `repository.ruforum.org` for the PDF downloads. The importer is idempotent on
> `legacy_drupal_nid`, and a no-flush re-run (`make deploy-migrate-repository`
> re-uses the uploaded dump) is safe.

**Enable semantic search** over the imported corpus for the IILMP Assistant
(needs `OPENAI_API_KEY`; otherwise the assistant uses its full-text fallback):

```bash
make deploy-shell
# inside the web container:
python manage.py embed_repository_documents
```

### Migrating the legacy Moodle + SME-Hub data (one-time, flag-gated)

Two further legacy platforms import into IILMP the same flag-gated way as the
Repository, but with one structural difference: their importers (`import_moodle`,
`import_smehub_ehc`) **connect to a live MariaDB over the network** rather than
reading a mounted `.sql` file. So each target uploads the dump, stands up an
**ephemeral `mariadb:10.11` container on the prod compose network**, loads the dump
into it, runs the importer against it (resolving the source by container name), then
tears the temporary DB down. Both importers are **idempotent** (upsert on
`external_ref` / `lms_ref` anchors), so the targets do **not** `--flush` — re-running
updates rather than duplicates, and every other module is untouched.

> **Precondition:** run `make redeploy` first so the Django schema migrations have
> created the target tables (badges, community posts, …). The migrate targets fail
> fast if the prod `web` service isn't running.

**Moodle** (courses, gradebook, badges, deadlines, forums + the `moodledata/filedir`
blob store — SCORM, avatars, assignment files, ~1.1 GB):

1. **Point `.env.deploy` at the local dump + blob archive.** `MOODLE_FILEDIR_ARCHIVE`
   may be either a `filedir`-rooted tar (`tar czf moodle_filedir.tar.gz -C /path/to/moodledata filedir`)
   **or** a full `moodledata_*.tar.gz` — the deploy script auto-locates the `filedir/`
   blob store inside whatever it extracts.

   ```bash
   MOODLE_DUMP_FILE=/absolute/path/to/db_moodle_db_*.sql           # ~148 MB
   MOODLE_FILEDIR_ARCHIVE=/absolute/path/to/moodledata_*.tar.gz    # ~960 MB compressed
   ```

2. **Run it** — folded into a redeploy via `MIGRATE_MOODLE=TRUE` in `.env.production`,
   or standalone:

   ```bash
   make deploy-migrate-moodle   # y/N prompt; YES=1 make deploy-migrate-moodle to skip
   ```

**SME-Hub (EHC)** (entrepreneurs, businesses, mentors, competitions, applications,
results, meetings, FAQs, subscribers, community archive) **+ UltimatePOS** (sales +
products). EHC uploaded files (Laravel `storage/`) are **not** in the dump and the
server pull is currently blocked, so this is a metadata-only EHC pass — the importer's
`files` stage self-skips and media can be attached later once the pull is unblocked:

1. **Point `.env.deploy` at both dumps:**

   ```bash
   EHC_DUMP_FILE=/absolute/path/to/smehub_ehc.sql            # ~31 MB
   POS_DUMP_FILE=/absolute/path/to/smehub_accounting.sql     # ~1 MB
   ```

2. **Run it** — via `MIGRATE_SMEHUB=TRUE` in `.env.production` on the next redeploy, or:

   ```bash
   make deploy-migrate-smehub   # y/N prompt; YES=1 make deploy-migrate-smehub to skip
   ```

   This stands up two ephemeral MariaDBs (`ehc-src`, `pos-src`) and runs
   `import_smehub_ehc --stage all` (metadata) then `--stage pos` (sales + products).

3. **Set the flag(s) back to `FALSE`** in `.env.production` after each migration so the
   next `make redeploy` doesn't re-run the import. The deploy prints this reminder on
   success.

> Imported avatars / logos / assignment files land in the persistent `uploads_data`
> volume. Each importer writes a per-stage CSV summary inside its one-off container
> (`/tmp/moodle_import.csv`, `/tmp/ehc_import.csv`) — watch the deploy output for the
> created / matched / skipped counts.

Both `.env.deploy` and `.env.production` are gitignored — never commit either. The GitLab PAT in `.env.deploy` only needs `read_repository` scope. The server's `.env` is overwritten by `LOCAL_ENV_FILE` on every redeploy, so edit `.env.production` locally rather than touching the server file directly.

## Makefile Commands

Use `make help` to list all targets.

Most-used commands:

```bash
make run               # alias for dev-start
make dev-start         # web + postgres + redis + celery + flower + tailwind watcher
make dev-stop          # stop development stack
make css-watch         # tailwind watcher in node container
make worker            # start celery worker service
make beat              # start celery beat service
make prod-deploy       # build/start services, migrate, collectstatic, then ensure app services are up
make prod-start        # start production-profile stack
make prod-stop         # stop production-profile stack
make test              # pytest in web container
make lint              # ruff check in web container
make format            # black + ruff --fix in web container
make check             # django checks in web container
```

## Dependency Matrix

Dependencies now align with the recommended architecture packages.

- Core web/app: `Django`, `django-htmx`, `django-tailwind`, `whitenoise`, `django-browser-reload`
- Forms/UI: `django-crispy-forms`, `crispy-tailwind`
- API/Auth: `djangorestframework`, `drf-spectacular`, `djangorestframework-simplejwt`, `django-allauth`, `django-guardian`
- Async/queue: `celery`, `django-celery-beat`, `django-celery-results`, `flower`
- Data/cache/storage: `psycopg2-binary`, `redis`, `django-redis`, `django-storages`, `boto3`, `python-magic`
- Query/filter/cors: `django-filter`, `django-cors-headers`
- Search/import/workflow: `django-haystack`, `elasticsearch-dsl`, `django-import-export`, `django-fsm`
- AI / vector: `langchain`, `langchain-openai`, `langchain-postgres`, `pgvector`, `scikit-learn` — REP recommendations (WS3.1) + repository AI insights
- HTML sanitisation: `bleach` — collaborative wiki body cleaning (WS4.3)
- Realtime/reporting: `channels`, `daphne`, `reportlab`, `weasyprint`, `openpyxl`
- Quality/testing: `pytest`, `pytest-django`, `pytest-cov`, `factory_boy`, `ruff`, `black`

## Project Layout

Top-level structure:

```text
apps/                  # domain modules
api/v1/                # DRF versioned API
config/settings/       # base/development/staging/production settings
templates/             # global layouts/components/partials
static/                # css/js/images assets
uploads/               # media uploads
docker/                # Dockerfiles + nginx config
requirements/          # env-specific Python requirements
scripts/               # utility scripts
tests/                 # shared pytest setup
```

## Demo data and test accounts

After Postgres and Redis are up, load roles, superuser, RIMS fixtures, and REP fixtures (from the repo root, using the Compose `web` service):

```bash
docker compose run --rm \
  -e DJANGO_SUPERUSER_EMAIL=admin@ruforum.org \
  -e DJANGO_SUPERUSER_PASSWORD=password123 \
  web sh -c "
    python manage.py migrate --noinput &&
    python manage.py seed_roles &&
    python manage.py seed_core &&
    python manage.py seed_rims --reset --comprehensive --password password123 &&
    python manage.py seed_rep --reset
  "
```

- **`seed_core`** creates/updates the superuser only when `DJANGO_SUPERUSER_EMAIL` is set. If `DJANGO_SUPERUSER_PASSWORD` is empty and `DEBUG=True`, the app falls back to `password123` (see `apps/core/management/commands/seed_core.py`).
- **`seed_rims --comprehensive`** sets every seeded `@demo.local` user’s password to the value of `--password` (default `password123`).
- **`seed_rep`** creates `@demo.rep` learner accounts with **`DemoPass123!`** on first creation; REP role accounts use **`password123`** to match RIMS seeds.

### Password conventions

| Family | Password | Notes |
|--------|----------|--------|
| Superuser (env) | `DJANGO_SUPERUSER_PASSWORD` or `password123` when allowed by `seed_core` | Example above uses `password123`. |
| RIMS + REP role demos (`@demo.local`, `rep.*@demo.local`) | `password123` | Override RIMS with `seed_rims ... --password <pw>`. |
| REP catalogue learners (`@demo.rep`) | `DemoPass123!` | Set only when the user row is first created by `seed_rep`. |

### Account list (local / CI demos)

| Email | Password | Role / use |
|-------|----------|------------|
| Set via `DJANGO_SUPERUSER_EMAIL` (e.g. `admin@ruforum.org`) | `DJANGO_SUPERUSER_PASSWORD` or `password123` in DEBUG | Django superuser / staff |
| `grants.manager@demo.local` | `password123` (or `--password`) | Grants manager |
| `reviewer@demo.local` | same | Reviewer |
| `reviewer2@demo.local` | same | Reviewer |
| `scholar@demo.local` | same | Scholar |
| `finance@demo.local` | same | Finance officer |
| `mel@demo.local` | same | M&E officer |
| `award.recipient@demo.local` | same | Award recipient flows |
| `eligibility.applicant@demo.local` | same | Eligibility E2E |
| `interview.applicant@demo.local` | same | Interview gate E2E |
| `conflict.applicant@demo.local` | same | Reviewer conflict E2E |
| `blind.applicant@demo.local` | same | Blind review E2E |
| `rep.instructor@demo.local` | `password123` | REP **course instructor** (seeded course author) |
| `rep.learning.admin@demo.local` | `password123` | REP **learning administrator** (programmes / certificate templates) |
| `amara.diallo@demo.rep` | `DemoPass123!` | REP learner |
| `fatima.njoku@demo.rep` | `DemoPass123!` | REP learner |
| `kwame.asante@demo.rep` | `DemoPass123!` | REP learner |
| `alice.mwangi@demo.rep` | `DemoPass123!` | REP learner |
| `ibrahim.toure@demo.rep` | `DemoPass123!` | REP learner |
| `grace.osei@demo.rep` | `DemoPass123!` | REP learner |

Re-run `python manage.py seed_roles` after adding new `UserRole` values so Django groups and default permission maps stay in sync.

## Notes

- **Tailwind:** `static/css/output.css` is built from `input.css` and **tracked in git** so the app works without Node on first load. After you add or change Tailwind classes in templates/JS, run `make css-build` (or `npm run build:css` locally). The dev `tailwind` Compose service runs `watch:css` to keep `output.css` updated. Dependencies are pinned to Tailwind **v3** (`tailwind.config.js` + `@tailwind` directives are not compatible with Tailwind v4 without a migration).
- Default local DB is PostgreSQL via Docker Compose (`postgres:5432`). The `pgvector/pgvector:pg16` image is used so the `vector` extension can be created on demand by `rep_courses/0008_course_embedding`.
- `AUTH_USER_MODEL` is set to `core.CustomUser`.
- Static is served from `staticfiles/` in production via WhiteNoise.
- **Vendored JS:** `static/js/vendor/` holds `quill.min.js` (rich-text editor for cert templates + wiki) and `pdf.min.js` + `pdf.worker.min.js` (PDF.js for the assignment annotation viewer, WS4.6). All three are tracked in git so the app runs without an npm install at boot.
- Media uploads are served from `uploads/` with module roots:
  - `uploads/rims`, `uploads/rep`, `uploads/repository`, `uploads/mel`, `uploads/alumni`, `uploads/smehub`
- Production compose mounts a shared `uploads_data` volume at `/app/uploads` for `web`, `celery_worker`, and `celery_beat`.

## REP advanced features (post-baseline sprints)

These five capabilities sit on top of the baseline REP module and each have a dedicated env / data-layer touchpoint. See `.env.example` for the variables and `setup/PRD.md` §5.5 for the functional requirements they implement.

| Feature | Where it lives | What to set up |
|---|---|---|
| **WS3.1 — ML course recommendations** (FRREP-PL002) | `apps/rep/personalisation/embeddings.py`, `Course.embedding` (1536-d pgvector) | Set `OPENAI_API_KEY` for production-grade embeddings; otherwise a deterministic local fallback runs. After migrating, run `python manage.py backfill_course_embeddings` to populate vectors for published courses. |
| **WS4.6 — PDF annotation on assignment grading** (FRREP-AG007) | `apps/rep/assessments/annotation_views.py`, `static/js/assignment_annotation.js`, `static/js/vendor/pdf.{min,worker.min}.js` | No env vars needed. Annotations are stored in PDF user-space coords and rendered on both the grader's view and the learner's submission view (read-only). |
| **WS4.3 — Collaborative wiki** (FRREP-LD006) | `apps/rep/wiki/` | Adds `bleach>=6.1` as a hard dependency. `REP_WIKI_PAGE_UPDATED` notification verb + template are seeded by `core/0028_seed_wiki_notification_template`. |
| **WS4.2 — SCORM 1.2 + xAPI score capture** (FRREP-LD008 / FRREP-LD009) | `apps/rep/content/scorm_views.py`, `apps/rep/content/scorm_xapi.py`, `static/js/scorm/api.js` | xAPI LRS endpoint authenticates via `X-Rep-Api-Key` (use `python manage.py create_rep_api_client <name>` to mint credentials). SCORM commits authenticate as the enrolled learner. Terminal pass / completed states emit `quiz_submitted` outbox events into the existing `rep-quiz-submissions` indicator. |
| **WS4.1 — BigBlueButton live classes** (FRREP-LD003 / FRREP-LD004) | `apps/rep/live/`, `apps/rep/live/bbb_client.py` | Set `BBB_URL` and `BBB_SHARED_SECRET` (defaults point at the BBB public test instance for dev). Two beat tasks (`rep-live-session-starting-soon`, `rep-live-recording-sync`) are wired in `config/celery.py` — make sure `celery_beat` is running. New M&EL indicators: `rep-live-sessions-scheduled`, `rep-live-attendance`. |