# sim.pdhc — Technical reference

A Python tool with two operator surfaces (CLI + Web UI at
`127.0.0.1:9099`) that generates synthetic Swedish-cohort FHIR R5 data
and pushes it to one or more `cdr.pdhc` instances. The Web UI is a
single-process Flask app — not a long-running prod service, but a
durable operator-facing surface that includes a Runs viewer reading
back from cdr_6. No DB of its own, no SSO, no Docker. Three long-form
references already exist — this document is the architectural overview
that ties them together:

- [`profile_v2_format.md`](profile_v2_format.md) — full v2 profile YAML reference
- [`describe_to_profile.md`](describe_to_profile.md) — prose → YAML tool
- [`phase_3_operator_runbook.md`](phase_3_operator_runbook.md) — T9 prep for `cdr_6.pdhc`
- [`user_manual.md`](user_manual.md) — operator-facing manual (CLI + Web UI)

---

## 1. What sim does and doesn't do

sim.pdhc is the **producer** in the longitudinal-data pipeline:

```
sim.pdhc   →   cdr_6.pdhc   →   analyse.pdhc (future)
 CREATE         STORE             READ + analyse
 cohort data    flat-row DB       to be built later
```

It generates longitudinal time series for synthetic cohorts based on
`plan.pdhc` PlanDefinitions + their leaf Terms (concepts), writes the
rows to `cdr_6.pdhc`, and exits. cdr_6 is the durable store;
`analyse.pdhc` (planned, not yet built) will be the primary consumer.

**Does:**
- Generates FHIR R5 transaction Bundles of `Patient`, `Observation`,
  `Condition`, `MedicationStatement`, `Encounter`.
- Writes per-run output to `results/<UTC-ts>_<label>/` with
  `manifest.json`, `bundles/`, `observations.csv`, `patients.csv`, and
  (for cdr_6 targets) `cdr_6_rows.ndjson`.
- Pushes those bundles to cdr1–cdr5 (FHIR bundle ingest) or cdr_6
  (flat-row ingest with characterisation columns).
- Purges a previous run from a CDR by `sim_run_id` tag.

**Doesn't:**
- Hold any PHI — every Patient carries `SYNTHETIC` identifier system
  and `meta.tag = SYNTHETIC`. Real-data dashboards filter this tag out
  by default.
- Run as a service. `start.sh` is a venv-activating wrapper, not a
  process supervisor.
- Operate without a seed. `--seed` is required on every run; same
  `(profile, count, seed)` ⇒ byte-identical NDJSON output.

---

## 2. Package layout

```
sim/
├── __main__.py            # `python -m sim` entry point
├── cli.py                 # argparse + dispatch
├── config.py              # SimConfig, TARGETS registry, concept canonicals
├── patient.py             # PatientFactory (v0.1 hardcoded path)
├── variables/             # one engine per variable kind (v0.1 path)
│   ├── cgm.py
│   ├── hba1c.py
│   ├── vitals.py
│   ├── labs.py
│   ├── encounter.py
│   └── condition_med.py
├── fhir.py                # FhirBuilder + Bundle chunking
├── client.py              # CdrClient — FHIR-bundle + cdr_6 flat-row push
├── plandef_client.py      # plan.pdhc PlanDefinition resolver
├── ips_client.py          # ips.pdhc — clinics list + patients-by-clinic
├── profile_schema.py      # v2 profile JSON-Schema + is_v2_profile()
├── v2_engine.py           # plandef-driven generator (Phase 1–2 of #178)
├── cohort_builder.py      # plandef + IPS roster → cdr_6 flat rows (Cohort Builder engine)
├── describe_to_profile.py # Anthropic tool-use → v2 YAML emission
├── tunnel.py              # SSH port-forward for loopback-only targets
├── run.py                 # generate → write outputs orchestration
├── csv_cohort.py          # v1 CSV-cohort engine (kept for callers; UI flow gone)
└── web/                   # Flask app — Cohort Builder + Runs viewer
    ├── app.py             #   routes, proxies, tunnel manager
    └── templates/
        ├── cohort.html    #   Cohort Builder page (4 panels + factor editors)
        └── runs.html      #   Runs viewer (cdr_6 list + sample + charts)
```

---

## 3. Generation paths

Three paths, picked by CLI subcommand:

| Path | Entry | Patients | Engine | Concepts |
|---|---|---|---|---|
| **v0.1** | `sim generate/run` with no plan_definition in profile | Synthesised | `patient.PatientFactory` + `sim/variables/*` | Hardcoded set |
| **v2** | `sim generate/run` with v2 profile | Synthesised | `v2_engine.run_v2` | Resolved from plan.pdhc PlanDefinition |
| **Cohort Builder** | `sim cohort-build`, Web UI `/` | **Real patients from ips.pdhc** | `cohort_builder.build_cohort_rows` (reuses v2_engine helpers) | Resolved from plan.pdhc |

`profile_schema.is_v2_profile()` discriminates v0.1 from v2:

```python
return bool(profile.get("plan_definition")
            or profile.get("concepts_override"))
```

The Cohort Builder is the production path for "simulate against this
real organisation's roster"; the v2 `run` path is the production path
for "simulate a synthetic cohort against this plandef". v0.1 is kept
for legacy tests and `cohort_a.yaml` examples. Schema lives in
`sim/profile_schema.py:PROFILE_V2_SCHEMA`; new fields are added there
first, then taught to the engine.

---

## 4. Determinism contract

Same `(profile, count, seed)` ⇒ byte-identical
`manifest.json`+`bundles/*.json`+`observations.csv`. Mechanism:

- One RNG per `(patient_seq, concept_index)` pair, seeded as
  `default_rng(effective_seed + seq*1_000_003 + concept_index*977)`
  (`v2_engine.py:588`).
- Loss draws use a separate parallel RNG with a +2³¹−1 offset
  (`v2_engine.py:166`) so adding `loss:` to a profile never perturbs
  the values of measurements that are *not* dropped.
- The value RNG is drawn against the full schedule even when a
  timestamp is dropped (`v2_engine.py:601-606`) — the mask is applied
  at emit time so loss only changes which rows ship, never their
  values.

This contract is load-bearing for diffing runs against each other.
Anything that breaks it (e.g., changing the RNG seeding scheme) is a
breaking change.

---

## 5. CDR targets

`sim/config.py:TARGETS` is the canonical registry:

```python
"cdr1":  TargetSpec("cdr1",  "https://cdr1.pdhc.se", 9046, 19_001),
"cdr2":  TargetSpec("cdr2",  "https://cdr2.pdhc.se", 9146, 19_002),
"cdr3":  TargetSpec("cdr3",  "https://cdr3.pdhc.se", 9246, 19_003),
"cdr4":  TargetSpec("cdr4",  "https://cdr4.pdhc.se", 9346, 19_004),
"cdr5":  TargetSpec("cdr5",  "https://cdr5.pdhc.se", 9446, 19_005),
"cdr_6": TargetSpec("cdr_6", None,                   9055, 19_006),  # loopback-only
```

- `cdr1`–`cdr5` are public via nginx → bundle ingest at
  `POST /api/v1/bundles`.
- `cdr_6` is loopback-only on miserver (Phase 3.b, 2026-05-29). No
  public URL. Reached via SSH port-forward (`sim/tunnel.py`).
  Authentication is service-key, **not** bearer token:
  - `X-Source-Service: sim.pdhc`
  - `X-Service-Key: <SIM_PDHC_SERVICE_KEY>`
  - Ingest endpoint: `POST /api/v1/ingest/batch` (flat rows)
  - Purge: `DELETE /api/v1/runs/<run_id>`

The push command picks a target via `--target <name>` (preferred) or
`--cdr <url>` (explicit override). When `--tunnel` is set the tunnel
is opened atomically with `tunnel_for_target` and closed on exit.

---

## 6. Plan.pdhc integration

`sim/plandef_client.py` resolves a PlanDefinition reference into its
leaf concept GUIDs.

- The v2 profile's `plan_definition:` block accepts any of `guid`,
  `title`, `fhir_id` (schema: `anyOf`). Resolution order in
  `plandef_client.py:98-118` is **guid > title > fhir_id**.
- A profile carrying both `guid` and `title` is valid; the resolver
  uses `guid`. (This was a #179.1 finding — the schema was wrongly
  `oneOf`; fixed in commit `24be44e`.)
- Concept-canonical references in emitted FHIR use indirect referencing:
  `system = https://plan.pdhc.se/Concept`, `code = <Concept GUID>`.
  The CDR resolves to the underlying canonical (LOINC/SNOMED/etc) on
  write. Mapping table: `sim/config.py:_ALIAS`.

The `concepts_override:` block bypasses plan.pdhc entirely — useful
for tests, dogfood profiles, and the dist-quality / scale soak runs.

---

## 7. CDR client

`sim/client.py:CdrClient` ships two unrelated push code paths:

| Method | Target | Payload | Endpoint |
|---|---|---|---|
| `push_bundles_concurrent` | cdr1–cdr5 | FHIR transaction Bundles, batched | `POST /api/v1/bundles` |
| `push_cdr_6_flat` | cdr_6 | Flat-row NDJSON batches with characterisation columns | `POST /api/v1/ingest/batch` |
| `purge_run` | cdr1–cdr5 | run-id tag | `DELETE /api/v1/runs/<run_id>` (bundle ingest) |
| `purge_run_cdr_6` | cdr_6 | run-id | `DELETE /api/v1/runs/<run_id>` (flat-row) |

Concurrency cap is `SIM_CDR_CONCURRENCY` (default 8). HTTP timeout is
`SIM_CDR_TIMEOUT` (default 30s).

Flat-row characterisation columns (cdr_6 only) — `author_org_guid`,
`plan_definition_guid`, `care_plan_guid`, `activity_guid`,
`transaction_guid` — are author-set on the profile/term blocks and
emitted as separate columns rather than embedded in FHIR. See
[`profile_v2_format.md`](profile_v2_format.md) for the full mapping.

---

## 8. describe-to-profile (Anthropic SDK)

`sim/describe_to_profile.py` is author-time only. No LLM in the run
path — once the YAML is committed, generation is fully deterministic.

- Uses the Anthropic Messages API with forced tool use (`emit_profile`
  tool, single string parameter `yaml`).
- Schema reminder block (`_SCHEMA_REMINDER`) is what Claude sees as
  the structural prompt. **Update it here when the schema changes.**
- The output goes through `validate_v2_profile` before being written
  to disk — bad emissions surface as a validation error instead of a
  broken YAML.

Key environment variable: `ANTHROPIC_API_KEY`. Failure to install the
`anthropic` SDK surfaces as a CLI-friendly error.

---

## 9. SSH tunnel for loopback-only targets

`sim/tunnel.py` opens an SSH master tunnel on a deterministic Unix
socket under `/tmp/`, forwards `local_tunnel_port → remote_app_port`,
registers an `atexit` cleanup, and yields the local URL for the
duration of the `with` block.

- Tunnel host comes from `SIM_TUNNEL_HOST` (default `miserver@192.168.1.154`).
- Used by `cdr_6` push/run/purge — no other target needs it.
- `--tunnel` on the CLI is the user-facing flag; the registry decides
  the ports.

---

## 10. Concept canonical registry

`sim/config.py:CANONICAL` and `coding_for(name)` build the FHIR coding
arrays. Source of truth:
`concepts/diabetes_v1.guids.json` (generated by `concepts/load_to_plan.py`).

- Indirect referencing — `coding[0]` is always
  `https://plan.pdhc.se/Concept/<guid>`. The downstream CDR resolves
  to LOINC/SNOMED/ICD via plan.pdhc.
- Diagnoses with `icd10_alt` in the manifest get a 2-coding list
  (Concept GUID + ICD-10) for dual-coding.
- Falls back to legacy direct URIs if the manifest isn't loaded
  (tests). Raises if neither path resolves.

---

## 11. Tests

`tests/` — pytest, 127 cases as of 2026-06-03 (was 74 pre-Step-B). Run with:

```bash
.venv/bin/python -m pytest -x -q
```

Test files map cleanly to modules: `test_v2_engine.py`,
`test_profile_schema.py`, `test_plandef_client.py`,
`test_describe_to_profile.py`, `test_cli_describe.py`,
`test_client_cdr_6.py`, and the v0.1 legacy tests.

---

## 12. Validation evidence (#179)

| Item | Result | Doc |
|---|---|---|
| Prose dogfood (8 inputs, EN+SV) | 8/8 pass post-schema-fix | `docs/179_validation/prose_dogfood.md` |
| Multi-concept end-to-end | pass | covered in 179_validation |
| Distribution quality (4 families × 1000) | all 4 within ≤1 SE of corrected targets | `docs/179_validation/distribution_quality.md` |
| Scale + duration (10k × 3yr × 5 concepts) | 1,157,528 rows / 0 errors / 364s | `docs/179_validation/scale_duration.md` |
| Dashboard dogfood | not yet run — #179.5 | — |
| Operational soak (≥7 days) | not yet run — #179.6 | — |

---

## 13. What lives where on disk

| Path | Purpose | Persistence |
|---|---|---|
| `profiles/*.yaml` | Author-committed v2 profiles | git |
| `concepts/diabetes_v1.guids.json` | Generated GUID manifest | git |
| `results/<UTC>_<label>/` | Per-run output | gitignored (Rule 11) |
| `scripts/*.py` | One-off investigation scripts (e.g. `180_repro.py`) | git |
| `.env` | Local secrets (CDR_AUTH_TOKEN, ANTHROPIC_API_KEY, SIM_PDHC_SERVICE_KEY, IPS_PDHC_API_KEY) | gitignored |
| `.env.example` | Schema for the operator | git |

---

## 14. Out of scope

- No DB of its own. cdr_6 holds all persistent observations; ips.pdhc
  holds the patient registry; plan.pdhc holds the concept + plandef
  catalogue.
- No SSO. Pushes use service-level credentials (bearer for cdr1–5,
  X-Service-Key for cdr_6, `Authorization: ApiKey` for ips.pdhc).
- No long-running web server. `sim/web/` is a single-process dev/ops
  surface, started on demand. It does NOT run on miserver as a service;
  containerisation (#157 Option C) doesn't apply.
- No analysis. cdr_6 stores; the future `analyse.pdhc` reads and
  analyses (see `project_sim_cdr6_analyse_pipeline` memory).

---

## 15. IPS integration (Cohort Builder)

Operator-facing flow: [`user_manual.md` §4 Cohort Builder Web UI](user_manual.md#4-web-ui--cohort-builder)
+ [§8 Scenario A](user_manual.md#a--cohort-builder-smoke-via-cli-current-default).

The Cohort Builder path treats `ips.pdhc` as the source of truth for
"which patients belong to which organisation". The relationship is
many-to-many — one patient can belong to several clinics; one clinic
has many patients. See `project_patients_live_in_ips` memory + the
ips.pdhc commit `c399add` for the data model.

### Dependency chain

```
Cohort Builder run
   │
   ├─► sim.plandef_client.get_plandefinition(guid)         (plan.pdhc, public HTTPS)
   ├─► sim.ips_client.list_clinic_patients(clinic_guid)    (ips.pdhc, ApiKey)
   ├─► sim.cohort_builder.walk_plandef_terms(plandef)
   ├─► sim.cohort_builder.build_cohort_rows(...)
   └─► sim.client.CdrClient.push_flat_rows(...)            (cdr_6, X-Service-Key + tunnel)
```

### ips_client

`sim/ips_client.py` — thin HTTP client:

- `IpsClient(base_url, api_key, timeout)` dataclass.
- `list_clinics()` → list of `{guid, name, organisation_guid, ...}` dicts.
- `list_clinic_patients(clinic_guid)` → list of `PatientIndex.to_dict()`
  shapes (the patient roster).
- Auth via `Authorization: ApiKey <raw>` header (ips.pdhc's
  `auth_service.require_auth` accepts SSO Bearer or `ApiKey`).
- `ClinicNotFoundError` on 404; `IpsClientError` on other non-2xx.
- `make_client_from_env()` factory pulls `IPS_PDHC_BASE_URL` +
  `IPS_PDHC_API_KEY` via `SimConfig`.

### cohort_builder

`sim/cohort_builder.py` — plandef-driven engine:

- `walk_plandef_terms(plandef)` — preserves every
  `(activity_guid, transaction_guid)` pair. **Different from
  `plandef_client.expand_plandef`** which dedupes by concept_guid for
  the FHIR path; the cohort path needs both characterisation guids on
  every row so cdr_6 attribution works.
- `_stable_patient_seed(seed, patient_guid)` — hashes
  `(seed, patient_guid)` into a 64-bit int. **Strengthens v2_engine's
  determinism**: adding, removing, or reordering patients in the
  roster does NOT shift other patients' streams. v2_engine uses
  positional `seq` which would reshuffle everyone when a patient is
  added at position 0; that's fine for synthetic cohorts but
  unacceptable for real-roster flows.
- `build_cohort_rows(plandef, ips_patients, params, run_id)` — reuses
  `v2_engine._parse_cadence / _schedule / _apply_loss / _loss_rng /
  _draw_value`, so Phase 2.b's loss-doesn't-shift-values contract
  still holds. Returns sorted list of `cdr_6` ingest-batch rows. Does
  NOT emit FHIR Patient resources — patients come from IPS.

### Required state in ips.pdhc for the Cohort Builder to work

- ips.pdhc must be at `c399add` or later (the patients-by-clinic
  endpoint exists).
- ips.pdhc must be at `ce0f928` or later (`create_patient` and
  `generate_mock_data` write `PatientClinicAssignment` rows; without
  this, the endpoint returns empty even though FHIR
  `managingOrganization` is set).
- The chosen clinic must have at least one row in
  `patient_clinic_assignments`. The IPS admin dashboard has a
  "Generate Mock Patients" form with a `skip_clinical` checkbox —
  ideal for populating Test Clinic for a smoke run without polluting
  cdr_6 with two sources of truth for the same patients.

### Population factors (Phase 1)

Operator-facing reference: [`user_manual.md` §7 Population factors](user_manual.md#population-factors-optional).

`_population_shift(patient, pop_cfg, concept_guid, period_start)`
returns an **additive** shift to add to the value drawn from the
distribution. Two factors:

- **age** — `(age_years - ref_age, clamped non-negative) × shift_per_year`,
  applied to concepts listed in `apply_to`.
- **sex** — additive shift looked up by lowercased gender in `shifts:`,
  applied to concepts listed in `apply_to`.

Factors compose additively. Missing demographics fall through silently
(zero contribution from that factor, not an error).

**Determinism preservation:** the shift is applied to `v` **after**
`_draw_value` returns. The RNG stream is never consulted for the
shift; the only inputs are the operator's static `pop_cfg` constants
and the patient's static IPS demographics. So same `(plandef, params,
patients)` with population turned on vs off differs by exactly the
constant shift per row — RNG draws are byte-identical either way.

Out of scope for Phase 1 (could be future Phase 2):
- Multiplicative shifts (`v *= factor`)
- Percentile-based shifts (e.g., "highest decile +5")
- Comorbidity-based factors (would need a `Condition` index)
- Seasonal factors (would need to consult `effective_at`, not just
  patient demographics)

### Synthea hookup (Shape C)

Operator-facing flow: [`user_manual.md` §8 Scenario E](user_manual.md#e--populate-test-clinic-with-synthea-generated-patients-shape-c).

`sim.synthea_import` provides a thin importer from Synthea's FHIR R4
output to ips.pdhc's clinic roster:

- `iter_bundle_files(source)` — yields `*.json` under a directory,
  skipping Synthea's `hospitalInformation*` and
  `practitionerInformation*` sidecars (they don't carry Patient
  resources).
- `extract_patient(bundle)` — walks the transaction Bundle, returns
  the first `Patient` resource.
- `to_ips_payload(patient)` — maps Synthea's Patient to the minimal
  JSON the new `POST /api/v1/clinics/<guid>/patients` accepts. Picks
  the official name; identifier preference is **SSN > MRN > Synthea
  internal id**.
- `import_synthea(source, clinic_guid, base_url, api_key, max_count)`
  — drives the loop, returns one `ImportResult` per bundle with
  success/failure attribution.

Why this is Shape C (not B): only the **Patient identity layer**
crosses over. Synthea's Observations, Conditions, MedicationRequests,
etc. stay in the source bundles. cdr_6 keeps a single source of truth
for measurements (sim's deterministic engine), and operators can opt
to use Synthea-imported patients as a richer roster than the IPS
admin's mock-data form provides — the rest of the Cohort Builder flow
is unchanged.

Companion endpoint on ips.pdhc:

- `POST /api/v1/clinics/<guid>/patients` accepts either a flat shape
  (`family_name`/`given_name`/`gender`/`birth_date`/`identifier_*`)
  or a full FHIR Patient (`{"fhir": {...}}`). Creates the
  `PatientIndex` via the existing `create_resource("Patient", …)` path
  (so `_sync_patient_index` populates the row), then INSERTs a
  `PatientClinicAssignment` so the patient is queryable via
  `GET /api/v1/clinics/<guid>/patients`. Same `@require_auth` as
  siblings. Logs `clinic_patient_create` events.

### Synthea hookup (Shape B) — Observations into cdr_6

Operator-facing flow: [`user_manual.md` §8 Scenario F](user_manual.md#f--push-synthea-observations-to-cdr_6-shape-b).

`sim.synthea_obs` provides the optional Shape B layer on top of Shape
C. Enabled via `sim import-synthea --push-observations`. Architecture:

```
Synthea bundle (FHIR R4 transaction Bundle)
  ├── Patient                          ─► Shape C: POST ips.pdhc
  └── Observation[] (subject.reference)
       ├── code.coding[0].system, .code
       └── valueQuantity                ─► Shape B: map → push cdr_6 flat row
```

Module entry points (`sim/synthea_obs.py`):

- `ConceptCatalogue` — in-memory `(lib_name_lower, code) → concept_guid`
  map plus the reverse `concept_guid → (name, lib, code, …)` index.
- `load_concept_catalogue(plan_base_url, *, api_key=None, per_page=200)`
  — one round-trip to `/api/v1/lookup/canonical-libs` (build
  `lib_guid → name` map) plus a paginated walk of
  `/api/v1/concepts`. Builds the catalogue. At present ~70-100
  concepts in production → one call typically.
- `iter_observations(bundle, patient_resource_id)` — yields
  Observation resources whose `subject.reference` points at the named
  Patient. Accepts both `Patient/<id>` and `urn:uuid:<id>` styles
  (Synthea uses both).
- `extract_observation_value(obs)` — pulls `(response_type, value,
  effective_at)` from `valueQuantity` only. Non-quantity codings,
  missing value, and missing effective time all return `(None, None,
  None)` → caller drops the row.
- `build_cdr6_rows_for_patient(bundle, synthea_patient_resource_id,
  ips_patient_guid, catalogue, author_org_guid, run_id)` — drives
  the loop. Returns `(rows, stats)` where `stats` has counts for
  `extracted`, `kept_quantity`, `mapped`, `dropped_unmapped`,
  `dropped_non_quantity`.
- `make_synthea_run_id()` — emits `synthea-<UTC-stamp>-<6-hex>` so
  cdr_6's `/api/v1/runs` aggregate visibly distinguishes Synthea
  imports from `sim-...` cohort runs.

### System-URL normalisation

Synthea Observations use HL7 well-known URLs (`http://loinc.org`,
`http://snomed.info/sct`, …). plan.pdhc's canonical libraries carry
their own URLs (`https://LOINC.org`, etc.) that don't always
case-match. Sim normalises both sides into short lowercase names via
`_SYNTHEA_SYSTEM_TO_LIB_NAME` (hardcoded — small enumeration, less
fragile than URL parsing). Adding a new system (e.g., RxNorm,
ICD-10-CM) means adding one line there.

### Shape B characterisation contract

Every cdr_6 row emitted by Shape B has:

- `plan_definition_guid` = `NULL` (Synthea is not plandef-driven)
- `activity_guid` = `NULL`
- `transaction_guid` = `NULL`
- `care_plan_guid` = `NULL`
- `author_org_guid` = the clinic guid (same as Shape C)
- `sim_run_id` = `synthea-<stamp>-<rand>`
- `patient_guid` = the IPS guid (Shape C output)

NULL plandef attribution is **deliberate honesty**. The Runs viewer's
chart layout still works (it groups by `concept_guid`), but a
downstream consumer querying "rows authored under PlanDefinition X"
won't see Synthea rows — which is the correct behaviour, because they
weren't.

### Coverage gap

Plan.pdhc's concept catalogue is small (74 concepts at last count).
Synthea emits 600+ distinct LOINC codes across its modules. The
practical mapping rate against one of Synthea's adult-male bundles
(2026-06-03 smoke) was 126 mapped / 411 extracted = 31%. The drop
is silent by design — sim never invents concept_guids. Raising the
rate is a plan.pdhc concept-curation task, not a sim change.

---

## 16. Web UI architecture

Operator-facing flow: [`user_manual.md` §4 Cohort Builder](user_manual.md#4-web-ui--cohort-builder).

Single-process Flask app, started via `python -m sim.web`. Binds
`127.0.0.1:9099`. Stops on `Ctrl+C`. **Does not auto-load `.env`** —
the operator must `set -a && source .env && set +a` before launching.

### Routes

| Route | Source | What |
|---|---|---|
| `GET /` | `templates/cohort.html` | Cohort Builder page |
| `GET /runs` | `templates/runs.html` | Runs viewer page |
| `GET /api/plan-definitions[?search=…]` | plan.pdhc proxy | List PlanDefinitions |
| `GET /api/plan-definitions/<guid>` | plan.pdhc proxy | Fetch + attach `_terms` from `walk_plandef_terms` so the UI builds the override accordion without re-walking |
| `GET /api/clinics` | ips.pdhc proxy | List clinics |
| `GET /api/clinics/<guid>/patients` | ips.pdhc proxy | Roster |
| `POST /api/cohort-build` | Engine entry | `body: {plandef_guid, clinic_guid, params, patients_max?, push?, target?, include_rows?}`. Returns counts + (optional) row list. Optionally pushes to cdr_6. |
| `GET /api/cdr6/runs` | cdr_6 proxy | Runs list (via tunnel) |
| `GET /api/cdr6/runs/<id>/observations[?…]` | cdr_6 proxy | Row sample (via tunnel) |
| `GET /api/health` | — | `{"status":"ok"}` |

### cdr_6 tunnel management

`sim.web` keeps a **single long-lived SSH port-forward** to cdr_6
(loopback-only on miserver). `_ensure_cdr6_tunnel()` opens the tunnel
on first viewer hit (`ssh -fN -M -S <sock>`), `atexit.register` tears
it down at process exit. This avoids paying ~1s tunnel-open per
request — important for the viewer's drilldown UX.

Trade-off: a multi-worker deploy (e.g. `gunicorn -w 4`) would have
each worker opening its own tunnel. Out of scope for this single-
process dev/ops surface; document if/when it ever needs scaling.

### Cohort Builder form architecture

`templates/cohort.html` is a single-page form with four panels (Step
1: PlanDefinition, Step 2: Organisation, Step 3: Parameters, Step 4:
Generate). The Parameters panel uses a **shared factor-editor
component** (`factorEditorHtml(prefix, seed)`) rendered both for the
defaults block and for each per-concept override card. Each editor
surfaces the **full engine surface**:

- Cadence — `every N <unit>` OR `count N`
- Distribution family — radio (normal / skewnormal / beta / lognormal)
- Begin block — mean, sd, skew (skew used only for skewnormal)
- Bounds — min, max (shown only when family=beta)
- Drift — toggle that exposes End block (mean, sd, skew)
- Loss — mode dropdown (none / bernoulli / bursty / adherence_drift)
  with mode-specific fields rendered conditionally

`buildParams()` walks the defaults editor + each open override
accordion card via `readFactorEditor(root)` and constructs the v2
`params` dict the engine accepts (see `profile_v2_format.md` for the
full schema). Only fields the operator actually filled get emitted —
empty inputs inherit from defaults at the engine level.

---

## 17. Runs viewer architecture

Operator-facing flow: [`user_manual.md` §5 Runs viewer](user_manual.md#5-web-ui--runs-viewer).

`templates/runs.html` is a separate page that reads from cdr_6.

### Data flow

```
Browser
   │  GET /runs                                              → static HTML
   │  GET /api/cdr6/runs                                     → tunnel → cdr_6 /api/v1/runs (aggregate)
   │  Click row → GET /api/cdr6/runs/<id>/observations?limit=N
   │                                                         → tunnel → cdr_6 /api/v1/runs/<id>/observations
   │  Resolve concept names: GET /api/plan-definitions/<plandef_guid>
   │                                                         → plan.pdhc (already on the public path)
   └─► Render: stats card + per-concept time-series + sample-rows table
```

### Charts

One `Chart.js` line plot per unique concept in the loaded
observations. One series per `patient_guid`. Values parsed as floats;
non-numeric rows skipped silently. Concept names resolved to
`"name (unit)"` via the run's `plan_definition_guid` (best-effort,
falls back to short guids).

### cdr_6 read endpoint

`GET /api/v1/runs/<sim_run_id>/observations` (added in
`cdr_6.pdhc f25d440`):

- Default limit 200, hard max 5000.
- Optional filters: `?concept_guid=<g>`, `?patient_guid=<g>`.
- Ordered by `(patient_guid, concept_guid, effective_at)` for stable
  windowing.
- Same `@require_sim_service_key` auth as ingest/stats/runs.

The proxy in `sim.web` forwards **only** known query params
(`concept_guid`, `patient_guid`, `limit`) — keeps the proxy surface
deliberately small.

## 18. generate-from-plandef — Synthea-from-PlanDefinition

A third generation path (alongside §3's v2 engine and §15's Synthea import):
drive **MITRE Synthea** *from* a plan.pdhc PlanDefinition so a cohort's
observations are exactly that plan's concepts, on its cadence, over a known
time window — then land them via the existing Shape C / Shape B importers.

### Pipeline

```
PlanDefinition (PlanDefClient)
  → synthea_module.build_modules   one looping GMF module per concept
  → synthea_runner.run_synthea     local jar, time-pinned + seeded, tmp workdir
  → synthea_import.import_synthea   Shape C: patients → ips.pdhc roster
  → _push_synthea_observations      Shape B: plan's stream → CDR (attributed)
```

CLI: `sim generate-from-plandef --plandef <guid|title> --clinic <guid>
[--patients N] [--seed S] [--start YYYY-MM-DD] [--end YYYY-MM-DD]
[--age MIN-MAX] [--push-observations] [--target cdr_6]`.

### `synthea_module.py` — PlanDefinition → GMF

One module file **per concept** (Synthea runs each `*.json` in the `-d` dir as
an independent concurrent state machine, so lanes need not be interleaved).
Each module loops continuously: `Initial → Onset_Delay → Encounter →
Observation → EncounterEnd → Cadence_Delay → Encounter …`.

- **Cadence** from the parent activity's timing (`timing_frequency` /
  `timing_period` / `timing_period_unit` d|wk|mo): `gap_days =
  period_days / max(frequency, 1)`. A concept appearing under several
  activities takes the *repeat* cadence (repeat pass first, then the rest).
- **Value** from the transaction reference range (`range_min`/`range_max`,
  falling back to `expected_value ±20 %`, then a neutral band) → the
  Observation `range`; `concept_unit_name` → `unit`.
- **Coding** = `PLAN_CONCEPT_SYS` (`https://plan.pdhc.se/Concept`) + the
  concept GUID as `code`. Synthea preserves an arbitrary coding system
  verbatim on export (verified), so this round-trips to ~100 % coverage
  regardless of any LOINC/canonical binding.

### `synthea_runner.py` — the jar

Locates java (brew `openjdk@17` → PATH) + the jar (`SYNTHEA_JAR_PATH`), runs
in a `tempfile.mkdtemp()` workdir, returns the `out/fhir/` path. **Time-window
pinning:** `-r`/`-e` do *not* filter output (Synthea simulates a whole life);
the lever is `-e <end>` + `--exporter.years_of_history = round((end-start) /
365.25)`, which trims to `[end − N y, end]`. Continuous-loop modules guarantee
points inside that window; residual year-granular bleed is removed by the
mapper's exact date filter (below).

### `synthea_obs.py` mapper edits

- `ConceptCatalogue.lookup()` recognises the plan.pdhc concept systems
  (`PLAN_CONCEPT_SYS` / `urn:pdhc:concept` / the local CodeSystem URL) and
  returns `code` **as** the concept_guid directly.
- `pdhc_only=True` drops every canonical (stock-LOINC) coding, isolating the
  plan's own stream from Synthea's built-in disease-module observations.
- `plan_definition_guid` is stamped on each row; `window_start`/`window_end`
  give an exact inclusive effective-date filter.

### Run-level identity GUIDs

A cohort is generated *as if* under a contract, requested by an org, by a
user. These three are **constant for the whole simulation** and stamped on
every emitted row (cdr_6's ingest already persists them):

- `generate-from-plandef`: `--contract-guid`, `--requesting-org-guid`,
  `--requester-user-guid` (threaded through `_push_synthea_observations` →
  `build_cdr6_rows_for_patient`).
- `cohort-build`: the same keys in the params YAML
  (`contract_guid` / `requesting_org_guid` / `requester_user_guid`), read
  once in `cohort_builder.build_rows` and applied to every row — alongside the
  existing run-level `provider_org_guid` / `plan_definition_guid` /
  `care_plan_guid`.

Omitted → honest NULLs (the legacy `import-synthea` path leaves them None).

### Deferred (v1)

Quantity concepts only (categorical/valueset skipped — the mapper keeps
`valueQuantity`). Plan cadence *bounds* (count/duration) are recorded in the
module `remarks` but not enforced — the export window defines the period.
Local-only (needs Java 17 + the jar); no server deploy.

## 19. cdr-transfer — move a CDR's content into another CDR

`sim cdr-transfer --from <src> --to <dst> [--sim-run-id X] [--purge-source]
[--dry-run] [--batch-size 100]` copies (or, with `--purge-source`, moves) a
source CDR's complete content into a destination CDR. The motivating use:
generate a synthetic cohort into the cdr_6 sink, then promote it into cdr2–5.

Path: source `GET /api/v1/export` (cursor-paged, **unfiltered** — see below)
→ map each row to the flat ingest item (`_row_to_ingest_item`, dropping cdr_6
internals like `id`/`received_at`) → destination `POST /api/v1/ingest/batch`
in ≤100-row batches (cdr1–5 cap). Reads land in the target's `observation`
table via `IngestPipeline.process` — the same shape cdr2–5 already hold.

- **Complete + faithful:** the transfer reads cdr_6's new
  `GET /api/v1/export`, which — unlike the sim.web viewer endpoint — applies
  **no** spärr/consent/org filtering and pages by the integer PK, so the whole
  content moves, not a filtered sample.
- **Safe move:** `--purge-source` deletes the source run(s) **only** after a
  verified-complete copy (`http_err == 0 and rejected == 0 and
  accepted+duplicate == rows_read`). A partial/failed transfer leaves the
  source intact. `--dry-run` reports counts first.
- **Destination auth (important):** cdr1–5 validate `sim.pdhc` against their
  own `SIM_PDHC_SERVICE_KEY` env (`cdr.pdhc/.../api/auth.py`). To transfer
  into a production CDR, that CDR's `.env` must carry the same
  `SIM_PDHC_SERVICE_KEY` sim sends; otherwise the push returns 403
  `invalid service key` and (correctly) nothing is purged. cdr_6 already
  trusts sim's key.