# sim.pdhc — User manual

> **Front page:** [Sim overview](frontend.html) — a one-page, at-a-glance
> summary of what Sim does and its four workflows.

sim.pdhc generates synthetic longitudinal clinical data, attributes it
to real PlanDefinitions and organisations, and lands it in `cdr_6` (or
the older `cdr1`–`cdr5`). It has two operator-facing surfaces — a
**Web UI** at `http://127.0.0.1:9099` and a **CLI** at
`python -m sim …`. Both share the same engine and produce the same
output.

There are three operator workflows the rest of this manual covers:

1. **Cohort Builder** — pick a PlanDefinition, pick an organisation,
   set parameters, generate (and optionally push to cdr_6). The
   primary flow (§4 Web UI, §6 CLI, §8 Scenario A).
2. **Runs viewer** — browse what already landed in cdr_6 with charts
   and a sample-row table. Read-only (§5).
3. **Synthea integration** — `sim import-synthea` consumes
   Synthea-generated FHIR bundles. Two shapes that can be mixed:
   **Shape C** imports Patient identities into ips.pdhc as roster
   entries (§8 Scenario E); **Shape B** adds `--push-observations`
   and pushes the bundle's mapped Observations to cdr_6 (§8 Scenario F).

For architectural depth see [`technical.md`](technical.md); for the
full v2-profile reference see
[`profile_v2_format.md`](profile_v2_format.md); for the Cohort Builder
smoke history see [`../STEP_B_SMOKE.md`](../STEP_B_SMOKE.md).

---

## 1. Setting it up locally

```bash
cd ~/T7_sidewinder/sim.pdhc
cp .env.example .env       # then edit (see §2)
python3.14 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python -m pytest -q       # ~120 tests, all should pass
```

Python 3.14 is the supported interpreter — older versions are not
tested.

Both `python -m sim …` and `python -m sim.web` **auto-load** the
sim.pdhc `.env` on startup (see `sim/env_loader.py`). You don't need
`set -a && source .env && set +a` anymore. Explicit shell exports
still win (matching the behaviour of `source .env`), so you can
override single keys for one invocation without editing the file.

Confirm the setup with the bundled health check:

```bash
.venv/bin/python -m sim doctor
```

Output is a checklist: plan.pdhc reachable, ips.pdhc reachable + auth,
cdr_6 reachable through the SSH tunnel, Synthea jar present, etc.
Exit 0 if every check passes or is informational; exit 1 if any
check FAILs.

---

## 2. The `.env` file

Copy `.env.example` and fill in:

| Variable | Purpose | Required when |
|---|---|---|
| `IPS_PDHC_BASE_URL` | IPS server (clinics + patient roster) | always (default `https://ips.pdhc.se`) |
| `IPS_PDHC_API_KEY` | `Authorization: ApiKey <raw>` for IPS calls | Cohort Builder, list-clinics |
| `PLAN_PDHC_BASE_URL` | plan.pdhc resolver | always (default `https://plan.pdhc.se`) |
| `SIM_PDHC_SERVICE_KEY` | `X-Service-Key` for cdr_6 ingest + queries | pushing to cdr_6, Runs viewer |
| `SIM_TUNNEL_HOST` | SSH host for the cdr_6 tunnel | pushing/querying cdr_6 (default `miserver@192.168.1.154`) |
| `CDR_AUTH_TOKEN` | Bearer for cdr1–cdr5 bundle ingest | pushing to cdr1–cdr5 |
| `CDR_BASE_URL` | Default target if `--cdr` / `--target` omitted | optional |
| `ANTHROPIC_API_KEY` | `describe-to-profile` LLM call | only when authoring from prose |
| `TERMBANK_PDHC_BASE_URL` | Termbank lookup | always (default `http://127.0.0.1:9012`) |
| `SIM_SEED` | Default seed | optional (CLI `--seed` always wins) |
| `SIM_CDR_CONCURRENCY` | Push concurrency cap | optional (default 8) |
| `SIM_CDR_TIMEOUT` | HTTP timeout in seconds | optional (default 30) |

Generating an IPS API key (the only key sim can't generate for you) —
on miserver:

```bash
ssh miserver@192.168.1.154 bash << 'EOF'
set -e
RAW=$(python3 -c "import secrets; print(secrets.token_urlsafe(48))")
HASH=$(python3 -c "import hashlib,sys; print(hashlib.sha256(sys.argv[1].encode()).hexdigest())" "$RAW")
PREFIX="${RAW:0:8}"
PG_USER=$(docker exec ips-db-1 env | awk -F= '/^POSTGRES_USER=/{print $2}')
PG_DB=$(docker exec ips-db-1 env | awk -F= '/^POSTGRES_DB=/{print $2}')
docker exec ips-db-1 psql -U "$PG_USER" -d "$PG_DB" -c \
  "INSERT INTO api_keys (guid, key_hash, label, prefix, scopes, is_active, created_at)
   VALUES (gen_random_uuid(), '$HASH', 'sim.pdhc smoke <date>', '$PREFIX', '[]'::jsonb, true, now());"
echo "$RAW"
EOF
```

The last line is the plaintext key (shown once). Paste it as
`IPS_PDHC_API_KEY=…` in `.env`. Revoke later with:

```bash
docker exec ips-db-1 psql -U "$PG_USER" -d "$PG_DB" -c \
  "UPDATE api_keys SET revoked_at=now() WHERE label='sim.pdhc smoke <date>';"
```

Why ips.pdhc is the IPS authority for "this organisation's patients":
[`technical.md` §15 IPS integration](technical.md#15-ips-integration-cohort-builder).

**Never commit `.env`** — it is gitignored for a reason.

---

## 3. Two ways to use sim

| Path | When | Strength |
|------|------|----------|
| **Web UI** (`python -m sim.web`) | Interactive runs, exploration, demos | Click-driven; no YAML; Runs viewer with charts |
| **CLI** (`python -m sim …`) | Repeatable runs, scripts, large-scale (10k+) | Full engine surface via `--params` YAML; pipeable; no host process to keep alive |

Both produce identical rows for identical inputs. The Web UI's
"Parameters" form covers the common case; the CLI's `--params` YAML
covers the full engine surface (drift, distribution families, all loss
modes). See §7 for the params reference.

---

## 4. Web UI — Cohort Builder

```bash
set -a && source .env && set +a
.venv/bin/python -m sim.web        # → http://127.0.0.1:9099
```

Stop with `Ctrl+C`. The page binds 127.0.0.1; not reachable from
elsewhere on the network. Architectural reference:
[`technical.md` §16 Web UI architecture](technical.md#16-web-ui-architecture).

The Cohort Builder is a single page with four panels and a top nav
linking to the **Runs viewer**.

### Panel 1 — PlanDefinition

A searchable list of active PlanDefinitions on plan.pdhc. Type to
filter by title substring (case-insensitive). Click to pick. The
clear button (`×`) deselects. Picking populates the per-concept
override table in Panel 3.

### Panel 2 — Organisation

A list of active clinics from ips.pdhc. Pick one; the panel shows the
roster size (count of active patients linked via
`PatientClinicAssignment`). Empty rosters work but produce zero rows.

### Panel 3 — Parameters

Common knobs in the form:

- **Period start / end** — the time window observations are spread across.
- **Seed** — same seed + same plandef + same clinic + same params →
  byte-identical rows. Adding or reordering patients in the clinic
  does NOT shift other patients' streams (per-patient seed is
  `hash(seed, patient_guid)`).
- **Patients (max)** — cap the roster for smoke runs.
- **Cadence (days)** — one observation per N days, jittered.
- **Loss %** — bernoulli drop rate (0 = no loss).
- **Distribution mean / sd (default)** — applied to every concept
  unless overridden.

Open **Per-concept overrides** for a table of the picked
PlanDefinition's terms. Fill any cell (mean / sd / cadence) to
override the default for that concept; leave blank to inherit.

**Engine knobs in the form:** the per-concept editor and defaults
block both expose the full engine surface — distribution family
(normal / skewnormal / beta / lognormal), drift via begin → end, all
loss modes (bernoulli / bursty / adherence_drift), cadence as
`every N` or `count: N`. Beta bounds appear conditionally when family
= beta. Population-level shifts (age / sex) live in the separate
"Population factors" block — see §7.

### Panel 4 — Generate / Push

- **Generate** — fetches plandef + roster, builds rows in memory,
  returns counts. Nothing is sent. Good for sanity-checking row count
  before pushing.
- **Generate & push to cdr_6** — same plus an ingest call. sim.web
  auto-opens the SSH tunnel to cdr_6 the first time it's needed and
  holds it open for the process lifetime.

The stats card shows: `patients`, `terms`, `rows`, `run id`. The full
JSON response (with per-batch push results, when applicable) is shown
below.

---

## 5. Web UI — Runs viewer

Visit `http://127.0.0.1:9099/runs` (or click "Runs viewer" in the top
nav). Architectural reference:
[`technical.md` §17 Runs viewer architecture](technical.md#17-runs-viewer-architecture).

- **Runs table** — every run in cdr_6, most recent first. Columns:
  run_id, cohort name, patients, concepts, observations,
  last_received_at, author_org_guid (first 8 chars).
- **Click a row** to load that run's observations. Three things appear:
  - Stats card (patients / concepts / rows loaded / period).
  - **Per-concept time series**: one Chart.js line plot per concept,
    one series per patient. Concept names resolved via plan.pdhc.
  - Sample rows table (first 500 visible; charts use the full set).
- **Limit selector** (200 / 500 / 1000 / 5000) and **Reload** widen
  the window without re-picking the run.
- **Refresh** at the top reloads the runs list.

Read-only. Purge a run from the CLI (§8).

### Health tab

The same nav exposes a **Health** page that mirrors `sim doctor`. It
shows the same checklist (✓/?/✗ rows) without leaving the browser, and
exposes `GET /api/health/check` for scripted polling — returns the
results as JSON, HTTP 200 when every check passes and HTTP 503 when
any check fails (INFO-only stays 200).

---

## 6. CLI commands at a glance

| Command | What it does |
|---|---|
| `doctor` | Environment + dependency health check (`✓`/`?`/`✗` checklist; exit 1 on FAIL) |
| `quickstart` | End-to-end smoke: doctor → `import-synthea` → `cohort-build` (push to cdr_6 via tunnel) → print Runs viewer URL |
| `mint-ips-key` | SSH to miserver, mint a fresh ips.pdhc API key, write it to `.env` (and live env). Pair with `doctor` to fix `IPS_PDHC_API_KEY` failures in one shot. |
| `cohort-build` | Plandef + IPS roster + params YAML → rows + (optional) push to cdr_6 |
| `list-clinics` | Print clinic list from ips.pdhc |
| `import-synthea` | Import Synthea Patients into ips.pdhc (Shape C); add `--push-observations` to also push that bundle's Observations to cdr_6 (Shape B) |
| `generate-from-plandef` | Drive Synthea *from* a PlanDefinition: reproducible cohort whose observations are exactly the plan's concepts, on its cadence, over a known window → Shape C + optional plandef-attributed Shape B |
| `list-plan-definitions` | Print PlanDefinitions on plan.pdhc |
| `show-plan-definition` | Print one PlanDefinition's leaf concepts |
| `list-concepts` | Browse plan.pdhc Concepts |
| `describe-to-profile` | Prose → v2 YAML profile (Claude-backed) |
| `generate` | v0.1/v0.2 path: build bundles + CSVs to a local dir; no network |
| `push` | Push an existing manifest's bundles to a CDR |
| `run` | `generate` + `push` in one shot |
| `purge` | Delete every row tagged with a given `sim_run_id` |

Detailed flag help: `python -m sim <cmd> -h`.

### `sim quickstart` — first-cohort smoke

The one-liner for a fresh operator who has a Synthea bundle directory
and wants live data in the Runs viewer. It chains the same handlers
you'd run individually:

```bash
python -m sim quickstart \
  --source ./synthea_output \
  --clinic <clinic-guid> \
  --plandef <plandef-guid> \
  --params profiles/cohort_build_smoke.yaml \
  --out /tmp/quickstart \
  --max 20 --patients-max 20
```

Stages (each prints `[N/4] …`):

1. `doctor` precheck. Exit on FAIL; bypass with `--skip-doctor`.
2. `import-synthea --source X --clinic Y` (loads Synthea Patients into
   ips.pdhc as roster entries).
3. `cohort-build --plandef A --clinic Y --params P --out OUT --push
   --target cdr_6 --tunnel`.
4. Prints `http://127.0.0.1:5099/runs` (override with `--viewer-port`).

If the viewer isn't already running, start it in another shell with
`python -m sim.web`. The Runs viewer's **Health** tab mirrors
`sim doctor` so you can re-verify environment health from the browser.

### `sim generate-from-plandef` — synthetic data *for a specific plan*, over a known window

`import-synthea` imports whatever a generic Synthea run produced.
`generate-from-plandef` goes the other way: you hand it a plan.pdhc
**PlanDefinition** and it drives Synthea to produce a cohort whose
observations are exactly that plan's concepts, on the plan's cadence,
over a time window you choose — reproducibly.

```bash
python -m sim generate-from-plandef \
  --plandef <guid|title> \        # the PlanDefinition on plan.pdhc
  --clinic  <clinic-guid> \       # ips.pdhc destination roster (Shape C)
  --patients 20 \
  --seed 42 \                     # omit to get a random seed (printed)
  --start 2020-01-01 --end 2024-12-31 \
  --push-observations --target cdr_6   # optional Shape B, plandef-attributed
```

What it does, in stages:

1. **Fetch** the PlanDefinition and expand its `activities[*].transactions[*]`.
2. **Generate** one Synthea GMF module per concept — each a continuous
   `Encounter → Observation → Delay` loop. The `Delay` comes from the
   activity's cadence (`timing_frequency`/`period`/`period_unit`); the
   Observation value is drawn from the concept's reference range
   (`range_min`/`range_max`, falling back to `expected_value ±20%`); the
   coding is `https://plan.pdhc.se/Concept` + the concept GUID.
3. **Run Synthea** (local jar, Java 17) pinned to `[--start, --end]` and
   `--seed`, in a throwaway temp dir.
4. **Shape C** — import the generated Patients into the clinic roster.
5. **Shape B** (only with `--push-observations`) — push the plan's own
   observation stream to the CDR, **stamped with `plan_definition_guid`**,
   filtered to exactly the window, and isolated to the plan's concepts
   (Synthea's stock lab observations are dropped).

Notes:

- **Reproducible:** the same `--plandef`, `--patients`, `--seed`, and
  window produce the same cohort every time. The seed is printed so you
  can pin a random run afterwards.
- **Known window:** defaults to the last 5 years ending today; override
  with `--start`/`--end` (`YYYY-MM-DD`). Because Synthea trims history at
  year granularity, the mapper additionally filters to the exact dates.
- **Prereqs:** Java 17 (`brew install openjdk@17`) and the jar at
  `~/T7_sidewinder/synthea/synthea.jar` (override via `SYNTHEA_JAR_PATH`).
  `sim doctor` checks both.
- **v1 limits:** quantity concepts only (categorical/valueset deferred);
  the plan's cadence *bounds* (e.g. "12 times") are not enforced — the
  window defines the period.

### `sim mint-ips-key` — fix `IPS_PDHC_API_KEY` failures in one shot

`sim doctor` flags `IPS_PDHC_API_KEY` as FAIL when the value is unset
or rejected by ips.pdhc. To mint a fresh key, hash + insert it via
`docker exec ips-db-1 psql`, and write it back into `.env`:

```bash
python -m sim mint-ips-key
```

Flags:

- `--label "<text>"` — label written to `api_keys.label`. Defaults to
  `sim.pdhc <UTC-date>`.
- `--host miserver@1.2.3.4` — override the SSH target. Defaults to
  `$SIM_TUNNEL_HOST`, falling back to `miserver@192.168.1.154`.
- `--no-write` — print the key but do NOT touch `.env` (useful for
  ad-hoc handover; copy + paste yourself).
- `--json` — machine-readable output (key + label + env_path).

The plaintext key is **never written to disk by the bash block on
miserver** — it's only echoed on stdout, captured by ssh, then
either printed to your terminal or merged into your local `.env`. The
script also injects the new value into `os.environ` so a subsequent
`sim doctor` in the same shell picks it up without re-sourcing.

The minted key is active immediately (the SQL `INSERT` sets
`is_active = true`). To revoke a leaked key, flip `is_active = false`
on the matching row directly on the ips DB.

The **Cohort Builder** flow (Web UI) is the CLI's `cohort-build` under
the hood. The older `generate` / `push` / `run` flow is still
supported for v0.1/v0.2 profiles (synthetic patients, hand-authored
cohorts).

---

## 7. Params YAML reference — the full engine surface

A params YAML drives `cohort-build`. Worked example at
[`../profiles/cohort_build_smoke.yaml`](../profiles/cohort_build_smoke.yaml).

```yaml
period: {start: "2026-01-01", end: "2026-04-01"}
seed: 42

# Optional cohort metadata — copied verbatim to cdr_6 so it shows up
# in the Runs viewer.
# author_org_guid: <guid>          # defaults to --clinic if omitted
# care_plan_guid:  <guid>          # optional

defaults:
  cadence: {every: "30 days"}      # OR {count: 12} for N evenly-spaced
  distribution:
    family: normal                  # normal | skewnormal | beta | lognormal
    begin: {mean: 100.0, sd: 10.0}
    # end:   {mean: 110.0, sd: 8.0}    # add drift over the period
  # loss is optional — omit for a complete stream
  # loss: {mode: bernoulli, pct: 0.05}
  # loss: {mode: bursty, on_days: 14, off_days: 21}
  # loss: {mode: adherence_drift, alpha: 2.0, beta: 8.0}

# Optional per-concept overrides (keyed by concept_guid).
# Each override merges onto defaults — keys not set inherit.
# overrides:
#   "<concept-guid>":
#     cadence: {every: "7 days"}
#     distribution:
#       family: skewnormal
#       begin: {mean: 50.0, sd: 5.0, skew: 0.5}
#       end:   {mean: 60.0, sd: 4.0, skew: 0.2}
#     loss: {mode: bernoulli, pct: 0.10}

# Optional population-level shifts based on IPS demographics
# (birth_date, gender). Applied additively after the RNG draw —
# determinism is preserved.
# population:
#   age:
#     apply_to: ["<concept-guid-1>", "<concept-guid-2>"]
#     ref_age:        50      # patients younger than this get 0 shift
#     shift_per_year: 0.5     # additive units / year above ref_age
#   sex:
#     apply_to: ["<concept-guid-1>"]
#     shifts:
#       male:   20            # additive shift for gender='male'
#       female: 0
```

### Distribution families

| Family | Required fields | Notes |
|---|---|---|
| `normal` | `mean`, `sd` | Default; lightest weight |
| `skewnormal` | `mean`, `sd`, `skew` (γ1, in `[-0.995, 0.995]`) | Visible Fisher–Pearson γ1; δ solved via Brent |
| `beta` | `mean`, `sd`, `bounds: {min, max}` | Reparameterised; bounds required |
| `lognormal` | `mean`, `sd` (of the output distribution, not the underlying log) | `mean > 0` enforced |

### Drift

Set both `begin:` and `end:` to interpolate linearly across the
period. Omit `end:` for a stationary distribution. Drift works on
`mean`, `sd`, and `skew` (the latter only for `skewnormal`). The
interpolation uses each observation's timestamp position within the
period — see [`179_validation/distribution_quality.md`](179_validation/distribution_quality.md)
for the #180 deflation-isn't-real story.

### Loss modes

| Mode | Fields | Behaviour |
|---|---|---|
| `bernoulli` | `pct` ∈ `[0, 1]` | Independent coin flip per row |
| `bursty` | `on_days`, `off_days`, `pct` (currently ignored; #181) | Alternating "on" and "off" dwells |
| `adherence_drift` | `alpha`, `beta` | Per-patient drop rate drawn from `Beta(α, β)` |

Loss only changes which rows are emitted — it never shifts the value
of a row that survives (Phase 2.b determinism contract).

### Cadence

- `{every: "N <unit>"}` — N>0, unit one of `minute / hour / day / week
  / month / year` (s suffix optional). Adds ±20% jitter unless
  `jitter_pct: 0`.
- `{count: N}` — N evenly-spaced timestamps across the period (no
  jitter).

### Population factors (optional)

The `population` block adds per-patient shifts based on real
demographics from ips.pdhc — `birth_date` and `gender`. Two factors
today:

| Factor | Field | What it does |
|---|---|---|
| `age` | `apply_to: [<concept-guid>, ...]` | List of concepts the factor affects |
| | `ref_age: 50` | Reference age; patients younger get 0 shift (clamped) |
| | `shift_per_year: 0.5` | Additive shift per year above `ref_age` |
| `sex` | `apply_to: [<concept-guid>, ...]` | List of concepts the factor affects |
| | `shifts: {male: 20, female: 0, ...}` | Additive shift per gender value |

Both factors are independent and additive: total shift is the sum of
each factor's contribution. **Patients missing a demographic** (no
`birth_date` for age, no `gender` for sex) silently receive no shift
from that factor — never an error.

**The shift is applied AFTER the value draw**, not inside the RNG
stream. This means: same `(plandef, params, patients)` with population
turned ON vs OFF differs by exactly the constant shift per row — the
RNG draws are byte-identical either way. See
[`technical.md` §15 IPS integration](technical.md#15-ips-integration-cohort-builder)
for the determinism proof.

### Web UI vs CLI parity

Everything above is in **both** the Web UI's Parameters panel and the
CLI's `--params` YAML. The Web UI's "Population factors" details block
collapses by default; check the concepts it applies to and set the
constants. The CLI takes the same YAML the Web UI emits.

---

## 8. Common scenarios

### A — Cohort Builder smoke via CLI (current default)

```bash
set -a && source .env && set +a
.venv/bin/python -m sim list-clinics                # find a clinic
.venv/bin/python -m sim cohort-build \
    --plandef <plandef-guid> \
    --clinic  <clinic-guid> \
    --params  profiles/cohort_build_smoke.yaml \
    --out     /tmp/smoke \
    --patients-max 5
# inspect /tmp/smoke/{rows.ndjson,manifest.json} — no push yet
.venv/bin/python -m sim cohort-build \
    --plandef <plandef-guid> --clinic <clinic-guid> \
    --params profiles/cohort_build_smoke.yaml \
    --out /tmp/smoke_push --patients-max 5 \
    --push --target cdr_6
```

Then visit `http://127.0.0.1:9099/runs` to see it in the viewer.

### B — Describe a cohort in prose, save, run via cdr1

```bash
.venv/bin/python -m sim describe-to-profile \
    --plan-definition-title Test_medituner \
    --prose "200 asthma patients, monthly FEV1 for a year, ~15% missed visits" \
    --out profiles/asthma_monthly_v2.yaml

# review profiles/asthma_monthly_v2.yaml, edit, commit

.venv/bin/python -m sim run \
    --profile profiles/asthma_monthly_v2.yaml \
    --count 200 --seed 1234 --target cdr1
```

### C — 10k cohort over 3 years to cdr_6

```bash
.venv/bin/python -m sim run \
    --profile profiles/scale_10k_v2.yaml \
    --count 10000 --seed 7 --days 1095 \
    --target cdr_6 --tunnel
```

Allow ~5–10 minutes. The 2026-05-29 benchmark was 364 s end-to-end,
0 errors, ~1 GB cdr_6 DB growth.

### D — Roll back a bad run

```bash
.venv/bin/python -m sim purge \
    --run-id sim-20260603T123755Z-0000002a \
    --target cdr_6 --tunnel
```

Get the run-id from `manifest.json` or from the Runs viewer.

### E — Populate Test Clinic with Synthea-generated patients (Shape C)

Use Synthea to mint realistic Patient identities, import them into
ips.pdhc as roster entries, then run sim against them as you would
any other clinic.

Synthea is installed locally at `~/T7_sidewinder/synthea/synthea.jar`
(precompiled jar) with `openjdk@17` via brew. Generate a batch:

```bash
cd ~/T7_sidewinder/synthea
/opt/homebrew/opt/openjdk@17/bin/java -jar synthea.jar -p 20 \
    --exporter.fhir.export=true \
    --exporter.hospital.fhir.export=false \
    --exporter.practitioner.fhir.export=false
# → ~20 *.json bundles in output/fhir/
```

Import their Patient resources into a clinic:

```bash
.venv/bin/python -m sim import-synthea \
    --source ~/T7_sidewinder/synthea/output/fhir \
    --clinic <clinic-guid> \
    --max 20
```

You'll see one row per patient with `created` / `failed` and either
the new IPS guid or the error. The Synthea identifier (SSN if present,
else Synthea's UUID) lands as the IPS `identifier_value` so you can
trace a row back to its source bundle. Names follow Synthea's
suffix-numbered style (e.g., `Abbott774 Johnnie679`).

Run sim against them with the existing Cohort Builder flow — they're
just IPS patients now:

```bash
.venv/bin/python -m sim cohort-build \
    --plandef <plandef-guid> \
    --clinic  <clinic-guid> \
    --params  profiles/cohort_build_smoke.yaml \
    --out     /tmp/synthea_run \
    --patients-max 20 \
    --push --target cdr_6
```

**What this gives you that the mock-data form doesn't:** realistic age
distributions, real disease prevalence in the source bundles (which
analyse.pdhc could later consume separately), and demographic patterns
the engine's age/sex shifts (Phase 1 population factors) can exercise
non-trivially.

**What Scenario E does NOT do:** import Synthea's clinical resources
(Observations, Conditions, …) into cdr_6. That's **Scenario F (Shape
B)** below. Operators who want the full Synthea history landed pick
both together.

### F — Push Synthea Observations to cdr_6 (Shape B)

Adds the LOINC/SNOMED → plan.pdhc concept mapping layer. Use the same
`import-synthea` command with `--push-observations`:

```bash
.venv/bin/python -m sim import-synthea \
    --source ~/T7_sidewinder/synthea/output/fhir \
    --clinic <clinic-guid> \
    --push-observations \
    --target cdr_6
```

What happens per bundle:
1. Patient lands in ips.pdhc (Shape C — same as Scenario E).
2. The bundle's Observations are walked. For each one:
   - `code.coding[0].system` is normalised (`http://loinc.org` →
     `loinc`, `http://snomed.info/sct` → `snomed`, etc.).
   - `(library_name, canonical_refnumber)` is looked up in a cached
     plan.pdhc concept catalogue.
   - If a concept_guid is found, a cdr_6 flat row is built. If
     not, the observation is silently dropped — sim never invents
     concept_guids.
   - Only `valueQuantity` observations are kept; non-quantity codings
     (CodeableConcept, string, multi-component) are skipped.
3. All mapped rows are pushed to cdr_6 in one batch with run_id
   `synthea-<UTC-stamp>-<rand>` (distinct namespace from the
   `sim-...` cohort runs so you can tell them apart in the Runs
   viewer).

Hit-rate observation (2026-06-03 smoke against one bundle):

```
extracted from bundle:    411
kept quantity-typed:      379
mapped to plan.pdhc:      126   (of 15 distinct concepts)
dropped unmapped:         253
dropped non-quantity:      32
```

The 31% mapping rate is the natural ceiling of what plan.pdhc covers
today (71 concepts) against Synthea's breadth (it emits 600+ distinct
LOINC codes across its modules). To raise it, add more concepts to
plan.pdhc — sim picks them up automatically on the next run.

Characterisation on every Shape B row:
- `plan_definition_guid`, `activity_guid`, `transaction_guid`,
  `care_plan_guid` are **all NULL**. Synthea is not plandef-driven, so
  attributing rows to a plandef would be a lie.
- `author_org_guid` = the clinic guid (same as Shape C).
- `sim_run_id` = `synthea-<stamp>-<rand>`.
- `patient_guid` = the IPS guid assigned during the Shape C step of
  the same import.

What this does NOT do: it does not import Synthea Conditions,
MedicationRequests, Encounters, or other non-Observation resources.
Those are out of scope for cdr_6 (which only stores Observations) and
land elsewhere if needed.

---

## 9. Browsing plan.pdhc from the CLI

```bash
.venv/bin/python -m sim list-plan-definitions --search medituner
.venv/bin/python -m sim show-plan-definition --title Test_medituner
.venv/bin/python -m sim list-concepts --search hba1c
```

`show-plan-definition` uses a local cache (`~/.sim/cache/plandef/`);
pass `--no-cache` to force a refresh.

---

## 10. Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| `Missing Authorization header` on `list-clinics` | `IPS_PDHC_API_KEY` not in env | Generate a key (§2) and `source .env` before invoking |
| `Invalid or expired API key` on IPS calls | Key revoked or rotated | Generate a new one |
| `Clinic not found` on a clinic in `list-clinics` | The patients-by-clinic endpoint isn't deployed | Check `ips.pdhc` is at `c399add` or later on miserver |
| Cohort Builder generates 0 rows | Roster is empty OR plandef has no transactions with `concept_guid` | Check Panel 2 roster size; `show-plan-definition` on the plandef |
| Roster shows N patients but `cohort-build` says 0 | `PatientClinicAssignment` rows missing despite FHIR `managingOrganization` | Check `ips.pdhc` is at `ce0f928` or later; admin form writes both since then |
| cdr_6 push 4xx `invalid service key` | Stale `SIM_PDHC_SERVICE_KEY` | Update `.env` to match `/Users/miserver/.cdr_6/service_key.txt` on miserver |
| `Connection refused` on cdr_6 push without `--tunnel` | cdr_6 is loopback-only | Add `--tunnel`, or use the web UI (auto-tunnel) |
| Tunnel won't open (`Permission denied`) | SSH key not authorised on miserver | Check `~/.ssh/config`; verify key is in `miserver:.ssh/authorized_keys` |
| `unknown cadence unit: 'foo'` | Profile typo | Units: `minute / hour / day / week / month / year` |
| Runs viewer "0 runs returned" | sim.web tunnel didn't open OR no runs yet | Check the sim.web log; do a push first |
| `import-synthea` fails: "Source not found" | `--source` path doesn't exist or is empty | Confirm Synthea ran; default output is `output/fhir/*.json` from the directory you invoked java in |
| `import-synthea --push-observations` reports very low `mapped` count | Plan.pdhc has few concepts covering Synthea's emitted LOINCs | Expected — plan.pdhc has ~70-100 concepts, Synthea emits 600+. Add concepts to plan.pdhc to raise the rate; sim picks them up next run |
| `import-synthea --push-observations` `extracted=0` | Patient ID mismatch between bundle's Patient.id and subject refs | Synthea normally emits both `Patient/<id>` and `urn:uuid:<id>` styles — both accepted. If neither matches, something modified the bundle |
| Concept catalogue load times out | plan.pdhc unreachable | `curl https://plan.pdhc.se/api/v1/concepts?per_page=1` — should return 200. If not, plan.pdhc is down; retry later |
| `address already in use :19006` when starting `sim.web` | Stale tunnel from a previous run | `lsof -ti :19006 \| xargs kill`. The auto-tunnel didn't clean up — common after a sim.web crash |
| services.html healthcheck shows green when cdr_6 is broken | services.html uses `mode: 'no-cors'` and resolves opaque responses as green | Don't trust it; query `/api/v1/runs` or tail the run log |
| `anthropic SDK is required for describe-to-profile` | venv missing the dep | `.venv/bin/pip install anthropic` |

---

## 11. What sim won't do

- **No PHI.** Every Patient sim *synthesises* (the v0.1/v0.2 path)
  carries `SYNTHETIC` identifier + `meta.tag = SYNTHETIC`. The Cohort
  Builder path uses real IPS patient GUIDs but produces synthetic
  *observations* against them — IPS provides identities, sim provides
  values. Shape C imports of Synthea Patients are also synthetic by
  definition (Synthea's intent).
- **No unseeded runs.** Wall-clock seeding is deliberately
  unsupported — breaks reproducibility.
- **No unregistered targets.** `sim/config.py:TARGETS` is the source
  of truth for what's pushable.
- **No invented concept_guids.** Shape B drops observations whose
  `(code_system, code)` doesn't match a concept already curated in
  plan.pdhc. There is no fuzzy match, no fallback, no autogeneration.
  Silent drops are the right behaviour: the alternative would seed
  cdr_6 with rows whose plan.pdhc concept_guid doesn't actually exist.
- **No Shape B attribution to a PlanDefinition.** Synthea Observations
  arrive with NULL `plan_definition_guid` / `activity_guid` /
  `transaction_guid` — they weren't produced by a plandef execution,
  so attributing them to one would be a lie. Downstream consumers
  filtering on plan_definition_guid will correctly not see Synthea
  rows; consumers filtering by `sim_run_id` prefix (`synthea-…`) can
  pick them out explicitly.
- **No auto-tunnel teardown mid-process for the Web UI viewer.** The
  tunnel opens on first viewer hit and closes at process exit. Stop
  `sim.web` when you're done.

---

## 12. Where to look next

- [`profile_v2_format.md`](profile_v2_format.md) — full v2 schema
  reference
- [`describe_to_profile.md`](describe_to_profile.md) — prose tool
  deep dive
- [`technical.md`](technical.md) — architecture, determinism, module
  layout. Of particular interest:
  - [`§15 IPS integration`](technical.md#15-ips-integration-cohort-builder)
    — Cohort Builder dependency chain
  - [`§15 Population factors (Phase 1)`](technical.md#population-factors-phase-1)
    — age + sex shift mechanics
  - [`§15 Synthea hookup (Shape C)`](technical.md#synthea-hookup-shape-c)
    — Synthea Patient import internals
  - [`§15 Synthea hookup (Shape B)`](technical.md#synthea-hookup-shape-b--observations-into-cdr_6)
    — Synthea Observation mapping internals
  - [`§16 Web UI architecture`](technical.md#16-web-ui-architecture)
    — sim.web routes + tunnel management
  - [`§17 Runs viewer architecture`](technical.md#17-runs-viewer-architecture)
    — viewer data flow + charts
- [`179_validation/distribution_quality.md`](179_validation/distribution_quality.md)
  — distribution-family validation evidence
- [`../STEP_B_SMOKE.md`](../STEP_B_SMOKE.md) — Cohort Builder smoke
  history + re-open recipe
- `cdr_6.pdhc/docs/dashboard_handoff.md` — what a downstream
  dashboard needs to consume sim runs
- `progress.md` — current development state, change log
- `REFINEMENT_PLAN.md` — long-form roadmap (#178 + #179 context)

### `sim cdr-transfer` — promote a cohort from cdr_6 into another CDR

Generate into the cdr_6 sink, then move the result into a real CDR:

```bash
sim cdr-transfer --from cdr_6 --to cdr2 --sim-run-id <run> --dry-run   # preview
sim cdr-transfer --from cdr_6 --to cdr2 --sim-run-id <run>             # copy
sim cdr-transfer --from cdr_6 --to cdr2 --sim-run-id <run> --purge-source  # move
```

Omit `--sim-run-id` to transfer **all** of the source. `--purge-source`
removes the run from the source only after a verified-complete copy (no data
loss on a partial transfer). The destination CDR's `.env` must carry the
`SIM_PDHC_SERVICE_KEY` sim uses, or the push is rejected with `403 invalid
service key`.