All documentation Download (Markdown) Technical

Sim · sim.pdhc

Technical manual

sim.pdhc — Technical reference

A Python tool with two operator surfaces (CLI + Web UI at
127.0.0.1:9099) that generates synthetic Swedish-cohort FHIR R5 data
and pushes it to one or more cdr.pdhc instances. The Web UI is a
single-process Flask app — not a long-running prod service, but a
durable operator-facing surface that includes a Runs viewer reading
back from cdr_6. No DB of its own, no SSO, no Docker. Three long-form
references already exist — this document is the architectural overview
that ties them together:

1. What sim does and doesn't do

sim.pdhc is the producer in the longitudinal-data pipeline:

sim.pdhc   →   cdr_6.pdhc   →   analyse.pdhc (future)
 CREATE         STORE             READ + analyse
 cohort data    flat-row DB       to be built later

It generates longitudinal time series for synthetic cohorts based on
plan.pdhc PlanDefinitions + their leaf Terms (concepts), writes the
rows to cdr_6.pdhc, and exits. cdr_6 is the durable store;
analyse.pdhc (planned, not yet built) will be the primary consumer.

Does:
- Generates FHIR R5 transaction Bundles of Patient, Observation,
Condition, MedicationStatement, Encounter.
- Writes per-run output to results/<UTC-ts>_<label>/ with
manifest.json, bundles/, observations.csv, patients.csv, and
(for cdr_6 targets) cdr_6_rows.ndjson.
- Pushes those bundles to cdr1–cdr5 (FHIR bundle ingest) or cdr_6
(flat-row ingest with characterisation columns).
- Purges a previous run from a CDR by sim_run_id tag.

Doesn't:
- Hold any PHI — every Patient carries SYNTHETIC identifier system
and meta.tag = SYNTHETIC. Real-data dashboards filter this tag out
by default.
- Run as a service. start.sh is a venv-activating wrapper, not a
process supervisor.
- Operate without a seed. --seed is required on every run; same
(profile, count, seed) ⇒ byte-identical NDJSON output.

2. Package layout

sim/
├── __main__.py            # `python -m sim` entry point
├── cli.py                 # argparse + dispatch
├── config.py              # SimConfig, TARGETS registry, concept canonicals
├── patient.py             # PatientFactory (v0.1 hardcoded path)
├── variables/             # one engine per variable kind (v0.1 path)
│   ├── cgm.py
│   ├── hba1c.py
│   ├── vitals.py
│   ├── labs.py
│   ├── encounter.py
│   └── condition_med.py
├── fhir.py                # FhirBuilder + Bundle chunking
├── client.py              # CdrClient — FHIR-bundle + cdr_6 flat-row push
├── plandef_client.py      # plan.pdhc PlanDefinition resolver
├── ips_client.py          # ips.pdhc — clinics list + patients-by-clinic
├── profile_schema.py      # v2 profile JSON-Schema + is_v2_profile()
├── v2_engine.py           # plandef-driven generator (Phase 1–2 of #178)
├── cohort_builder.py      # plandef + IPS roster → cdr_6 flat rows (Cohort Builder engine)
├── describe_to_profile.py # Anthropic tool-use → v2 YAML emission
├── tunnel.py              # SSH port-forward for loopback-only targets
├── run.py                 # generate → write outputs orchestration
├── csv_cohort.py          # v1 CSV-cohort engine (kept for callers; UI flow gone)
└── web/                   # Flask app — Cohort Builder + Runs viewer
    ├── app.py             #   routes, proxies, tunnel manager
    └── templates/
        ├── cohort.html    #   Cohort Builder page (4 panels + factor editors)
        └── runs.html      #   Runs viewer (cdr_6 list + sample + charts)

3. Generation paths

Three paths, picked by CLI subcommand:

Path Entry Patients Engine Concepts
v0.1 sim generate/run with no plan_definition in profile Synthesised patient.PatientFactory + sim/variables/* Hardcoded set
v2 sim generate/run with v2 profile Synthesised v2_engine.run_v2 Resolved from plan.pdhc PlanDefinition
Cohort Builder sim cohort-build, Web UI / Real patients from ips.pdhc cohort_builder.build_cohort_rows (reuses v2_engine helpers) Resolved from plan.pdhc

profile_schema.is_v2_profile() discriminates v0.1 from v2:

return bool(profile.get("plan_definition")
            or profile.get("concepts_override"))

The Cohort Builder is the production path for "simulate against this
real organisation's roster"; the v2 run path is the production path
for "simulate a synthetic cohort against this plandef". v0.1 is kept
for legacy tests and cohort_a.yaml examples. Schema lives in
sim/profile_schema.py:PROFILE_V2_SCHEMA; new fields are added there
first, then taught to the engine.

4. Determinism contract

Same (profile, count, seed) ⇒ byte-identical
manifest.json+bundles/*.json+observations.csv. Mechanism:

This contract is load-bearing for diffing runs against each other.
Anything that breaks it (e.g., changing the RNG seeding scheme) is a
breaking change.

5. CDR targets

sim/config.py:TARGETS is the canonical registry:

"cdr1":  TargetSpec("cdr1",  "https://cdr1.pdhc.se", 9046, 19_001),
"cdr2":  TargetSpec("cdr2",  "https://cdr2.pdhc.se", 9146, 19_002),
"cdr3":  TargetSpec("cdr3",  "https://cdr3.pdhc.se", 9246, 19_003),
"cdr4":  TargetSpec("cdr4",  "https://cdr4.pdhc.se", 9346, 19_004),
"cdr5":  TargetSpec("cdr5",  "https://cdr5.pdhc.se", 9446, 19_005),
"cdr_6": TargetSpec("cdr_6", None,                   9055, 19_006),  # loopback-only

The push command picks a target via --target <name> (preferred) or
--cdr <url> (explicit override). When --tunnel is set the tunnel
is opened atomically with tunnel_for_target and closed on exit.

6. Plan.pdhc integration

sim/plandef_client.py resolves a PlanDefinition reference into its
leaf concept GUIDs.

The concepts_override: block bypasses plan.pdhc entirely — useful
for tests, dogfood profiles, and the dist-quality / scale soak runs.

7. CDR client

sim/client.py:CdrClient ships two unrelated push code paths:

Method Target Payload Endpoint
push_bundles_concurrent cdr1–cdr5 FHIR transaction Bundles, batched POST /api/v1/bundles
push_cdr_6_flat cdr_6 Flat-row NDJSON batches with characterisation columns POST /api/v1/ingest/batch
purge_run cdr1–cdr5 run-id tag DELETE /api/v1/runs/<run_id> (bundle ingest)
purge_run_cdr_6 cdr_6 run-id DELETE /api/v1/runs/<run_id> (flat-row)

Concurrency cap is SIM_CDR_CONCURRENCY (default 8). HTTP timeout is
SIM_CDR_TIMEOUT (default 30s).

Flat-row characterisation columns (cdr_6 only) — author_org_guid,
plan_definition_guid, care_plan_guid, activity_guid,
transaction_guid — are author-set on the profile/term blocks and
emitted as separate columns rather than embedded in FHIR. See
profile_v2_format.md for the full mapping.

8. describe-to-profile (Anthropic SDK)

sim/describe_to_profile.py is author-time only. No LLM in the run
path — once the YAML is committed, generation is fully deterministic.

Key environment variable: ANTHROPIC_API_KEY. Failure to install the
anthropic SDK surfaces as a CLI-friendly error.

9. SSH tunnel for loopback-only targets

sim/tunnel.py opens an SSH master tunnel on a deterministic Unix
socket under /tmp/, forwards local_tunnel_port → remote_app_port,
registers an atexit cleanup, and yields the local URL for the
duration of the with block.

10. Concept canonical registry

sim/config.py:CANONICAL and coding_for(name) build the FHIR coding
arrays. Source of truth:
concepts/diabetes_v1.guids.json (generated by concepts/load_to_plan.py).

11. Tests

tests/ — pytest, 127 cases as of 2026-06-03 (was 74 pre-Step-B). Run with:

.venv/bin/python -m pytest -x -q

Test files map cleanly to modules: test_v2_engine.py,
test_profile_schema.py, test_plandef_client.py,
test_describe_to_profile.py, test_cli_describe.py,
test_client_cdr_6.py, and the v0.1 legacy tests.

12. Validation evidence (#179)

Item Result Doc
Prose dogfood (8 inputs, EN+SV) 8/8 pass post-schema-fix docs/179_validation/prose_dogfood.md
Multi-concept end-to-end pass covered in 179_validation
Distribution quality (4 families × 1000) all 4 within ≤1 SE of corrected targets docs/179_validation/distribution_quality.md
Scale + duration (10k × 3yr × 5 concepts) 1,157,528 rows / 0 errors / 364s docs/179_validation/scale_duration.md
Dashboard dogfood not yet run — #179.5 —
Operational soak (≥7 days) not yet run — #179.6 —

13. What lives where on disk

Path Purpose Persistence
profiles/*.yaml Author-committed v2 profiles git
concepts/diabetes_v1.guids.json Generated GUID manifest git
results/<UTC>_<label>/ Per-run output gitignored (Rule 11)
scripts/*.py One-off investigation scripts (e.g. 180_repro.py) git
.env Local secrets (CDR_AUTH_TOKEN, ANTHROPIC_API_KEY, SIM_PDHC_SERVICE_KEY, IPS_PDHC_API_KEY) gitignored
.env.example Schema for the operator git

14. Out of scope

15. IPS integration (Cohort Builder)

Operator-facing flow: user_manual.md §4 Cohort Builder Web UI
+ §8 Scenario A.

The Cohort Builder path treats ips.pdhc as the source of truth for
"which patients belong to which organisation". The relationship is
many-to-many — one patient can belong to several clinics; one clinic
has many patients. See project_patients_live_in_ips memory + the
ips.pdhc commit c399add for the data model.

Dependency chain

Cohort Builder run
   │
   ├─► sim.plandef_client.get_plandefinition(guid)         (plan.pdhc, public HTTPS)
   ├─► sim.ips_client.list_clinic_patients(clinic_guid)    (ips.pdhc, ApiKey)
   ├─► sim.cohort_builder.walk_plandef_terms(plandef)
   ├─► sim.cohort_builder.build_cohort_rows(...)
   └─► sim.client.CdrClient.push_flat_rows(...)            (cdr_6, X-Service-Key + tunnel)

ips_client

sim/ips_client.py — thin HTTP client:

cohort_builder

sim/cohort_builder.py — plandef-driven engine:

Required state in ips.pdhc for the Cohort Builder to work

Population factors (Phase 1)

Operator-facing reference: user_manual.md §7 Population factors.

_population_shift(patient, pop_cfg, concept_guid, period_start)
returns an additive shift to add to the value drawn from the
distribution. Two factors:

Factors compose additively. Missing demographics fall through silently
(zero contribution from that factor, not an error).

Determinism preservation: the shift is applied to v after
_draw_value returns. The RNG stream is never consulted for the
shift; the only inputs are the operator's static pop_cfg constants
and the patient's static IPS demographics. So same (plandef, params, patients) with population turned on vs off differs by exactly the
constant shift per row — RNG draws are byte-identical either way.

Out of scope for Phase 1 (could be future Phase 2):
- Multiplicative shifts (v *= factor)
- Percentile-based shifts (e.g., "highest decile +5")
- Comorbidity-based factors (would need a Condition index)
- Seasonal factors (would need to consult effective_at, not just
patient demographics)

Synthea hookup (Shape C)

Operator-facing flow: user_manual.md §8 Scenario E.

sim.synthea_import provides a thin importer from Synthea's FHIR R4
output to ips.pdhc's clinic roster:

Why this is Shape C (not B): only the Patient identity layer
crosses over. Synthea's Observations, Conditions, MedicationRequests,
etc. stay in the source bundles. cdr_6 keeps a single source of truth
for measurements (sim's deterministic engine), and operators can opt
to use Synthea-imported patients as a richer roster than the IPS
admin's mock-data form provides — the rest of the Cohort Builder flow
is unchanged.

Companion endpoint on ips.pdhc:

Synthea hookup (Shape B) — Observations into cdr_6

Operator-facing flow: user_manual.md §8 Scenario F.

sim.synthea_obs provides the optional Shape B layer on top of Shape
C. Enabled via sim import-synthea --push-observations. Architecture:

Synthea bundle (FHIR R4 transaction Bundle)
  ├── Patient                          ─► Shape C: POST ips.pdhc
  └── Observation[] (subject.reference)
       ├── code.coding[0].system, .code
       └── valueQuantity                ─► Shape B: map → push cdr_6 flat row

Module entry points (sim/synthea_obs.py):

System-URL normalisation

Synthea Observations use HL7 well-known URLs (http://loinc.org,
http://snomed.info/sct, …). plan.pdhc's canonical libraries carry
their own URLs (https://LOINC.org, etc.) that don't always
case-match. Sim normalises both sides into short lowercase names via
_SYNTHEA_SYSTEM_TO_LIB_NAME (hardcoded — small enumeration, less
fragile than URL parsing). Adding a new system (e.g., RxNorm,
ICD-10-CM) means adding one line there.

Shape B characterisation contract

Every cdr_6 row emitted by Shape B has:

NULL plandef attribution is deliberate honesty. The Runs viewer's
chart layout still works (it groups by concept_guid), but a
downstream consumer querying "rows authored under PlanDefinition X"
won't see Synthea rows — which is the correct behaviour, because they
weren't.

Coverage gap

Plan.pdhc's concept catalogue is small (74 concepts at last count).
Synthea emits 600+ distinct LOINC codes across its modules. The
practical mapping rate against one of Synthea's adult-male bundles
(2026-06-03 smoke) was 126 mapped / 411 extracted = 31%. The drop
is silent by design — sim never invents concept_guids. Raising the
rate is a plan.pdhc concept-curation task, not a sim change.

16. Web UI architecture

Operator-facing flow: user_manual.md §4 Cohort Builder.

Single-process Flask app, started via python -m sim.web. Binds
127.0.0.1:9099. Stops on Ctrl+C. Does not auto-load .env —
the operator must set -a && source .env && set +a before launching.

Routes

Route Source What
GET / templates/cohort.html Cohort Builder page
GET /runs templates/runs.html Runs viewer page
GET /api/plan-definitions[?search=…] plan.pdhc proxy List PlanDefinitions
GET /api/plan-definitions/<guid> plan.pdhc proxy Fetch + attach _terms from walk_plandef_terms so the UI builds the override accordion without re-walking
GET /api/clinics ips.pdhc proxy List clinics
GET /api/clinics/<guid>/patients ips.pdhc proxy Roster
POST /api/cohort-build Engine entry body: {plandef_guid, clinic_guid, params, patients_max?, push?, target?, include_rows?}. Returns counts + (optional) row list. Optionally pushes to cdr_6.
GET /api/cdr6/runs cdr_6 proxy Runs list (via tunnel)
GET /api/cdr6/runs/<id>/observations[?…] cdr_6 proxy Row sample (via tunnel)
GET /api/health — {"status":"ok"}

cdr_6 tunnel management

sim.web keeps a single long-lived SSH port-forward to cdr_6
(loopback-only on miserver). _ensure_cdr6_tunnel() opens the tunnel
on first viewer hit (ssh -fN -M -S <sock>), atexit.register tears
it down at process exit. This avoids paying ~1s tunnel-open per
request — important for the viewer's drilldown UX.

Trade-off: a multi-worker deploy (e.g. gunicorn -w 4) would have
each worker opening its own tunnel. Out of scope for this single-
process dev/ops surface; document if/when it ever needs scaling.

Cohort Builder form architecture

templates/cohort.html is a single-page form with four panels (Step
1: PlanDefinition, Step 2: Organisation, Step 3: Parameters, Step 4:
Generate). The Parameters panel uses a shared factor-editor
component (factorEditorHtml(prefix, seed)) rendered both for the
defaults block and for each per-concept override card. Each editor
surfaces the full engine surface:

buildParams() walks the defaults editor + each open override
accordion card via readFactorEditor(root) and constructs the v2
params dict the engine accepts (see profile_v2_format.md for the
full schema). Only fields the operator actually filled get emitted —
empty inputs inherit from defaults at the engine level.

17. Runs viewer architecture

Operator-facing flow: user_manual.md §5 Runs viewer.

templates/runs.html is a separate page that reads from cdr_6.

Data flow

Browser
   │  GET /runs                                              → static HTML
   │  GET /api/cdr6/runs                                     → tunnel → cdr_6 /api/v1/runs (aggregate)
   │  Click row → GET /api/cdr6/runs/<id>/observations?limit=N
   │                                                         → tunnel → cdr_6 /api/v1/runs/<id>/observations
   │  Resolve concept names: GET /api/plan-definitions/<plandef_guid>
   │                                                         → plan.pdhc (already on the public path)
   └─► Render: stats card + per-concept time-series + sample-rows table

Charts

One Chart.js line plot per unique concept in the loaded
observations. One series per patient_guid. Values parsed as floats;
non-numeric rows skipped silently. Concept names resolved to
"name (unit)" via the run's plan_definition_guid (best-effort,
falls back to short guids).

cdr_6 read endpoint

GET /api/v1/runs/<sim_run_id>/observations (added in
cdr_6.pdhc f25d440):

The proxy in sim.web forwards only known query params
(concept_guid, patient_guid, limit) — keeps the proxy surface
deliberately small.

18. generate-from-plandef — Synthea-from-PlanDefinition

A third generation path (alongside §3's v2 engine and §15's Synthea import):
drive MITRE Synthea from a plan.pdhc PlanDefinition so a cohort's
observations are exactly that plan's concepts, on its cadence, over a known
time window — then land them via the existing Shape C / Shape B importers.

Pipeline

PlanDefinition (PlanDefClient)
  → synthea_module.build_modules   one looping GMF module per concept
  → synthea_runner.run_synthea     local jar, time-pinned + seeded, tmp workdir
  → synthea_import.import_synthea   Shape C: patients → ips.pdhc roster
  → _push_synthea_observations      Shape B: plan's stream → CDR (attributed)

CLI: sim generate-from-plandef --plandef <guid|title> --clinic <guid> [--patients N] [--seed S] [--start YYYY-MM-DD] [--end YYYY-MM-DD] [--age MIN-MAX] [--push-observations] [--target cdr_6].

synthea_module.py — PlanDefinition → GMF

One module file per concept (Synthea runs each *.json in the -d dir as
an independent concurrent state machine, so lanes need not be interleaved).
Each module loops continuously: Initial → Onset_Delay → Encounter → Observation → EncounterEnd → Cadence_Delay → Encounter ….

synthea_runner.py — the jar

Locates java (brew openjdk@17 → PATH) + the jar (SYNTHEA_JAR_PATH), runs
in a tempfile.mkdtemp() workdir, returns the out/fhir/ path. Time-window
pinning: -r/-e do not filter output (Synthea simulates a whole life);
the lever is -e <end> + --exporter.years_of_history = round((end-start) / 365.25), which trims to [end − N y, end]. Continuous-loop modules guarantee
points inside that window; residual year-granular bleed is removed by the
mapper's exact date filter (below).

synthea_obs.py mapper edits

Run-level identity GUIDs

A cohort is generated as if under a contract, requested by an org, by a
user. These three are constant for the whole simulation and stamped on
every emitted row (cdr_6's ingest already persists them):

Omitted → honest NULLs (the legacy import-synthea path leaves them None).

Deferred (v1)

Quantity concepts only (categorical/valueset skipped — the mapper keeps
valueQuantity). Plan cadence bounds (count/duration) are recorded in the
module remarks but not enforced — the export window defines the period.
Local-only (needs Java 17 + the jar); no server deploy.

19. cdr-transfer — move a CDR's content into another CDR

sim cdr-transfer --from <src> --to <dst> [--sim-run-id X] [--purge-source] [--dry-run] [--batch-size 100] copies (or, with --purge-source, moves) a
source CDR's complete content into a destination CDR. The motivating use:
generate a synthetic cohort into the cdr_6 sink, then promote it into cdr2–5.

Path: source GET /api/v1/export (cursor-paged, unfiltered — see below)
→ map each row to the flat ingest item (_row_to_ingest_item, dropping cdr_6
internals like id/received_at) → destination POST /api/v1/ingest/batch
in ≤100-row batches (cdr1–5 cap). Reads land in the target's observation
table via IngestPipeline.process — the same shape cdr2–5 already hold.