Feature: Automated ingestion, decryption, text extraction, patient matching, and filing of The Doctor's Laboratory (TDL) pathology reports.
App:lab_documents(Django 5.2)
Status: Production — Phases 1-3 complete
User uploads PDF ──► Django API ──► Celery: process_lab_document()
│
┌─────────┴──────────┐
▼ ▼
pdfplumber ClinicLabConfig
(text extract) (TDL password)
│ │
▼ ▼
parse_patient_data() try_decrypt_file()
(regex parser) (pikepdf / zipfile)
│ │
▼ ▼
find_matching_patient() ──► Decrypted PDF
(Python scoring)
│
▼
┌────────┴───────┐
▼ ▼
Auto-Matched Needs Review
(score ≥ 90) (score < 90)
│ │
▼ ▼
Filed to Verify & File
Patient Record (split-screen UI)
For email ingestion, a Celery beat task (poll_lab_report_mailboxes) polls connected shared mailboxes every 5 minutes via Microsoft Graph API (app-only auth), downloads encrypted attachments, decrypts them, and feeds them into the same pipeline.
All models live in the lab_documents app (routed to sector_gp / sector_client_* via the existing SectorDatabaseRouter).
LabDocument| Field | Type | Purpose |
|---|---|---|
id |
UUID (PK) | Auto-generated |
patient |
FK → CustomUser (nullable) |
Matched patient (null until matched) |
file |
FileField |
Uploaded PDF (stored in MinIO / local filesystem at lab_documents/%Y/%m/%d/) |
source |
ChoiceField | DIRECT, EMAIL, PORTAL |
status |
ChoiceField | PENDING → MATCHED / REVIEW / ERROR → FILED |
clinic_ref |
CharField |
Lab reference number / Order ID from the report |
extracted_raw_text |
TextField |
Full text extracted by pdfplumber |
metadata |
JSONField |
Parsed patient data: name, dob, lab_ref, nhs_number |
uploaded_by |
FK → CustomUser |
Who uploaded / filed the document |
processing_notes |
TextField |
Status messages, error details |
Status workflow:
PENDING ──► MATCHED ──► FILED
│
└──► REVIEW ──► FILED
│
ERROR ◄─────────────┘
ClinicLabConfig| Field | Type | Purpose |
|---|---|---|
practice |
FK → Practice (unique) |
Practice this config belongs to |
tdl_account_id |
CharField |
TDL account identifier |
tdl_password |
EncryptedCharField |
TDL decryption password (AES-256 encrypted at rest) |
lab_report_mailbox |
EmailField |
Shared mailbox email for email ingestion |
is_active |
BooleanField |
Toggle integration on/off |
LabReportMailbox| Field | Type | Purpose |
|---|---|---|
clinic_config |
FK → ClinicLabConfig |
Parent config |
mailbox_email |
EmailField (unique) |
Shared mailbox address |
oauth_token |
EncryptedCharField |
App-only OAuth token (auto-managed) |
is_connected |
BooleanField |
Connection status |
last_fetched_at |
DateTimeField |
Last successful poll |
| URL | Purpose | Auth |
|---|---|---|
/lab/ |
Lab inbox — all unfiled documents | LoginRequired |
/lab/upload/ |
Drag-and-drop PDF upload | LoginRequired |
/lab/config/ |
Practice TDL settings | LoginRequired |
/lab/documents/<uuid>/ |
Document detail + PDF preview | LoginRequired |
/lab/documents/<uuid>/verify/ |
Split-screen verify & file | LoginRequired |
/lab/documents/<uuid>/pdf/ |
PDF serving (iframe-safe) | LoginRequired |
/lab/api/documents/ |
CRUD + upload API | Token/Session |
/lab/api/documents/<uuid>/match/ |
POST to match & file | Token/Session |
/lab/api/documents/<uuid>/reprocess/ |
POST to reprocess | Token/Session |
/lab/api/mailboxes/ |
CRUD for connected mailboxes | Token/Session |
/lab/api/clinic-config/<practice_id>/ |
Get/update clinic config | Token/Session |
/lab/api/patients/search/?q= |
Patient search (decrypts in Python) | Token/Session |
/lab/api/pending-review/ |
All documents needing review | Token/Session |
A "Lab Integration" section is also visible on the Practice Manager Dashboard, linking to Inbox, Upload, and Settings.
Uses pdfplumber (free, Python-native) to extract text from digital PDFs. No OCR needed — TDL reports are digitally generated PDFs with selectable text.
The parser targets standard TDL field labels (defined in lab_documents/tasks.py):
| Field | Pattern | Example |
|---|---|---|
| Name | Patient Name: / Patient: / Name: |
Patient: Smith, John |
| DOB | DOB: / Date of Birth: / Birth Date: |
DOB: 15/01/1980 |
| Lab Ref | Lab Ref: / Order ID: / Specimen ID: |
Lab Ref: TDL-98472-X |
| NHS Number | NHS Number: / NHS No: |
NHS Number: 123 456 7890 |
Matching runs entirely in Python (not SQL) because patient first_name, last_name, and date_of_birth are encrypted at rest via EncryptedCharField / EncryptedDateField.
The algorithm in find_matching_patient():
Smith, John and John Smith formats)DD/MM/YYYY, DD-MM-YYYY, YYYY-MM-DD, DD.MM.YYYY)When pdfplumber fails to extract text (encrypted PDF), the pipeline uses the active ClinicLabConfig.tdl_password to decrypt:
| Format | Library | Method |
|---|---|---|
| AES-encrypted PDF | pikepdf (C++ backed) | pikepdf.open(file, password=...) |
| Fallback PDF | PyPDF2 | PdfReader.decrypt(password) |
| Encrypted ZIP | zipfile (stdlib) | ZipFile.setpassword(password) |
gp booking app with Mail.Read Application permission (admin-consented)gp booking app with Mail.Send Application permission (admin-consented)poll_lab_report_mailboxes fires every 300 secondsLabReportMailbox records_get_app_only_token() (client credentials flow)GET /users/{mailbox}/messages?$filter=from/emailAddress/address eq '@tdlpathology.com'pdf_utils)LabDocument with source=EMAILprocess_lab_document.delay() to parse, match, and file/lab/config/Path: /lab/config/
EncryptedCharField)| Concern | Measure |
|---|---|
| Password storage | AES-256 encrypted at rest via django-encrypted-model-fields |
| PDF data | Stored in MinIO (S3-compatible object storage) |
| Patient data isolation | All LabDocument records stored in the tenant's own sector_client_* database |
| Transport encryption | All API calls over HTTPS; database connections via WireGuard tunnel |
| Email access | App-only OAuth (no user password), MS Graph API |
| PDF viewing | Served via dedicated xframe_options_exempt view to bypass X-Frame-Options |
| Audit trail | All document accesses logged via the existing AuditLogger infrastructure |
# Run migrations (already applied)
python manage.py migrate lab_documents --database=sector_gp
python manage.py migrate lab_documents --database=sector_client
# Test text extraction
python -c "
from lab_documents.tasks import extract_text_from_pdf, parse_patient_data
text = extract_text_from_pdf('path/to/report.pdf')
data = parse_patient_data(text)
print(data)
"
# Trigger reprocess on a specific document
python -c "
from lab_documents.tasks import process_lab_document
process_lab_document.delay('<doc-uuid>')
"
# Manual mailbox poll
python -c "
from lab_documents.tasks import poll_lab_report_mailboxes
poll_lab_report_mailboxes()
"