Receipt OCR API

ENGINEERING CASE STUDY · DOCUMENT INTELLIGENCE

Engineering case study

Converting receipts across hospitals, layouts, and scan conditions into one JSON contract that downstream systems can consume.

01

Problem & constraints

Problem

Medical receipts vary in layout, labels, table structure, and image quality. Raw OCR can read text but cannot preserve field meaning, monetary relationships, or a stable API format.

Constraints

• Inputs include scans and phone photos with skew, shadows, perspective, and low contrast.
• Hospitals use different labels, table positions, and fee categories.
• Insurance type, dates, totals, and line items must retain their relationships.
• A new hospital must plug in without rewriting the recognition pipeline.

02

Architecture & decisions

System flow

1. 01 Image input Scanned receipts · phone photos
2. 02 Image correction UVDoc · deskew · shadow removal · OpenCV
3. 03 Layout detection YOLOv7 Stage 1 · region cropping
4. 04 Text and table OCR PaddleOCR · YOLOv7 Stage 2
5. 05 Hospital normalization HospitalPipeline · regex config · field mapping
6. 06 Structured output Unified schema · API-ready JSON

Technology decisions

YOLOv7 + PaddleOCR

Detecting layout regions before OCR preserves field and table order better than reading the full receipt at once.

UVDoc + OpenCV

Treats perspective, skew, and shadows as engineering problems before recognition.

HospitalPipeline

Isolates hospital variation in configuration, regex, and mappings while keeping the core pipeline stable.

Fixed JSON contract

The pipeline is designed backward from what API consumers need, not from raw OCR text.

03

My contribution

1. 01

   Designed the end-to-end path from image correction and region detection to JSON.

2. 02

   Implemented layout selection, field/table detection, regex, and field mappings.

3. 03

   Built the HospitalPipeline extension boundary for hospital-specific adapters.

4. 04

   Defined the downstream API schema and verified real receipts field by field.

04

Evaluation & outcomes

End-to-end acceptance uses real layouts from multiple hospitals. Verification covers insurance type, dates, hospital, department, totals, and line items mapped into one schema, retaining manual review for low-resolution and atypical scans.

5+

Hospital formats normalized NTU, Chang Gung, CCH, VGHTPE, Chi Mei

6

Processing stages From image input to JSON contract

1

Unified API Schema Directly consumable by downstream APIs

05

Failures & corrections

01

Failure

Full-page OCR produced readable text but unstable field order and table relationships.

Correction

Added two-stage YOLO detection before OCR.

Lesson

Document understanding includes layout, not only character accuracy.

02

Failure

One shared regex set caused new hospital formats to break old parsing.

Correction

Separated each hospital into configuration, mapping, and a pipeline adapter.

Lesson

Fast-changing rules belong behind a stable boundary.

03

Failure

Raw OCR output could not be consumed by finance or claims systems.

Correction

Defined the JSON contract first, then designed normalization and acceptance fields.

Lesson

Integration usability matters more than whether the model can read text.

06

Evidence & further reading

Repository GitHub Repository ↗ Documentation Pipeline and output examples →

Deep dive

Technical implementation notes

Detailed workflows, implementation decisions, diagrams, and project artifacts.

Context

Medical receipts need to be integrated with financial, claims, or internal systems, but the layout and field formats vary across hospitals, making manual entry time-consuming and prone to errors. The context requires generating unified, machine-readable structured data from scanned documents or photos for direct use by downstream APIs.

Challenge

• The receipt layouts and field positions of major hospitals in Taiwan vary greatly; a single rule cannot cover them all.
• Scan quality (skew, shadows, low resolution) affects OCR recognition rates.
• Some receipts contain tables (fee details), which require block detection before field parsing.

Solution

This project implements end-to-end receipt normalization: regardless of differences in scan quality or hospital layouts, it can output a unified structured JSON. The approach automatically switches between a two-stage YOLO detection and hospital-specific field parsing based on “whether a table is included,” paired with UVDoc flattening, skew/shadow correction, and PaddleOCR to reduce errors caused by image quality.

• Supports over 5 common hospitals including NTU Hospital, Chang Gung, CCH, Veterans General Hospital, and Chi Mei, using customized regex and field extraction scripts for recognition.
• Output fields include nhi, admissionDate, dischargeDate, receivedAmount, items (fee details), etc., which can be integrated with existing APIs.

Processing Pipeline

1. Image Preprocessing — Orientation detection, UVDoc flattening, shadow and noise suppression (ocr_methods.py, correct_skew_eliminate_shadows.py, UVDoc/).
2. YOLO Stage 1 — Detect receipt regions (yolov7_detect.py).
3. Cropping and Re-correction — If necessary, use crop_image_from_label.py to crop out blocks.
4. OCR — Use PaddleOCR (det + rec) to get the full text, and determine the hospital category based on hospital_key.txt.
5. Table Detection — If a table is present, enable YOLO Stage 2 to detect table blocks.
6. Hospital Pipeline — Enter the corresponding HospitalPipeline (hospital_pipeline.py), perform field regularization and table enhancement according to the hospital (receipt_uni/info/*.py, receipt_uni/config/regex_*.txt).
7. Output — Convert to standard JSON with convert_df_to_api_format.py, and output via generate_json_result.

When adding a new hospital, the same logic is applied: determine if a table is included → write field regex and a custom extraction script.

Pipeline and Output Examples

Processing Pipeline — The flow from image input to JSON output.

Below are RFC 8259-compliant JSON samples demonstrating unified output across hospitals (samples use de-identified test data; fields cover NHI status, admission/discharge dates, department, total amount, and fee breakdown).

NTU Hospital

{
  "file": "ntu_sample_1.jpg",
  "result": {
    "nhi": "Y",
    "admissionDate": "2023/07/19",
    "dischargeDate": "2023/07/23",
    "hospitalName": "National Taiwan University Hospital",
    "dept": "Orthopedics",
    "receivedAmount": 84327,
    "items": {
      "medicationFee": 251,
      "treatmentFee": 520,
      "materialFee": 69006,
      "certificateFee": 150,
      "wardFee": 14400
    }
  }
}

Chang Gung Memorial Hospital

{
  "file": "cg_sample_1.jpg",
  "result": {
    "nhi": "Y",
    "admissionDate": "2023/07/28",
    "dischargeDate": "2023/07/28",
    "hospitalName": "Linkou Chang Gung Memorial Hospital",
    "dept": "General Surgery",
    "receivedAmount": 20610,
    "items": {
      "inpatientCopay": 4651,
      "medicationFee": 553,
      "materialFee": 5520,
      "procedureFee": 9886
    }
  }
}

Changhua Christian Hospital (CCH)

{
  "file": "ck_sample_1.jpg",
  "result": {
    "nhi": "Y",
    "admissionDate": "2023/07/21",
    "dischargeDate": "2023/07/27",
    "hospitalName": "Changhua Christian Hospital",
    "dept": "Otolaryngology — Head and Neck",
    "receivedAmount": 49430,
    "items": {
      "medicationFee": 1349,
      "materialFee": 41919,
      "treatmentFee": 650,
      "copay": 5512
    }
  }
}

Field Mapping and Manual Review Boundaries

• Field Extraction Sources: hospitalName is inferred from key detection; nhi, admissionDate, and dischargeDate map to header text; dept and receivedAmount are extracted via Stage 1 bounding boxes; items fee entries are extracted via Stage 2 table detection and normalized through regex mapping.
• Manual Review Triggers:
  1. Low-resolution scans (< 150 DPI) or severe perspective distortion trigger human review flags instead of forced extrapolation.
  2. Arithmetic mismatch between fee items sum and receivedAmount triggers a validation alert.
  3. Unregistered hospital templates or handwritten receipts are routed directly to manual review.

Tech Stack

• OCR — PaddleOCR (det / rec), Traditional Chinese weights (e.g., ch_PP-OCRv4_det, tw_PP-OCRv3_rec).
• Detection — YOLOv7 (Stage 1 for receipt regions, Stage 2 for table blocks).
• Image Preprocessing — UVDoc flattening, deskew, shadow elimination; OpenCV, scikit-image.
• Environment — Python 3.9+; Optional CUDA GPU acceleration.

Dependencies: paddleocr, paddlepaddle-gpu, torch, torchvision, opencv-python-headless, numpy, pandas, Pillow, scikit-image, PyYAML, etc.

Expanding to New Hospitals

• hospital_pipeline.py defines the abstract class HospitalPipeline and its implementations for various hospitals (NTU, Chang Gung, CCH, Veterans General Hospital, Chi Mei, etc.).
• receipt_uni/info/*.py contains hospital-specific field logic; receipt_uni/config/regex_*.txt holds mappings between fields and regex.

Suggested Steps:

1. Add the hospital keyword and key to hospital_key.txt.
2. Create a new parsing script in info/ and regex_<HOSP>.txt (and regex_<HOSP>_table.txt if necessary).
3. Implement a new class in hospital_pipeline.py (get_ocr_result, crop_from_label, text_info, table_info, etc.).
4. Adjust hospital_api_map.txt as needed.

Impact and Boundary Notes

• Format Coverage: Over 5 hospitals supported (NTU, Chang Gung, CCH, Veterans General Hospital, Chi Mei, etc.) with a single unified pipeline.
• Unified Contract: Delivers a stable schema directly consumable by downstream financial/claims APIs without layout-specific code.
• Boundary Clarification: This is a format coverage and structural normalization metric; it does not claim 100% unsupervised accuracy in production. Degraded or atypical scans retain human verification boundaries.

Extension

• Expand to more hospitals and receipt types (outpatient, clinics, long-term care receipts).
• Integrate with claims or billing workflows for one-click completion from scanning to review.
• Add accuracy monitoring and manual sampling interfaces to continuously optimize recognition and field mapping.