Project note

Receipt OCR API

PaddleOCR · YOLOv7 · Hospital Receipt Structuring · End-to-end Normalization

Flagship Jan 10, 2025 4 min read Updated Jul 27, 2026

ENGINEERING CASE STUDY · DOCUMENT INTELLIGENCE

Engineering case study

Converting receipts across hospitals, layouts, and scan conditions into one JSON contract that downstream systems can consume.

Problem & constraints

Problem

Medical receipts vary in layout, labels, table structure, and image quality. Raw OCR can read text but cannot preserve field meaning, monetary relationships, or a stable API format.

Constraints

Inputs include scans and phone photos with skew, shadows, perspective, and low contrast.
Hospitals use different labels, table positions, and fee categories.
Insurance type, dates, totals, and line items must retain their relationships.
A new hospital must plug in without rewriting the recognition pipeline.

Architecture & decisions

System flow

01 Image input Scanned receipts · phone photos
02 Image correction UVDoc · deskew · shadow removal · OpenCV
03 Layout detection YOLOv7 Stage 1 · region cropping
04 Text and table OCR PaddleOCR · YOLOv7 Stage 2
05 Hospital normalization HospitalPipeline · regex config · field mapping
06 Structured output Unified schema · API-ready JSON

Technology decisions

YOLOv7 + PaddleOCR

Detecting layout regions before OCR preserves field and table order better than reading the full receipt at once.

UVDoc + OpenCV

Treats perspective, skew, and shadows as engineering problems before recognition.

HospitalPipeline

Isolates hospital variation in configuration, regex, and mappings while keeping the core pipeline stable.

Fixed JSON contract

The pipeline is designed backward from what API consumers need, not from raw OCR text.

My contribution

01
Designed the end-to-end path from image correction and region detection to JSON.
02
Implemented layout selection, field/table detection, regex, and field mappings.
03
Built the HospitalPipeline extension boundary for hospital-specific adapters.
04
Defined the downstream API schema and verified real receipts field by field.

Evaluation & outcomes

End-to-end acceptance uses real layouts from multiple hospitals. Verification covers insurance type, dates, hospital, department, totals, and line items mapped into one schema—not just character recognition.

Hospital formats Each with layout and field rules

Processing stages From image input to JSON contract

Unified output Directly consumable by downstream APIs

Failures & corrections

Failure: Full-page OCR produced readable text but unstable field order and table relationships.
Correction: Added two-stage YOLO detection before OCR.
Lesson: Document understanding includes layout, not only character accuracy.

Failure: One shared regex set caused new hospital formats to break old parsing.
Correction: Separated each hospital into configuration, mapping, and a pipeline adapter.
Lesson: Fast-changing rules belong behind a stable boundary.

Failure: Raw OCR output could not be consumed by finance or claims systems.
Correction: Defined the JSON contract first, then designed normalization and acceptance fields.
Lesson: Integration usability matters more than whether the model can read text.

Evidence & further reading

Repository GitHub Repository Documentation Pipeline and output examples

Deep dive

Technical implementation notes

Detailed workflows, implementation decisions, diagrams, and project artifacts.

Context

Medical receipts need to be integrated with financial, claims, or internal systems, but the layout and field formats vary across hospitals, making manual entry time-consuming and prone to errors. The context requires generating unified, machine-readable structured data from scanned documents or photos for direct use by downstream APIs.

Challenge

The receipt layouts and field positions of major hospitals in Taiwan vary greatly; a single rule cannot cover them all.
Scan quality (skew, shadows, low resolution) affects OCR recognition rates.
Some receipts contain tables (fee details), which require block detection before field parsing.

Solution

This project implements end-to-end receipt normalization: regardless of differences in scan quality or hospital layouts, it can output a unified structured JSON. The approach automatically switches between a two-stage YOLO detection and hospital-specific field parsing based on “whether a table is included,” paired with UVDoc flattening, skew/shadow correction, and PaddleOCR to reduce errors caused by image quality.

Supports over 5 common hospitals including NTU Hospital, Chang Gung, CCH, Veterans General Hospital, and Chi Mei, using customized regex and field extraction scripts for recognition.
Output fields include nhi, admissionDate, dischargeDate, receivedAmount, items (fee details), etc., which can be integrated with existing APIs.

Processing Pipeline

Image Preprocessing — Orientation detection, UVDoc flattening, shadow and noise suppression (ocr_methods.py, correct_skew_eliminate_shadows.py, UVDoc/).
YOLO Stage 1 — Detect receipt regions (yolov7_detect.py).
Cropping and Re-correction — If necessary, use crop_image_from_label.py to crop out blocks.
OCR — Use PaddleOCR (det + rec) to get the full text, and determine the hospital category based on hospital_key.txt.
Table Detection — If a table is present, enable YOLO Stage 2 to detect table blocks.
Hospital Pipeline — Enter the corresponding HospitalPipeline (hospital_pipeline.py), perform field regularization and table enhancement according to the hospital (receipt_uni/info/*.py, receipt_uni/config/regex_*.txt).
Output — Convert to standard JSON with convert_df_to_api_format.py, and output via generate_json_result.

When adding a new hospital, the same logic is applied: determine if a table is included → write field regex and a custom extraction script.

Pipeline and Output Examples

Processing Pipeline — The flow from image input to JSON output.

Below are examples of output formats after recognizing receipts from various hospitals (using filename as the key, with fields including NHI, admission/discharge dates, department, received amount, and items details).

NTU Hospital

"ntu-receipt-1.jpg" : {
    'nhi': 'Y',
    'admissionDate': '2023/07/19',
    'dischargeDate': '2023/07/23',
    'hospitalName': 'National Taiwan University Hospital',
    'dept': 'Orthopedics',
    'receivedAmount': '84327',
    'items': {
        'medicationFee': '251',
        'treatmentFee': '520',
        'materialFee': '69006',
        'certificateFee': '150',
        'wardFee': '14400'
    }
},

Chang Gung

"chang-gung-receipt-1.jpg" : {
    'nhi': 'Y',
    'admissionDate': '2023/07/28',
    'dischargeDate': '2023/07/28',
    'hospitalName': 'Linkou Chang Gung Memorial Hospital',
    'dept': 'General Surgery',
    'receivedAmount': '20610',
    'items': {
        'inpatientCopay': '4651',
        'medicationFee': '553',
        'materialFee': '5520',
        'procedureFee': '9886'
    }
},

Changhua Christian Hospital (CCH)

"cch-receipt-1.jpg" : {
    'nhi': 'Y',
    'admissionDate': '2023/07/21',
    'dischargeDate': '2023/07/27',
    'hospitalName': 'Changhua Christian Hospital',
    'dept': 'Otolaryngology — Head and Neck',
    'receivedAmount': '49430',
    'items': {
        'medicationFee': '1349',
        'materialFee': '41919',
        'treatmentFee': '650',
        'copay': '5512'
    }
},

Tech Stack

OCR — PaddleOCR (det / rec), Traditional Chinese weights (e.g., ch_PP-OCRv4_det, tw_PP-OCRv3_rec).
Detection — YOLOv7 (Stage 1 for receipt regions, Stage 2 for table blocks).
Image Preprocessing — UVDoc flattening, deskew, shadow elimination; OpenCV, scikit-image.
Environment — Python 3.9+; Optional CUDA GPU acceleration.

Dependencies: paddleocr, paddlepaddle-gpu, torch, torchvision, opencv-python-headless, numpy, pandas, Pillow, scikit-image, PyYAML, etc.

Expanding to New Hospitals

hospital_pipeline.py defines the abstract class HospitalPipeline and its implementations for various hospitals (NTU, Chang Gung, CCH, Veterans General Hospital, Chi Mei, etc.).
receipt_uni/info/*.py contains hospital-specific field logic; receipt_uni/config/regex_*.txt holds mappings between fields and regex.

Suggested Steps:

Add the hospital keyword and key to hospital_key.txt.
Create a new parsing script in info/ and regex_<HOSP>.txt (and regex_<HOSP>_table.txt if necessary).
Implement a new class in hospital_pipeline.py (get_ocr_result, crop_from_label, text_info, table_info, etc.).
Adjust hospital_api_map.txt as needed.

Impact

Supported Hospitals: Over 5 (NTU Hospital, Chang Gung, CCH, Veterans General Hospital, Chi Mei, etc.), outputting unified JSON from a single pipeline.
Output Format: Fields like nhi, admissionDate, dischargeDate, receivedAmount, items (fee details), etc., which can be integrated with existing financial/medical APIs; downstream systems do not need to handle hospital layout differences.
Expansion Cost: Adding a new hospital only requires writing field regex and extraction logic; the same pipeline is reused to maintain a single API format.

Extension

Expand to more hospitals and receipt types (outpatient, clinics, long-term care receipts).
Integrate with claims or billing workflows for one-click completion from scanning to review.
Add accuracy monitoring and manual sampling interfaces to continuously optimize recognition and field mapping.

PaddleOCRYOLOv7Multi-hospital Pipeline