Datascape
Compensation terms read from any physician contract, built inside the client's HIPAA cloud
Datascape's analysts needed compensation terms from physician contracts, and the existing code read only one layout. Our engineers worked inside its Slack, GitHub and HIPAA AWS account and replaced layout rules with one RAG pipeline that reads any contract.
Healthcare & WellnessContract data extraction pipelineAI added to a live product
- Fields per contract
- 13, from provider to quality bonus
- Runs in
- The client's own HIPAA-aligned AWS account
- Worked in
- The client's Slack, GitHub and AWS, under a BAA
- Missing terms
- Returned as NULL, never guessed
The challenge
Extraction code built on three similar contracts failed on the fourth. Contracts put terms in tables, paragraphs or appendices, and amendments carry only a few fields.
What we did
One RAG pipeline over Markdown with a definition for every field, built in the client's GitHub and deployed on Lambda in its restricted AWS account, returning NULL when a term is absent.
The outcome
The client approved the second version after its first review, the pipeline went live in its own account, and Datascape started a second project with us right away.
The starting point
Datascape is a Denver data consultancy that works under HIPAA as a business associate for healthcare organizations. Its analysts needed key terms from physician employment contracts stored in S3, and existing notebooks used AWS Textract and Amazon Bedrock to pull them.
What had to change
The code worked on three contracts with a similar layout and failed on a fourth. Some contracts put compensation in tables, others in paragraphs, and amendments arrive with only a few fields.
What could not break
- Contracts never leave the client's restricted AWS account, where public traffic is denied at the network level
- We work only with the AWS permissions the client grants, under a signed business associate agreement
- Every change ships as a small, versioned release the client can review
- The function has to fit within Lambda's package size limit
- A term the contract does not state must come back empty, never invented
The calls we made
We treated it as a generalization problem, not a fix for one sample contract, and solved it inside the client's own accounts.
- 01
One generic pipeline instead of layout rules
Each contract is converted to Markdown, vector search finds the passages for each field, and a model returns clean key value pairs, with a definition per field so synonyms such as contract effective date still match.
Ruled outParsing rules written for each contract layout, which broke on the next format.
- 02
Roll back the cleverer step
A reranking step that kept only chunks with dollar amounts sometimes lost context in long contracts, so we returned to the simpler pipeline with field definitions, which read them correctly.
Ruled outStacking more retrieval tricks without a gain in accuracy.
- 03
NULL instead of a guess
When a contract does not state a term, the field comes back empty, so analysts never act on a value the contract does not hold.
Ruled outLetting the model infer missing terms.
- 04
Appendices handled on their own
Appendix sections are retrieved separately, summarized by a small model and added to the main prompt, because retrieval alone or heading detection both missed the figures that live there.
Ruled outRetrieval alone or heading detection, which both missed appendix figures.
- 05
Fit the pipeline to Lambda
The first package went over Lambda's size limit once unzipped, so we swapped Chroma for FAISS and shipped the function as a container image through Amazon ECR.
Ruled outKeeping Chroma, which pushed the package past the limit.
How the system fits together
What we built
Layout independent extraction
A RAG pipeline over Markdown that returns provider name, FTE status, clinical allocation, specialty, board certifications, hospital locations, sign on bonus, contract dates, base compensation, wRVU threshold, incentive rate and quality bonus as JSON, with the file name and run details.
Serverless deployment in the client's account
The pipeline runs on AWS Lambda from a container image in Amazon ECR, triggered by every upload to the raw documents bucket, so analysts never start a run by hand.
Amendments and appendices
Amendments that carry only a few fields are handled on their own, compensation is split into its clinical and academic portions, and appendix figures reach the model through a separate summary.
A fallback that keeps it running
A fallback function keeps extraction going when a step fails, so a single failed step does not stop the extraction.
How it works
Upload
An analyst drops a contract or amendment into the raw documents bucket.
Convert
The upload triggers the Lambda function, which turns the PDF into Markdown so tables and paragraphs read the same way.
Retrieve
Vector search finds the passages for each field, with a definition per field so synonyms still match.
Extract
A model on Amazon Bedrock returns clean key value pairs, with appendix figures added from a separate summary.
Output
The terms come back as JSON with the file name and run details, and NULL wherever a term is absent.
A closer look
{
"file_name": "employment_agreement_sample.pdf",
"execution_time_s": 14,
"fte_status": {
"KEY": "FULL TIME EQUIVALENT (FTE) STATUS:",
"VALUE": "Full-time, 1.0 FTE"
},
"specialty": {
"KEY": "SPECIALTY:",
"VALUE": "Cardiology"
},
"productivity_threshold": {
"KEY": "PRODUCTION THRESHOLD",
"VALUE": "5,200 Physician wRVUs per Contract Year"
},
"quality_bonus": {
"KEY": "NULL",
"VALUE": "NULL"
}
}What changed
The client approved the second version after its first round of feedback, and the pipeline went live on Lambda in its own AWS account. When the project closed, Datascape started a second project with us right away.
- Second version approved after one review
- Approval
- A second project started right away
- Repeat work
“Umair is a proven professional who knows what it takes to get things done. He has a deep well of knowledge he can tap for tackling complex development problems. I'm looking forward to continuing work together.”
DatascapeUnder the hood
- Datics role
- Forward deployed engineering
- Integrations
- Amazon S3 upload triggers, AWS Lambda from an Amazon ECR container image, Amazon Bedrock for extraction and Amazon Nova Micro for appendix summaries.
- Data and security boundary
- Contracts and results stay in the client's AWS account under a signed business associate agreement, with public traffic denied at the VPC level and permissions granted by the client.
- Retrieval
- Markdown chunks indexed with FAISS, chosen over Chroma to fit the Lambda package limit, with a definition per field for synonyms.
- Output
- JSON per contract with the matched contract label and value, the file name, run time and date, and NULL for absent terms.
AI and models
Cloud and infrastructure
Apps and APIs
Also in the stack
How it shipped
Review the existing notebooks
We read the client's Textract and Bedrock notebooks in its own GitHub, reproduced the failure on the fourth contract and proposed a generic approach instead of a fix for one sample.
First pipeline, live on Lambda
With the AWS permissions the client granted, the RAG pipeline went live on Lambda inside its account, and the client tested it on its own contracts. Code, documentation and a diagram went to its GitHub.
Client review and second version
Six changes from the client's first review, including run details in the output and support for amendments, and the client approved the second version.
Appendices and fallback
Later versions returned NULL instead of N/A, split compensation into its portions, added appendix summaries and a fallback, each redeployed to Lambda.
Inside the client's own tools
Under a signed business associate agreement, the client's CTO onboarded our engineers into its Slack. All code went to its GitHub and ran in its HIPAA AWS account, with access it granted.
Small releases, reviewed every week
Weekly syncs with the client's team, and every change shipped as a versioned release the client could test before the next.
Handed over in place
Each release left the code, documentation, a diagram and a recorded walkthrough in the client's own repository and account.
What this unlocked
Datascape started a second project with us as soon as this one closed: crawling provider data from practice websites and verifying it against the NPI Registry.
Related work
Have a product with the same kind of constraints?
Tell us what already exists and what you want it to do. We will tell you what it takes.