Flashpoint: Efficient & Scalable Entity-wise Document Comparison ¶
1. Introduction ¶
Document comparison is a notoriously difficult problem to solve in the world of NLP and analytics, even with the availability of modern LLMs. Between comparing terms of a contract, technical documents, or revisions of SOPs, enterprises in every sector stand to benefit greatly from novel document comparison approaches that are fast and scalable. That's what Flashpoint seeks to deliver.
This is also a great chance to practice AI-driven development approaches. As a data scientist, I spend every day developing custom AI solutions for business use cases, but most of that development work is still hands-to-keyboards with tools like coding agents supplementing my own work. This allows for a better degree of tailoring for the solutions I create along with much better maintainability, having a developer who actually knows the code base. With this solution, however, I'm going with an AI first development approach in an effort to produce a truly production-grade and maintainable AI product.
This writeup is fully human composed and written, but pretty much everything else in this project code base has been developed with the assistance of Claude Code.
1.1 Challenges of Document Comparison ¶
LLMs are great: they've opened up possibilities that traditional machine learning and NLP methods simply do not provide, including conceptual, semantically based comparisons of documents. With the continued development of more and more advanced LLMs with bigger and bigger context windows, the task of dumping two documents into an LLM and asking it to perform a comparison is trivial. However, you get what you pay for, low effort yielding low-quality results, at least if you need a detailed analysis. For broad, high-level differences and for small documents of only a few pages, this is a perfectly feasible approach, but for detecting differences in language of several hundred page documents, that approach will not be able to provide any meaningfully detailed analysis.
An alternative method would be to first chunk your documents, and then perform pair-wise comparisons of the chunks. The smaller chunks of text allow for a more focused and detailed analysis from the LLM, but then you run into the number of comparison tasks increasing with the square of the size of the documents, $O(n^2)$, so you suffer from quite a high tradeoff of computation time for the increase in detail.
1.2 A Faster Method ¶
To circumvent the tradeoff of document size versus detail of analysis, we need a method that utilizes the increased detail of chunking while avoiding the squared time complexity.
The method I propose isn't quite linear, but it stacks a couple of linear algorithms along with approximate nearest neighbors, which is $O(logn)$. Pretty good. It works like this:
Entity extraction
Chunk each document and pass each chunk to an LLM for entity extraction, paying attention to pronoun references to the same entity, aliases, and abbreviations. The name of each entity is extracted along with the surrounding two sentences of text for each reference so that the context of the entity within each document is understood. This is returned as JSON: {"entity_name": ..., "aliases": [...], "excerpts": [...]}Condensing and deduplication
Pass the entity JSON to another LLM for a condensing pass, combining like entities and redundant references.Indexing
Create an in-memory vector store (FAISS) from each entity JSON object, embedding the excerpts and maintaining the entity name and aliases as metadata.Entity pairing
Match the entities of each document by performing a similarity search using the excerpts of one document against the FAISS vector store of the other, done bidirectionally so that we have a full matching profile. We keep only the match with the highest similarity score, and a minimum similarity threshold is set so that needless pairings are avoided.Difference summary output
The matched pairs are sent to a final LLM to summarize the differences in how the entities are discussed in each document.
There is a glaring drawback to this method in that it relies on numerous calls to LLM APIs which in themselves can take a long time, detracting from efficiency gains of the algorithm itself. To mitigate, the implementation of this algorithm in Python employs threading wherever possible. While threading in Python isn't effective for all concurrency needs due to the global interpreter lock (GIL), for this type of IO-bound task -- making API calls and waiting for responses -- it performs very well.
1.3 Product-focused Development with Claude ¶
Along with making document comparison more manageable, an auxiliary goal of the project is to perform the develop the Flashpoint application in a way that would be suitable for an enterprise setting and to produce an enterprise-grade application. More often than not, application built with AI-first development end up being lackluster simply because their developers follow this process: have an idea, give it to Claude, say "go," tweak and adjust through additional prompts, and say "good enough" when their app is running. That's not how real application development happens. Developers in industry don't just have an idea, throw some spagetti code at it, and call it a day once it works; there has to be a business case, scoping and requirements, performance objectives, audience -- all of these things have to be determined before even starting to plan the architecture of the software.
To give us the best chance of creating a viable application, rather than an AI-first approach we need to take a business-first approach, identifying the value of the application and defining a true product.
1.3.1 Skill: Product Requirements Documents (PRD)¶
The first step towards creating a system for industry-level product development is to give Claude the tools it needs to create fully-fleshed out product definitions and scopes and to document them in PRDs. To do this, I created a new /prd-builder skill, making use of Claude's /skill-builder.
Using /skill-builder, I asked that Claude create a skill to generate PRD documents based on a loosely defined idea. The purpose of this skill would not just be to generate the PRD, but to guide the user through the process of fleshing out their idea, taking them through a questionnaire about the business value, scope, intended audience, success metrics, etc. This skill would then have Claude review the PRD draft with the user before finalizing it. The skill directory for /prd-builder also contains references for domain-specific questions to ask during the PRD drafting process -- an AI/ML product is quite different from a simple web application, and requires a completely different line of quesitoning, for example.
You can find the /prd-builder skill in the skills folder of this repository.
1.3.2 Skill: PRD to Design to Software¶
The next step is to take the PRD and translate it into software. This is where the /prd-to-product skill comes into play. The purpose of this skill is to, using the PRD as a baseline, help Claude in planning the execution of the application by asking the user guided questions to clarify the product and design choices before any coding takes place. The workflow for this skill is clarify -> design -> approve -> build, making sure the application that is built adheres to the intented product outlined in the PRD, and that the software is intentionally designed and implemented.
Before Claude generates any code, the user is guided through a questionnaire where they define coding languages, frameworks, design and architectural decisions, etc. This questionnaire is used not to directly generate code, but generate a design.md document, thoroghly outlining the design decisions, API structures, user interface, etc., of the application -- i.e. thoroughly and intentionally planning the application before any code is written. After the drafting and approval of the design documentation, this skill prompts Claude to start building the application, enforcing strict adherance to the design document, and in the case the something needs to change in the course of development, that the changes are percolated back to the design document, depending on developer approval.
prd-to-product too can be found in the skills folder of this repository.
2. Flashpoint ¶
Given the extensive documentation of each the entity extraction and comparison components of Flashpoint, I feel little need to revisit the purpose or high-level execution plan for this approach to document comparison. Rather, I'll focus here on the performance and evaluation of each component and provide code examples of how they work.
I will, however, provide a brief overview of design decisions that you can see reflected in the design.md documents contained in each module. Both of these modules are developed in python 3.10+, the native language of AI. One might expect the framework of choice for the LLM calls to be LangChain; however, given the simplicity of the LLM pipelines involved, LiteLLM is a more user-friendly and maintainable approach, due to its simpler API. LangChain does find use, however, in the comparison module, given its dependencies on FAISS databases, embeddings, and more complex LLM orchestration.
For the sake of my own frugality, we'll just use gpt-4o-mini for these examples, but the APIs allow for the substitution of any LLM.
2.1 Entity Extraction ¶
The entity extraction module is relatively straight forward: extract the text from each document, chunk it, pass the chunks through an entity extraction prompt, and then pass the output through a prompt for condensation and deduplication.
Let's set up our environment.
import os
import sys
import pickle
import time
from utils import timer
from entity_extraction.entity_extractor import extract_entities
import dotenv
dotenv.load_dotenv('webapp/backend/.env')
True
DATA_PATH = './data'
# decorate extraction module for timing
timed_extract_entities = timer(extract_entities)
One of the main performance metrics we want to optimize the run time and scalability of the entity extraction process. To gauge this, we'll perform entity extraction on first a one-page demo document that I had Claude generate, and then we'll have it extract the entities of the North Carolina Drivers Manual, which is 108 pages long.
# timed results of 1-page document
results_1 = timed_extract_entities(
pdf_path=os.path.join(DATA_PATH, 'millfield_sustainability_plan.pdf'),
model="gpt-4o-mini"
)
Extracting document text... Runtime: 19.9090s
# timed results of 108-page document
results_108 = timed_extract_entities(
pdf_path=os.path.join(DATA_PATH, 'nc_dm.pdf'),
model="gpt-4o-mini"
)
Extracting document text... Runtime: 217.4075s
The extraction from a single-page document took about 20 seconds, and for the 108-page document 217 seconds. That's a rate of 20 seconds/page for one page, and only 2 seconds per page for 108 pages, showing that the extraction is scaling extremely well, performing more efficiently on documents with more pages.
Let's look an example of the results.
results_108[:10]
[{'name': 'carbon monoxide',
'type': 'concept',
'aliases': [],
'excerpts': ['Carbon monoxide is an invisible gas that has no smell, taste or color but is poisonous, even deadly.',
'A leak in the exhaust system can allow poisonous carbon monoxide gas to enter the passenger compartment of the vehicle.']},
{'name': 'Provisional Licensee',
'type': 'concept',
'aliases': [],
'excerpts': ['Drivers under age 18 are provisional licensees.',
'Provisional Licensee (under age 18)']},
{'name': 'traffic signals',
'type': 'concept',
'aliases': [],
'excerpts': ['At intersections controlled by ordinary traffic signals, pedestrians must obey the same signals as drivers traveling in the same direction.',
'Through traffic and traffic turning left onto or off the interchange, is controlled by a single set of traffic signals.',
'Traffic signals, signs and pavement markings are used for traffic control to provide a smooth, orderly flow of traffic.']},
{'name': 'Social Security Card',
'type': 'document',
'aliases': [],
'excerpts': ['1. Social Security Card', 'Social Security Card']},
{'name': 'Flashing Yellow Arrow',
'type': 'concept',
'aliases': [],
'excerpts': ['Flashing Yellow Arrow: Turns are allowed, but first they must yield to oncoming traffic and pedestrians.',
'Flashing Yellow Arrow: Turns are allowed, but first they must yield to oncoming traffic and pedestrians.']},
{'name': 'Wildlife',
'type': 'concept',
'aliases': [],
'excerpts': ['Wildlife-vehicle collisions are a leading cause of injury and death for many animal species and often put drivers and passengers at risk as well.',
'By understanding animal behaviors and adjusting our driving habits, we can help protect our wild neighbors.']},
{'name': 'two-second rule',
'type': 'concept',
'aliases': [],
'excerpts': ['Allow a safe distance between you and the vehicle in front of you (the “two-second rule”).',
'The “two-second rule” says that you should allow two seconds between the time the vehicle ahead of you passes a given point and the time your vehicle reaches the same point.']},
{'name': 'DD-214',
'type': 'document',
'aliases': [],
'excerpts': ['6. DD-214 with full Social Security number',
'Veterans who are interested in applying for the designation should take their DD-214 discharge form to their local driver license office to show they have been honorably discharged.']},
{'name': 'I-94',
'type': 'document',
'aliases': [],
'excerpts': ['I-512L Authorization for Parole of an Alien into the U.S. w/ supporting immigration documents (I-551, I-766 or I-94)',
'7. I-94 Arrival/Departure Records-Electronic I-94']},
{'name': 'U.S. Passport',
'type': 'document',
'aliases': [],
'excerpts': ['U.S. Passport or U.S. Passport Card',
'9. U.S. Passport or U.S. Passport Card']}]
You can see that this algorithm is extracting the entity's name, aliases, type, and excerpts that are used for the pairing and comparison phases of the Flashpoint application.
2.2 Document Comparison ¶
The comparison module is a bit more complex of a process: pairing entities from each document, embedding, cross-comparing entities, and finally filtering the results for relevant comparisons. This module also allows for the user to specify a "pre_context," this is a focus area over which the comparison should be applied.
import pandas as pd
from compare.document_comparator import compare_documents
# NC and VA driver manuals
doc1 = os.path.join(DATA_PATH, 'nc_dm.pdf')
doc2 = os.path.join(DATA_PATH, 'va_dm.pdf')
# get timed comparison results
timed_compare_documents = timer(compare_documents)
result = timed_compare_documents(
pdf_path_1=doc1,
pdf_path_2=doc2,
extraction_model="gpt-4o",
comparison_model="gpt-4o",
embedding_model="text-embedding-3-small",
similarity_threshold=0.4
)
df = pd.DataFrame(result)
Extracting entities from both documents... Extracting document text... Extracting document text... Document 1: 232 entities found. Document 2: 83 entities found. Building FAISS indices... Matching entities... 50 matched pairs, 182 unmatched from document 1, 33 unmatched from document 2. Running comparison on 50 pairs... Filtering results... Done. 241 results returned. Runtime: 686.4570s
df.sort_values('similarity_score', ascending=False).head(15)
| document_1_entity_name | document_2_entity_name | similarity_score | document_1_context | document_2_context | difference_summary | consistency | |
|---|---|---|---|---|---|---|---|
| 25 | Crossbuck Sign | Railroad Crossbuck | 0.830945 | Document 1 describes the Crossbuck Sign as an ... | Document 2 refers to the Railroad Crossbuck as... | The key difference is that Document 1 focuses ... | entailment |
| 12 | Chapter 5 | Section 2: Signals, Signs and Pavement Markings | 0.727874 | Document 1 discusses traffic signs as part of ... | Document 2 addresses traffic signs within a se... | Both documents cover traffic signs within the ... | agreement |
| 23 | National Highway Traffic Safety Administration | National Highway Traffic Safety Administration | 0.709413 | Document 1 mentions the National Highway Traff... | Document 2 refers to the National Highway Traf... | The documents discuss the NHTSA in different c... | neutral |
| 20 | DonateLifeNC.org | www.donatelifevirginia.org | 0.693687 | Document 1 mentions DonateLifeNC.org as a sour... | Document 2 refers to donatelifevirginia.org as... | The key difference is that Document 2 provides... | neutral |
| 6 | Multi-Lane Roundabout | Roundabout | 0.693181 | Document 1 mentions a multi-lane roundabout bu... | Document 2 describes roundabouts as circular i... | Document 2 provides specific information about... | neutral |
| 10 | yield sign | Yield line | 0.691203 | Document 1 describes the yield sign as a trian... | Document 2 explains that a yield line consists... | The key difference is that document 1 focuses ... | neutral |
| 9 | Regulatory Signs | Traffic Signs | 0.685555 | Document 1 categorizes traffic signs into thre... | Document 2 explains that traffic signs, along ... | Document 1 focuses on the classification of tr... | neutral |
| 3 | Flashing Yellow Arrow | Flashing Arrow Boards | 0.671592 | Document 1 describes the flashing yellow arrow... | Document 2 discusses flashing arrow boards, wh... | The key difference is that document 1 focuses ... | neutral |
| 24 | Anti-lock Braking System | Antilock Brakes | 0.669329 | Document 1 explains that the Anti-lock Braking... | Document 2 emphasizes the importance of unders... | Document 1 provides specific guidance on how t... | neutral |
| 5 | Hybrid Beacons | Pedestrian Hybrid Beacons | 0.649782 | Document 1 describes hybrid beacons as traffic... | Document 2 explains pedestrian hybrid beacons ... | The key difference is that document 1 provides... | neutral |
| 7 | NO-ZONES | No-Zones | 0.631592 | Document 1 discusses the concept of NO-ZONES a... | Document 2 refers to No-Zones as specific area... | Both documents agree on the concept of No-Zone... | entailment |
| 15 | Canada | Canada | 0.611910 | Document 1 discusses Canada in the context of ... | Document 2 refers to Canada in the context of ... | The key difference between the documents is th... | neutral |
| 19 | Division | Virginia Department of Motor Vehicles | 0.590269 | The North Carolina Division of Motor Vehicles ... | The Virginia Department of Motor Vehicles (DMV... | The key difference between the documents is th... | neutral |
| 8 | Ramp Meter Traffic Signals | Traffic signals | 0.576850 | Document 1 describes ramp meter traffic signal... | Document 2 provides a general overview of traf... | The key difference is that document 1 focuses ... | neutral |
| 11 | CDL | Commercial Driver License Manual (DMV 60V and ... | 0.556221 | Document 1 mentions that the CDL handbook, alo... | Document 2 specifies that the Commercial Drive... | Both documents discuss the availability of CDL... | neutral |
This example shows how Flashpoint performs on two larger documents (100+ pages), for a more comprehensive evaluation of its results, you can look at a more standardized evaluation in notebooks/eval_sustainability_test.ipynb.
3. Serving Results via Wep App ¶
To serve the results of the Flashpoint document comparison algortithm, I wrapped it in an API and served it via a simple web application, coded with Claude. Like the extraction and comparison modules, I started the development of the web application using the /prd-builder and /prd-to-product skills.
In an effort to keep things "production grade," I specified that this application should be contianerized with seperate containers for the front-end and back-end for independent scalability, orchestrating with Docker Compose. This could be deployed easily through Amazon ECS using Fargate for autoscaling with application load balancers (ALBs).
Below is a screenshot of the landing page for the Flashpoint web app:
And here you can see the results displayed, downloadable as CSV data, with the option to re-tune the analysis settings without having to re-upload the documents.
Conclusion ¶
This was a truly enlightening project, not just from the perspective of developing a new and efficient techinique for comparing large documents, but also from the standpoint of creating a reliable, business and product-focused method for AI-first development using Claude and skills.
I think there are a few areas for improvements, particularly in the entity extraction module. It does a good job, but I feel that it could benefit from perhaps a multi-pass approach to increase its thoroughness, an easy enough adjustment with Claude. In a true industry setting, I would have also done the "rapid prototyping" approach for the development of the web app, creating ~5 alternatives and picking the one that satisfied the business user's objectives most completely.