Showing Taste in Judgment: An AI Workflow Case Study in Mailbox Analysis

There’s no shortage of opinions right now on how software should be built. Like most of us on X, I’m watching from the sidelines, testing out new workflows and exploring the ideas posted. There are already great discussions happening about this, but there is always room to explore further how we build the underlying skills and core judgment, and then communicate it, and get better at it in a way that outlasts any workflow.

In short, the solution I am betting on is a shift toward mainstream, detailed case studies, much like the one I share below, so that judgment can be displayed, evaluated, and improved in public. Consultants have been doing this for a while, and many senior software engineering roles already rely on it during hiring. But the days of showing just two or three screenshots and a simple link to previous work and a short description have come to an end.

Showing the outcome of a product in these times may not be enough. People used to immediately realize how much effort and thought you put into a project when you showed them a final walkthrough. Now, you are left open to being second-guessed on whether the project was actually well thought out. The person viewing your product is tasked with understanding the domain and putting in the effort to see if you have sound judgment, something they are unlikely to do, because it is so hard to test a person's judgment from an isolated result. Judgment needs to compound and form a predictable pattern so that others can truly believe you can implement the same pattern in their projects. Ultimately, this means we need to share parts of our thinking process that we are not usually familiar with, which can certainly feel a bit vulnerable, but it gives us an opportunity to communicate our judgment, open a window to the public, and allow for valuable feedback to sharpen and improve it.

Our case study is for an HVAC company based in the United Kingdom. They have been operating for five years installing and servicing heating and cooling equipment on businesses, real estate, and private property. They communicate with customers mainly through email and keep their data in spreadsheets. As the business grows, keeping track of customers has become harder over the years.

The owner has considered using a standard CRM featuring an email pipeline that syncs conversations and automatically populates customer names. However, the client bets on the fact that more can be done if we are able to analyze data within the email before it reaches a CRM.

To go deeper into what the client plans to understand from their mailbox, we have to understand how the business operates. The enterprise functions as one large continuous loop, stretching from generating work and receiving payments to reinvesting revenue into generating more work.

Within this operational cycle, every job follows a specific path from start to finish. The flow diagram below outlines the complete progression of how an incoming opportunity turns into a completed job.

The client bets on re-engaging customers with expiring warranties to secure repeat servicing. Although individual service visits rarely turn a profit on their own, profitability becomes achievable when bundling multiple jobs in the same neighborhood or pairing them with a nearby installation, which cuts down on travel time and transportation expenses.

After getting all this background on the business and the client's bet, we need to set a goal that lets us constantly check if we are actually moving toward it. Establishing this goal brings structure and clarity, sitting right in the sweet spot of not being too broad or too narrow, which gives us the flexibility to switch bets if the current one doesn't pan out.

So below is the goal.

It clearly defines what we plan to do, not too narrow, but not too broad either.

Our goal functions as a distinct unit within a system. My notes in Systems Thinking give me a way to break this down into three key concepts:

Our order of work is borrowed from Sam Bhagwat’s Principles of Building AI Agents: make it work, make it right, make it fast, a philosophy widely attributed to Kent Beck.

The last part (Make it useful) is my addition to Kent Beck's structure. Product discovery happens throughout the project, but giving it its own section makes room to look for opportunities where the extracted data can produce value. Similar parallels exist: some put 'Make it Useful' at the top, some at the end, and some treat it as continuous.

For this part, we divide the process into six stages and go through them one by one.

Preparing the data covers getting the export and storing it in a form we can manipulate and analyze.

So we ask, how do we get the client’s email without disrupting their day-to-day work?

We have several opportunities, each with their own pros and cons, which are mapped out in the opportunity tree below.

Asking for an .mbox file gave all the access we wanted and the freedom to store and manipulate the data at hand. This is the first of multiple .mbox files to be acquired.

Which language should I use?

I wanted to use a language I am comfortable with. I found no reason to pay the cost of switching languages. Python would also fit the import and processing work, but I just enjoy reading TypeScript, though I can't claim I write it anymore.

Choosing SQLite to store our data naturally leads to the next question, how should we design our database model? Drawing from Michael J. Hernandez’s Database Design for Mere Mortals, the process starts with a clear mission statement and objective, which aligns with our core extraction goals.

Database design is inherently iterative and refined continuously throughout the design phases, but our logical design phase is now complete after multiple feedback loops, ending up with the database diagram below.

In the filtering phase, we need to remove emails unrelated to our core goal. These filtering opportunities are apparent by simply reviewing the email archive.

The bigger question is the age-old debate of deterministic versus AI filters. While the modern consensus favors a hybrid approach, we still have to ask: where should we draw the line, and what parts of the task can AI actually take over from traditional deterministic filters?

Instead of letting that assumption bias our approach again, let's keep it front and center while we first evaluate what a purely deterministic check can achieve and how far it can take us. Specifically, deterministic checks can easily filter out labeled Gmail messages and routine transactional business emails.

Rather than adding a separate AI filter to clean out the remaining noise, I chose to let the extraction agent absorb it, trusting that it would only surface records when actual job data was present. We are not optimizing to eliminate all noise entirely; the goal is simply to extract relevant details, not to achieve zero noise.

Fast-forwarding to the results, letting the email extraction agent handle the remaining data paid off. While the agent was busy extracting relevant details, it indirectly categorized 11% of the emails as unrelated noise.

We can group the separate emails into threads using their unique X-GM-THRID headers found within the exported mbox data.

To gather enough context regarding the data we need to extract, we ask ourselves and the client, what specific fields does the business need to know about a past customer or job, and exactly where within the messages are they located?

Based on our findings, the required fields are buried across different parts of the thread, some reside in the email body, others are located within attachments, and a few must be inferred from the conversation.

To figure out how to approach this set of extractions, we first needed to determine what pattern to use, adapting to the reality that certain groups of extractions could be handled by deterministic logic while others required an agent. As Antonio Gulli’s Agentic Design Patterns highlights, we shouldn't treat these patterns as an either-or choice; instead, we approach them as modular blocks you build upon along the way. In our case, we ended up using all the patterns, starting from basic prompt chaining to parallelization, routing, and finally agent-directed execution.

Fundamentally, this aligns with Anthropic’s perspective on building effective agents, where a workflow is defined by paths set in code. We know the desired end result because the output and the broad sequence are known beforehand, while the model is left to handle the interpretation within each step.

At a high level, the proposed workflow is the following.

This orchestration can be done bare bones or with a framework, or even with a no-code tool.

I preferred using TypeScript since we already had the mailbox scripts built with it. LangChain (which also has a TypeScript package now) and no-code solutions could still work, but it gets a bit subjective at this point. I just really like Mastra for its trace viewers, evals UI, and their open-source philosophy. Plus, they put out great content for the community. Two of the key resources I used to understand agents more came from Sam Bhagwat's books, Principles of Building AI Agents and Patterns for Building AI Agents, and they also put out podcasts and workshops that are well worth checking out.

So, Mastra it is. Once we have a filtered golden set of grouped emails and threads, the first step Mastra handles is dealing with the attachments based on our proposed pattern in Figure 10.

I used Daytona for the sandbox, though several providers could have done the job. The server decodes the selected attachments from the email archive and uploads them through Daytona’s file API. Mastra Workspaces then gives the agent access to those files through the sandbox’s filesystem.

Reading the attachments inside the sandbox can be done in multiple ways. It is actually quite nuanced, as there are several tool options available, each tailored to different scenarios.

Luckily, Anthropic provides a PDF skill that I can hand off to the agent, letting it use its own judgment to pick the best package for any given PDF to read. This required pre-installing all the necessary libraries and CLI tools so the agent can call them.

Our analysis has three steps. We find evidence through job signals and lead-source clues, interpret what it supports, then record the result in the structured fields shown in Table 2.

The job signals and lead-source clues we look for in the first step are listed below.

Now the question becomes, do we need a dedicated agent for analysis, or can the agent that extracts these details and signals also interpret them and return the analysis?

The best way to choose between them is simply to try both and check which provides better results. It is best to decrease failure points. In the event that a single agent handles both correctly, it’s preferable to pick that approach.

I kept extraction and analysis in a single agent. Feedback loops and evals gave me enough confidence to go this path

That leaves a harder question: how do I tell whether the records are reliable enough to use?

Setting up evals helps us test how reliable our agent is. Is the extracted information correct, and does it meet the client’s expectations?

There are 3 ways we can do evals:

Fields like customer name, service address, quote number, and invoice number are good candidates to score using deterministic evaluations provided with grounded data.

Acquiring a grounded dataset requires the painstaking process of manually reviewing a subset of our mailbox and extracting the scored fields. I looked through 20 threads, extracted the fields, and created a dataset in Mastra so we can run multiple evaluations on these emails.

Each scorer compares the agent’s output against the manually reviewed answer, scoring a match as 1 and a mismatch as 0. Flexible matching rules accommodate minor variations (like a dropped middle name or alternate address formatting) while still catching actual errors. Correctly identifying the absence of a quote or invoice on a thread also earns a full score.

The first headline score looked encouraging at 0.909 for the customer name. But only eleven of the twenty conversations had a score. Six had errored out and three finished without one. While the average represented ten correct names out of those eleven scored examples, it completely obscured the nine missing results.

The first reaction should not be to try a better prompt or a stronger model. Instead, it is to follow a framework that allows us to work in a loop, particularly the loop below.

Many failures came from the evidence gates the coding agent kept creating in code. I had required exact quotations to keep the model grounded, but the rigid checks were discarding correct answers as well.

Switching between models provided this evidence.

I chose DeepSeek for the combination of observed performance and input cost. The short records it returned made output cost a smaller part of the decision.

After multiple iterations, the score averages fluctuated up and down. The table below shows how the results progressed from the lowest scores to the highest

Our LLM-as-a-Judge evaluation is used to measure the accuracy and completeness of the email content agent. The judge, which is a different model from DeepSeek (currently GPT 5.6 Luna), checks whether the job signal events described in Table 2 (such as quote acceptance, payment, and installation booking) are supported by the original emails and PDF attachments, whether their details are correct, and whether any events were missed.

I applied the same failure-driven loop to the judge. Its first pass sometimes returned no events, making the extractor look wrong even when it had found something real. Later failures exposed disagreements in the rules.

Including a human in the loop requires us to streamline the process of giving feedback between the domain expert and the person developing the system, with as little friction as possible. We focus this approach on our riskiest agent, the reconciliation agent. This is a new agent, separate from our email content agent, called at a different place, and used for a completely different problem.

A customer may appear under different names or email addresses, and a job may have a quote in one conversation and a booking in another. Matching those records is a completely different decision from extraction our email content agent does and much more riskier in a sense a wrong merge can incorrectly combine two people's histories.

Because of this, the reconciliation agent lacks access to update the database directly and only suggests possible matching jobs or customers for review. We initially scaled back its role because we lacked enough reviewed examples to establish its accuracy, but the workflow below gives us a way to build that evidence while keeping people firmly responsible for those final decisions.

The AI finds potentially related records and presents them in a dashboard alongside its reasons. A domain expert then decides whether the records describe the same customer or job, should remain separate, or need further investigation.

Each decision is saved with the exact records and evidence the reviewer saw. This creates a dataset for evaluating future reconciliation decisions, including cases where similar looking records should stay separate. We use data to refine the system and test it against held out cases. Once we have enough audit history and evaluations, we can use this human judgment to automatically notice and merge similar records automatically.

We have reached the stage where we can scale this workflow to handle all relevant email threads we can get. That means being able to handle all email extraction, attachment extractions, and analysis.

To find opportunities in scaling, we need to push our current minimal workflow and see where it breaks. In doing so, we found three key opportunities:

This part of the work reminds me of The Goal by Eliyahu M. Goldratt and Jeff Cox, where systems thinking plays a big role in implementing these types of tasks. It teaches us how we should treat workflow optimization the same way we treat optimizing factories. The book highlights scenarios where over-optimizing a single part of a system simply creates a bottleneck for the next step. Instead, we need to view optimization as orchestrating multiple moving parts and treat the entire system as a whole.

The same reasoning applies to our Mastra workflow. Increasing the number of parallel threads allows more tasks to request sandbox execution at once, but it doesn't automatically increase how many tasks the sandboxes can actually process. When those requests outpace available capacity, work starts backing up or competing for resources, meaning we might have more tasks running concurrently even while individual jobs take longer to finish.

Because of this, we need to tune our thread concurrency to match the actual limits of our sandbox pool. This balance depends on the total number of sandboxes, the resources allocated to each one, and the heavy demands of each individual task. That doesn't strictly mean a one-to-one ratio of threads to sandboxes. Instead, it means opening up just enough parallel work to fully utilize our available capacity, while holding any excess work in a queue until resources free up. At the end of the day, our true measure of success is how much total work the entire workflow can push through reliably over time.

Once we dial in the knobs for concurrency and the number of sandboxes to an optimal level, we see another opportunity to cut down the time it takes to read attachments from the sandboxes, based on what we discovered in the traces

Caching pays off especially on repeated runs as we tweak the prompts. I might change the email prompt several times without changing the attachments, and re-reading every unchanged PDF makes that feedback loop unnecessarily slow. It also pays off by lowering compute costs for the sandboxes. Ultimately, there are various caching opportunities we can leverage throughout the entire process of extracting content from attachments

If I had to map out this whole system in one go, it might look something like this, at least, as best as I can picture it.

You’ve got Jev feeding attachments into a cluster of Daytona sandboxes, and DeepSeek turning that raw evidence into clean records. Throw in our scorers, the signal judge, and human review, and you've got quality control keeping everything grounded. Thanks to the anatomy skill we can generate this factory floor view.