Why testing matters more when AI agents write the code

Leavely has a function that calculates the number of leave days between two dates. For someone who works Monday through Friday, a full week off counts as five days of leave, provided there are no public holidays. Writing a test for that case is straightforward. The test expects five and the function returns five. For that case, the calculation is correct.

If Wednesday is a public holiday, the result should be four. If the holiday falls on Saturday, we must not subtract a day we've already excluded. For a company that works Monday through Thursday, Friday shouldn't count either. Taking only the morning of a working day should count as half a day.

Leavely's calculation tests cover these cases. When we ask an agent to change the function, those tests make the rules it needs to preserve explicit. A short request like "calculate leave days correctly" contains several decisions that need to be made separately.

This is why I think testing becomes more important as we delegate coding to AI agents. We also delegate some implementation decisions. An agent might choose a helper function, combine conditions, or suggest a shorter calculation. We still need to determine whether the change preserves the behavior we expect from the product.

A short, readable function that follows the language's conventions makes that review easier. But code built on a wrong assumption about the working week can have all those qualities.

Tests give the agent feedback

Suppose an agent simplifies the calculation by counting working days first, then subtracting every public holiday in the date range. For a period running from Monday through Sunday, it counts five working days. Saturday is a public holiday, so it subtracts another day and returns four.

The test for a week without holidays still passes. The test that includes a Saturday holiday shows that the function returned four when it should have returned five. When the agent runs that test, it gets concrete feedback about the behavior its change broke. It can correct the calculation and run the tests again.

That loop can happen before a human starts reviewing the code. The agent has criteria it can check while it works. It can catch some of the errors we've defined without waiting for us to notice them and explain them again. Asking it to run the tests is part of the task for this reason. A test file in the repository doesn't tell us whether the agent used its results during the work.

I see the value of engineering experience in recognizing which situations need testing. What we call an "edge case" can be an ordinary working day for someone else. A function that assumes a Monday-to-Friday schedule will exclude Sunday even when it is part of an employee's working week.

Finding these cases sometimes means stepping away from the code and thinking about the people using the product. Work schedules, public holidays, and half-day rules determine the calculation's inputs. We also need to decide what happens when someone selects a half-day on a day they don't work. These are decisions to settle before writing the loop.

Trying to express an expectation as a test can reveal questions we haven't answered yet. If we can't say what the result should be, we may not have decided how the product should behave in that situation. An agent producing a number doesn't resolve that uncertainty for us.

When the agent writes the tests too

We can ask the agent to write the tests as well. It can turn rules into scenarios, suggest cases we've overlooked, and prepare test data. We need to pay attention to where the expected results come from.

If the agent treats the current implementation as correct, it can carry the same mistaken assumption into both the code and the test. Expecting four because the function returns four gives us no independent reason to trust the calculation. In the leave example, we can explain why we expect five by referring to the company's work schedule and the fact that Saturday was already excluded.

When reviewing tests written by an agent, we need to look beyond their names and passing results. What data does the test set up, what does it assert, and why is that the expected outcome? This deserves particular attention when the agent changes an existing test. The product rule may have changed. But changing an expectation solely because the new code returns something different can mean accepting the bug we wanted to catch.

We can also run a small experiment to check whether a test detects the error we care about. Temporarily change the calculation so it subtracts a Saturday holiday twice. If the test meant to cover that case still passes, look again at its setup and assertions. This gives us evidence that the test responds to a specific error, beyond knowing that it exists.

We also need to understand what each test covers. A calculation test that returns four days for the right holiday list doesn't establish that the application selects and passes in the right list. A correct function given another employee's holiday calendar can still produce the wrong leave duration. We need to check that connection by testing the relevant parts together. During human review, we still have to consider which expectations remain untested.

We can include this verification in the task we give the agent. Alongside "simplify this calculation," we can specify the work schedules, overlapping holidays, and half-day behavior it must preserve. We can provide the commands for running the relevant tests and ask it to report which passed, which failed, and which it couldn't run. That gives us both the code and the verification results when we assess the work.

We won't anticipate every possibility. But once we discover an assumption that produces an incorrect leave duration, we can make it something we check again in later changes. Our knowledge that companies can have different working weeks becomes an explicit expectation in the tests, rather than staying in a single conversation.

As we delegate code to agents, the time we spend defining correct behavior helps them check their own changes. The next agent can run those expectations and get feedback on its work. We can begin our review knowing which behaviors it has checked.

When you ask an agent to "preserve the existing behavior," which behaviors can it check on its own using your tests?