Notes from Peer-Reviewing AI-Built Projects

As part of the AI Dev Tools Zoomcamp 2026 (course on building software with AI coding agents), we were required to peer review other students’ projects to evaluate if they had fulfilled the rubric/certain features required of each applicaton. In this course, each student had built a full-stack application, complete with a frontend, backend, database, tests, and CI/CD workflows.

Code generation now is cheap. But I think that means the bar for creating software is higher because expectations have changed for “working software” even at the first couple iterations.

But what does working actually mean?

Working ≠ not breaking

When you review a repo of code the presence of tests, documentation, and deployment pipelines can give the impression of engineering maturity. While reviewing I noticed some problems that were not immediately obvious from the applications themselves.

When a Test Becomes Wrong Without the Code Changing

One project had approximately 450 unit tests. Of these, 449 passed. The remaining test checked whether a historical sports-data snapshot had been recorded before November 1, 2026, instead of the day the test was run by another uer

It failed when I ran it on November 4. It is a simple test, and not a fatal flaw.

The application itself was functioning correctly. The problem was that the test assumed a particular relationship between the current date and a hardcoded timestamp. That assumption had been true when the test was written but has ceased to be true.

In other words, it is a time-dependent test failure. Instead of checking whether one snapshot preceded another, the test checked whether a timestamp preceded a fixed date.

The distinction is subtle but consequential. A test is supposed to provide evidence that the software behaves as intended. Yet here, the test's validity depended on circumstances outside the software - and these mistakes are easier to make when we heavily rely on AI to generate code.

When a Green CI/CD Pipeline Means Nothing Was Deployed

Another project had a CI/CD pipeline that appeared to run successfully. Yet the deployment script contained this condition:

if [ -z "$VPS_HOST" ]; then
  echo "Deploy secrets not set, skipping deploy"
  exit 0
fi

The exit 0 tells the system that the operation succeeded, even though no deployment occurred.

There is a reasonable motivation for this. Contributors working on forks or pull requests may not have access to deployment secrets, and we don't necessarily want every such contribution to trigger a deployment failure. But this conceals real deployment failures when credentials are supposedly not intentionally excluded (missing) from the main project. In other words thereis a discrepancy between the observable signal (a green check) and the underlying event (nothing was deployed). We often rely on automated checks precisely because we cannot manually inspect every operation. But what happens when the automation is designed to report success in the absence of the expected outcome?

Instead we could add a GitHub Actions warning annotation (::warning::) and a job summary whenever deployment is skipped due to missing secrets in this case. This keeps the pipeline green without penalizing contributors who lack deployment credentials, while making it more visible that no deployment actually occurred. Although the warning doesn't change the success status, it helps distinguish a successful pipeline run from a successful deployment.

3. When the AI Agent Doesn't Know What Its Author Knows

A third project was technically impressive: a detailed product specification, extensive tests, cloud deployment, and monitoring.

Its AGENTS.md, however, contained approximately five lines of instructions, mostly about package management and committing changes.

For context, AGENTS.md is supposed to provide coding agents with information about a project: its architecture, conventions, development commands, and constraints. Unlike a human collaborator, an agent starting a new session does not necessarily retain the context established in previous conversations.

The project also lacked documentation of an agent team, despite this being part of the course requirements.

I find this omission particularly interesting because it exposes an asymmetry between what the developer knows and what the agent is given.

A human developer might think the instructions are sufficient because they already understand the architecture. An agent can still produce functional code with minimal instructions, but that doesn't mean it has the necessary context to make consistent decisions across sessions.

How much of what we call agent autonomy is actually sustained by the developer repeatedly supplying missing context?

And if those corrections remain in chat rather than being incorporated into persistent documentation, are we really improving the development process?

]=

What AI changes, perhaps, is the speed and scale at which we can produce artifacts that look complete: hundreds of tests, elaborate CI/CD configurations, extensive documentation, and comprehensive self-assessments.

I don't think the solution is simply to demand more tests or more documentation. These projects already had plenty of both.

Generating working code is one thing. Establishing why we should trust that it works, and under what circumstances, is another.