Adding Observability stack to a Web Application

Adding Observability stack to a Web Application

Why Do We Need Observability?

One of the project assignments for DataTalks Club’s course implementing observability with an existing project/repo called Order Tracker.

Users can create orders, update their status, and retrieve existing orders.

You've already learned how to:

  • Build the application: Use FastAPI to implement API endpoints.
  • Test it: Use pytest to check that the code behaves correctly.
  • Containerize it: Use Docker to package the application.
  • Deploy it: Make it accessible to users.
  • Check its health: Use an endpoint such as /healthz to confirm the server is responding.

But once people start using your application, new questions arise.

  • How many requests is the application receiving?
  • Are users encountering errors?
  • Which API endpoints are slow?
  • What happened inside the application when a request failed?

A healthcheck cannot answer all these questions. It can tell you the server is responding, but not whether users are successfully completing their tasks.

This is where observability comes in.

Observability is the ability to understand what is happening inside a running application by examining the information it produces.

The Three Main Types of Observability Data

Applications typically produce three kinds of information: metrics, logs, and traces.

Type
What it tells you
Example
Metrics
How the application performs over time
20 requests per second
Logs
What happened at a particular moment
"Order 123 not found"
Traces
What happened during one request
Request entered FastAPI, ran a database query, then returned 404

Metrics: Understanding patterns

A metric is a numerical measurement collected over time.

For example, Order Tracker might receive 100 requests in one minute, of which 10 return errors.

From these numbers, we can calculate:

  • Request rate: approximately 1.67 requests per second.
  • Error rate: 10% of requests.
  • Average response time: how long requests take to complete.

Metrics help identify patterns, such as a sudden rise in errors after deployment.

Logs: Understanding events

A log is a record of something that happened.

For example:

12:01:03  INFO   GET /orders/123
12:01:03  ERROR  Order 123 not found
12:01:03  INFO   Response status: 404

Logs provide details that aggregated metrics cannot.

Traces: Understanding individual requests

A trace follows one request as it moves through the application.

For example:

GET /orders/123
      |
      v
FastAPI receives request
      |
      v
Application looks up order
      |
      v
Order not found
      |
      v
Return HTTP 404

Each trace can contain smaller operations called spans.

A trace ID helps identify the request, while span IDs identify operations within that trace.

When configured correctly, logs can also carry trace IDs, allowing an engineer to connect a particular error message to the request that produced it.

Think of metrics as the big picture, logs as the details, and traces as the story of one request.

How Do We Collect This Information?

A web application does not automatically provide a complete monitoring system.

We need tools to collect, store, and display the information it produces.

For Order Tracker, we use five technologies:

Tool
Job
OpenTelemetry
Collects observability data from the application
OpenTelemetry Collector
Receives and routes that data
Prometheus
Stores and queries metrics
Loki
Stores and searches logs
Tempo
Stores and retrieves traces
Grafana
Displays and explores all three

These tools play different roles. OpenTelemetry is responsible for generating and transporting telemetry. Prometheus, Loki, and Tempo are specialized storage and query systems. Grafana provides the interface for examining the results.

Understanding the Architecture

Order Tracker remains a single FastAPI application.

We are not splitting it into microservices or changing its business logic.

Instead, we add a monitoring system alongside it.

           Order Tracker
             (FastAPI)
                 |
                 v
       OpenTelemetry Collector
                 |
        +--------+--------+
        |        |        |
        v        v        v
    Prometheus  Loki     Tempo
     (metrics)  (logs)  (traces)
        |        |        |
        +--------+--------+
                 |
                 v
              Grafana
             (dashboard)

This is a simplified diagram. Technically, Prometheus collects metrics by scraping an endpoint exposed by the Collector, while the Collector sends logs and traces to their respective backends.

Why use a Collector?

You could configure an application to send data directly to each storage system.

However, that means the application needs to know about several different monitoring tools.

The Collector acts as an intermediary:

  1. Order Tracker sends observability data to the Collector.
  2. The Collector receives and processes it.
  3. The Collector sends each type of data to the appropriate destination.

This separation makes it easier to change the monitoring infrastructure later without rewriting the application.

A useful analogy is a mail sorting center: the application sends its messages to one place, and the Collector directs them to the correct destinations.

Instrumentation: Teaching the Application What to Report

Instrumentation means adding the ability to record what an application is doing.

There are two common approaches.

Manual instrumentation

We explicitly write code to record particular events.

For example, we might record how many orders were created or how long a database operation took.

This gives us precise control, but requires additional application code.

Automatic instrumentation

Some frameworks can be instrumented automatically.

For FastAPI, OpenTelemetry can automatically capture information such as:

  • Incoming HTTP requests.
  • API route information.
  • HTTP response status codes.
  • Request duration.
  • Traces of supported operations.

Order Tracker uses automatic instrumentation.

Instead of changing every API endpoint, we launch the application with an OpenTelemetry wrapper:

uv run --no-sync opentelemetry-instrument uvicorn app.main:app --host 0.0.0.0 --port 8000

This looks complicated, but it is doing something straightforward.

  • v run starts the command using the project's Python environment.
  • opentelemetry-instrument enables automatic instrumentation.
  • vicorn starts the FastAPI server.
  • app.main:app identifies the FastAPI application.
  • -port 8000 tells the server where to listen.

The application still behaves as before. The difference is that its HTTP activity can now be observed.

Configuration: How the Application Finds the Collector

Remember environment variables from the earlier discussion of databases and deployment? They let us change how software behaves without changing its source code. OpenTelemetry uses environment variables for the same reason.

For example:

OTEL_SERVICE_NAME=order-tracker

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

OTEL_EXPORTER_OTLP_PROTOCOL=grpc

These settings tell OpenTelemetry three things.

Service name: Identify the application as order-tracker, so its data can be distinguished from other applications.

Endpoint: Send telemetry to the service called otel-collector, listening on port 4317.

Protocol: Use gRPC to communicate with the Collector.

Here, otel-collector is a Docker Compose service name. Docker's internal networking lets containers find one another by those names.

What is OTLP?

OTLP stands for OpenTelemetry Protocol. It is a standard format and transport mechanism for exchanging observability information. In this setup, the application sends OTLP data to the Collector using gRPC, a network communication technology.

Docker Compose: Running the Monitoring Stack

Previously, Docker Compose started the application and its supporting services.

We now extend the same Compose file to include the observability infrastructure.

The main services are:

Container
Responsibility
order-tracker
Runs the FastAPI application
otel-collector
Receives and routes telemetry
prometheus
Collects metrics from the Collector
loki
Receives logs from the Collector
tempo
Receives traces from the Collector
grafana
Provides dashboards and investigation tools

Instead of installing and launching six separate programs manually, Docker Compose coordinates them.

docker compose up --build -d --wait

Here:

  • p creates and starts the services.
  • -build rebuilds images that need building.
  • d runs the services in the background.
  • -wait waits until services are running or healthy, according to their configured checks.

What are Docker healthchecks doing here?

Just like the application's /healthz endpoint, supporting services can expose health or readiness endpoints. Compose can use those checks to wait until services are ready. However, merely starting a container does not prove that the entire monitoring pipeline works. For example, Grafana might be running even if Prometheus is not receiving metrics. We still need to verify the actual data flow.

Why bind ports to localhost?

For local development, published ports should generally be limited to 127.0.0.1. This makes services accessible from your own computer without deliberately exposing them on every network interface. Grafana can then be opened at:

http://127.0.0.1:3000

The example configuration uses admin / admin as development credentials. These should not be reused for a publicly accessible or production deployment.

Grafana: Turning Telemetry into a Dashboard

After the monitoring services are running, we need a way to understand the collected data. Grafana provides dashboards with charts and other visualizations. For Order Tracker, we'll create a dashboard called Order Tracker requests. It will initially have two charts.

Chart 1: Request rate

This answers:

How frequently are people making requests to each API endpoint?

For example, the chart might show:

API route
Requests per second
GET /orders
5
POST /orders
2
GET /orders/{id}
8

This helps us understand usage patterns and detect unexpected traffic changes.

Chart 2: Error rate

This answers:

How frequently are requests returning HTTP errors?

For example:

API route
Error responses per second
GET /orders/{id}
0.3
POST /orders
0.1

HTTP status codes beginning with 4 or 5 indicate errors, but they mean different things:

  • 4xx: The request could not be fulfilled because of a client-side condition, such as requesting a nonexistent order.
  • 5xx: The server failed to fulfill the request because of a server-side problem.

A 404 can be expected behavior rather than evidence of a software bug. It is still useful to monitor because an increase may reveal a broken link or incorrect frontend behavior.

Why exclude /healthz?

Docker Compose periodically requests /healthz to check whether the app is healthy.These requests are generated by the monitoring setup, not by actual users. If we include them in the usage charts, they may distort our understanding of real application activity. We therefore exclude /healthz from these dashboard queries.