Why Do We Need Observability?
One of the project assignments for DataTalks Club’s course implementing observability with an existing project/repo called Order Tracker.
Users can create orders, update their status, and retrieve existing orders.
You've already learned how to:
- Build the application: Use FastAPI to implement API endpoints.
- Test it: Use pytest to check that the code behaves correctly.
- Containerize it: Use Docker to package the application.
- Deploy it: Make it accessible to users.
- Check its health: Use an endpoint such as
/healthzto confirm the server is responding.
But once people start using your application, new questions arise.
- How many requests is the application receiving?
- Are users encountering errors?
- Which API endpoints are slow?
- What happened inside the application when a request failed?
A healthcheck cannot answer all these questions. It can tell you the server is responding, but not whether users are successfully completing their tasks.
This is where observability comes in.
Observability is the ability to understand what is happening inside a running application by examining the information it produces.
The Three Main Types of Observability Data
Applications typically produce three kinds of information: metrics, logs, and traces.
Type | What it tells you | Example |
Metrics | How the application performs over time | 20 requests per second |
Logs | What happened at a particular moment | "Order 123 not found" |
Traces | What happened during one request | Request entered FastAPI, ran a database query, then returned 404 |
Metrics: Understanding patterns
A metric is a numerical measurement collected over time.
For example, Order Tracker might receive 100 requests in one minute, of which 10 return errors.
From these numbers, we can calculate:
- Request rate: approximately 1.67 requests per second.
- Error rate: 10% of requests.
- Average response time: how long requests take to complete.
Metrics help identify patterns, such as a sudden rise in errors after deployment.
Logs: Understanding events
A log is a record of something that happened.
For example:
12:01:03 INFO GET /orders/123
12:01:03 ERROR Order 123 not found
12:01:03 INFO Response status: 404Logs provide details that aggregated metrics cannot.
Traces: Understanding individual requests
A trace follows one request as it moves through the application.
For example:
GET /orders/123
|
v
FastAPI receives request
|
v
Application looks up order
|
v
Order not found
|
v
Return HTTP 404Each trace can contain smaller operations called spans.
A trace ID helps identify the request, while span IDs identify operations within that trace.
When configured correctly, logs can also carry trace IDs, allowing an engineer to connect a particular error message to the request that produced it.
Think of metrics as the big picture, logs as the details, and traces as the story of one request.
How Do We Collect This Information?
A web application does not automatically provide a complete monitoring system.
We need tools to collect, store, and display the information it produces.
For Order Tracker, we use five technologies:
Tool | Job |
OpenTelemetry | Collects observability data from the application |
OpenTelemetry Collector | Receives and routes that data |
Prometheus | Stores and queries metrics |
Loki | Stores and searches logs |
Tempo | Stores and retrieves traces |
Grafana | Displays and explores all three |
These tools play different roles. OpenTelemetry is responsible for generating and transporting telemetry. Prometheus, Loki, and Tempo are specialized storage and query systems. Grafana provides the interface for examining the results.
Understanding the Architecture
Order Tracker remains a single FastAPI application.
We are not splitting it into microservices or changing its business logic.
Instead, we add a monitoring system alongside it.
Order Tracker
(FastAPI)
|
v
OpenTelemetry Collector
|
+--------+--------+
| | |
v v v
Prometheus Loki Tempo
(metrics) (logs) (traces)
| | |
+--------+--------+
|
v
Grafana
(dashboard)This is a simplified diagram. Technically, Prometheus collects metrics by scraping an endpoint exposed by the Collector, while the Collector sends logs and traces to their respective backends.
Why use a Collector?
You could configure an application to send data directly to each storage system.
However, that means the application needs to know about several different monitoring tools.
The Collector acts as an intermediary:
- Order Tracker sends observability data to the Collector.
- The Collector receives and processes it.
- The Collector sends each type of data to the appropriate destination.
This separation makes it easier to change the monitoring infrastructure later without rewriting the application.
A useful analogy is a mail sorting center: the application sends its messages to one place, and the Collector directs them to the correct destinations.
Instrumentation: Teaching the Application What to Report
Instrumentation means adding the ability to record what an application is doing.
There are two common approaches.
Manual instrumentation
We explicitly write code to record particular events.
For example, we might record how many orders were created or how long a database operation took.
This gives us precise control, but requires additional application code.
Automatic instrumentation
Some frameworks can be instrumented automatically.
For FastAPI, OpenTelemetry can automatically capture information such as:
- Incoming HTTP requests.
- API route information.
- HTTP response status codes.
- Request duration.
- Traces of supported operations.
Order Tracker uses automatic instrumentation.
Instead of changing every API endpoint, we launch the application with an OpenTelemetry wrapper:
uv run --no-sync opentelemetry-instrument uvicorn app.main:app --host 0.0.0.0 --port 8000This looks complicated, but it is doing something straightforward.
v runstarts the command using the project's Python environment.opentelemetry-instrumentenables automatic instrumentation.vicornstarts the FastAPI server.app.main:appidentifies the FastAPI application.-port 8000tells the server where to listen.
The application still behaves as before. The difference is that its HTTP activity can now be observed.
Configuration: How the Application Finds the Collector
Remember environment variables from the earlier discussion of databases and deployment? They let us change how software behaves without changing its source code. OpenTelemetry uses environment variables for the same reason.
For example:
OTEL_SERVICE_NAME=order-tracker
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpcThese settings tell OpenTelemetry three things.
Service name: Identify the application as order-tracker, so its data can be distinguished from other applications.
Endpoint: Send telemetry to the service called otel-collector, listening on port 4317.
Protocol: Use gRPC to communicate with the Collector.
Here, otel-collector is a Docker Compose service name. Docker's internal networking lets containers find one another by those names.
What is OTLP?
OTLP stands for OpenTelemetry Protocol. It is a standard format and transport mechanism for exchanging observability information. In this setup, the application sends OTLP data to the Collector using gRPC, a network communication technology.
Docker Compose: Running the Monitoring Stack
Previously, Docker Compose started the application and its supporting services.
We now extend the same Compose file to include the observability infrastructure.
The main services are:
Container | Responsibility |
order-tracker | Runs the FastAPI application |
otel-collector | Receives and routes telemetry |
prometheus | Collects metrics from the Collector |
loki | Receives logs from the Collector |
tempo | Receives traces from the Collector |
grafana | Provides dashboards and investigation tools |
Instead of installing and launching six separate programs manually, Docker Compose coordinates them.
docker compose up --build -d --waitHere:
pcreates and starts the services.-buildrebuilds images that need building.druns the services in the background.-waitwaits until services are running or healthy, according to their configured checks.
What are Docker healthchecks doing here?
Just like the application's /healthz endpoint, supporting services can expose health or readiness endpoints. Compose can use those checks to wait until services are ready. However, merely starting a container does not prove that the entire monitoring pipeline works. For example, Grafana might be running even if Prometheus is not receiving metrics. We still need to verify the actual data flow.
Why bind ports to localhost?
For local development, published ports should generally be limited to 127.0.0.1. This makes services accessible from your own computer without deliberately exposing them on every network interface. Grafana can then be opened at:
http://127.0.0.1:3000
The example configuration uses admin / admin as development credentials. These should not be reused for a publicly accessible or production deployment.
Grafana: Turning Telemetry into a Dashboard
After the monitoring services are running, we need a way to understand the collected data. Grafana provides dashboards with charts and other visualizations. For Order Tracker, we'll create a dashboard called Order Tracker requests. It will initially have two charts.
Chart 1: Request rate
This answers:
How frequently are people making requests to each API endpoint?
For example, the chart might show:
API route | Requests per second |
GET /orders | 5 |
POST /orders | 2 |
GET /orders/{id} | 8 |
This helps us understand usage patterns and detect unexpected traffic changes.
Chart 2: Error rate
This answers:
How frequently are requests returning HTTP errors?
For example:
API route | Error responses per second |
GET /orders/{id} | 0.3 |
POST /orders | 0.1 |
HTTP status codes beginning with 4 or 5 indicate errors, but they mean different things:
- 4xx: The request could not be fulfilled because of a client-side condition, such as requesting a nonexistent order.
- 5xx: The server failed to fulfill the request because of a server-side problem.
A 404 can be expected behavior rather than evidence of a software bug. It is still useful to monitor because an increase may reveal a broken link or incorrect frontend behavior.
Why exclude /healthz?
Docker Compose periodically requests /healthz to check whether the app is healthy.These requests are generated by the monitoring setup, not by actual users. If we include them in the usage charts, they may distort our understanding of real application activity. We therefore exclude /healthz from these dashboard queries.
