Logging

Logging is the practice of recording discrete, text-based events that happen within a software system. It provides a detailed, chronological trail of what an application did, making it the primary tool for debugging errors and auditing user actions after the fact.

Structured vs. Unstructured Logging

• Unstructured: 2023-10-27 10:00:00 INFO User 'alice' logged in from 192.168.1.1. Easy for humans to read, but hard to search programmatically.
• Structured: {"time": "2023-10-27T10:00:00Z", "level": "INFO", "user": "alice", "event": "login", "ip": "192.168.1.1"}. Written in JSON. Harder to read raw, but easily queried in log aggregators (e.g., "Find all failed logins where IP starts with 192"). Always use structured logging in modern systems.

Log Levels

Logs are categorized by severity so you can filter out noise during normal operation.

• DEBUG: Deep diagnostic info for developers. Usually disabled in production.
• INFO: Routine milestones (e.g., "Service started", "Order placed").
• WARN: Something unexpected happened, but the system recovered (e.g., "DB retry successful", "Disk at 85%").
• ERROR: A specific operation failed (e.g., "Failed to charge credit card"). Requires investigation.
• FATAL: The application is crashing and cannot recover.

Centralized Log Aggregation

In a distributed system, you cannot SSH into 100 different servers to grep text files. Logs must be asynchronously shipped from the application servers to a centralized indexing system (like the ELK stack: Elasticsearch, Logstash, Kibana, or Splunk). This pattern is often called log forwarding.

Logging is expensive. Writing to disk slows down the application (so log asynchronously), and storing terabytes of indexed text in Elasticsearch is costly. Therefore, you must balance verbosity with cost. A common tradeoff is to log at INFO level in production, but dynamically switch to DEBUG level for specific services only when actively investigating an incident.

• Always specify that logs should be structured (JSON) and centrally aggregated.
• Mention that logging should be asynchronous so writing a log doesn't block the critical path of an API request.
• Bring up PII (Personally Identifiable Information) - explicitly state that passwords, credit card numbers, and raw social security numbers must be masked or scrubbed before being written to logs.