Logging Standards

Owner: Anchor MSP Operations Lead Last reviewed: 2026-05-24

Purpose

Define logging standards for all systems under Anchor managed production. Consistent, structured logging enables effective monitoring, debugging, alerting, and forensic analysis.

Scope

All applications, services, and infrastructure components running on systems managed by Anchor MSP. This includes application logs, system logs, access logs, and security logs.

Policy

Structured Log Format

1. All application logs must use structured JSON format. Unstructured plaintext logs are not acceptable for new systems. Legacy systems must have a parsing pipeline to convert to structured format.
2. Each log entry must be a single JSON object on one line (JSONL format). Multi-line log entries break log aggregation pipelines.

Required Fields

Every log entry must include the following fields:

Field	Type	Description	Example
timestamp	ISO 8601 string	Time the event occurred, in UTC	"2026-05-24T14:30:00.000Z"
level	string	Log level (see Log Levels below)	"ERROR"
service	string	Name of the service producing the log	"api-gateway"
message	string	Human-readable description of the event	"Database connection timeout"
trace_id	string	Distributed trace identifier for request correlation	"abc123def456"

Optional but recommended fields:

Field	Type	Description
host	string	Hostname of the server
environment	string	production, staging
user_id	string	Authenticated user identifier (if applicable)
request_id	string	Unique request identifier
error_code	string	Application-specific error code
duration_ms	number	Operation duration in milliseconds
metadata	object	Additional context specific to the event

Log Levels

Use the following log levels consistently across all services:

Level	Usage	Example
FATAL	The process is about to terminate due to an unrecoverable error.	Database connection pool exhausted, cannot recover.
ERROR	An operation failed and requires attention. The service continues running.	Failed to process payment. API call returned 500.
WARN	Something unexpected happened but the service handled it. May indicate a developing problem.	Retrying database connection (attempt 2 of 3). Disk usage at 85%.
INFO	Normal operational events. The service is doing what it should.	Server started on port 8080. User login successful. Backup completed.
DEBUG	Detailed information for diagnosing problems. Not enabled in production by default.	SQL query executed: SELECT ... Request payload: {...}.

Production log level: INFO is the default production log level. DEBUG may be temporarily enabled for troubleshooting but must be reverted after investigation.

Loki and Promtail Configuration

1. Promtail is the standard log shipping agent. It runs on every managed host and ships logs to Loki.
2. Promtail configuration must include:
   • job label matching the service name.
   • host label matching the hostname.
   • client label identifying the client/project.
3. Promtail scrapes log files from standard paths:
   • Application logs: /var/log/<service>/ or stdout (for containerized services).
   • System logs: /var/log/syslog, /var/log/auth.log.
   • Nginx/web server logs: /var/log/nginx/.
4. For containerized workloads, promtail uses Docker log driver integration or reads from /var/lib/docker/containers/.

Log Retention

Tier	Duration	Storage	Description
Hot	30 days	Loki (local storage)	Fully queryable via Grafana. Used for active monitoring and debugging.
Cold	90 days	Object storage (R2)	Compressed and archived. Queryable with additional effort. Used for compliance and forensic review.
Security logs	1 year	Object storage (R2)	Access logs, auth logs, and security event logs. Required for audit compliance.

Logs older than the retention period are automatically purged.

Sensitive Data

1. Logs must never contain: passwords, API keys, tokens, credit card numbers, or other secrets.
2. Personally identifiable information (PII) should be minimized. If PII must be logged, it should be masked or truncated (e.g., email: j***@example.com).
3. Request/response bodies should not be logged at INFO level. If needed for debugging at DEBUG level, sensitive fields must be redacted.

Log Alerting

1. ERROR and FATAL log entries feed into the alerting pipeline via Loki alerting rules.
2. Log-based alerts follow the same severity matrix as metric-based alerts. See Alert Severity Matrix.
3. Repeated identical errors should be deduplicated in alerts to avoid alert fatigue.

Exceptions

Legacy systems that cannot produce structured JSON logs must have a promtail pipeline stage that parses their log format into structured fields. The parsing configuration must be documented in the system's runbook.