Operations

Health Checks

Pastebox exposes two health endpoints:

GET /healthz -- Liveness

Returns 200 OK if the process is alive. Does not check dependencies.

{ "status": "ok" }

Use this for load balancer liveness probes and PM2/systemd health checks.

GET /readyz -- Readiness

Executes SELECT 1 against PostgreSQL to verify database connectivity.

Success (200):

{ "status": "ok", "db": "connected" }

Failure (503):

{ "status": "error", "db": "disconnected" }

Use this for load balancer readiness probes and deployment verification. The Ansible playbook checks both endpoints after deployment.

PM2 Management

The production setup runs two PM2 processes defined in infra/pm2/ecosystem.config.js:

Process	Description
pastebox-api	Fastify HTTP server (2 cluster instances)
pastebox-worker	Background workers (1 fork instance)

Common commands

# Start all processes
pm2 start /opt/pastebox/infra/pm2/ecosystem.config.js

# Stop all
pm2 stop all

# Restart all
pm2 restart all

# Zero-downtime reload (API only, cluster mode)
pm2 reload pastebox-api

# Restart workers (after code changes)
pm2 restart pastebox-worker

# View status
pm2 status

# View real-time logs
pm2 logs
pm2 logs pastebox-api
pm2 logs pastebox-worker

# Interactive monitor (CPU, memory, logs)
pm2 monit

# Save current process list for auto-start on reboot
pm2 save

# Generate systemd startup script
pm2 startup

Deployment with PM2

After pulling new code and rebuilding:

cd /opt/pastebox
git pull
pnpm install --frozen-lockfile
pnpm build
pnpm db:migrate
pm2 reload pastebox-api
pm2 restart pastebox-worker

Log Locations

Log	Path
API stdout	/var/log/pastebox/api-out.log
API stderr	/var/log/pastebox/api-error.log
Worker stdout	/var/log/pastebox/worker-out.log
Worker stderr	/var/log/pastebox/worker-error.log
Nginx access	/var/log/nginx/access.log
Nginx error	/var/log/nginx/error.log
Certbot renewal	/var/log/pastebox/certbot-renew.log
Backup	journalctl -u pastebox-backup.service

API and worker logs are JSON (Pino format in production). Use pino-pretty for human-readable output:

cat /var/log/pastebox/api-out.log | pnpm --filter @pastebox/api exec pino-pretty

Or use PM2's built-in log viewer:

pm2 logs pastebox-api --lines 100

Sensitive headers (Authorization, X-Api-Key) are redacted from request logs.

Database Migrations

Migrations are managed by drizzle-kit.

Generate a migration from schema changes

cd /opt/pastebox
pnpm db:generate
# Creates a new SQL migration file in apps/api/drizzle/

Run pending migrations

pnpm db:migrate

Push schema directly (development only)

Pushes the current Drizzle schema to the database without generating migration files. Useful for rapid iteration in development:

pnpm db:push

Drizzle config

The drizzle-kit configuration is at apps/api/drizzle.config.ts. It reads DATABASE_URL from the environment and uses the schema defined in apps/api/src/db/schema/index.ts. Migration files are output to apps/api/drizzle/.

Backup and Restore

Automated backups

Daily backups run at 03:00 UTC via a systemd timer (pastebox-backup.timer). The backup script (infra/restic/backup.sh) performs:

1. pg_dump of the pastebox database in custom format
2. restic backup of the dump and /opt/pastebox/.env
3. restic forget --prune with retention: 7 daily, 4 weekly, 6 monthly

Manual backup

sudo systemctl start pastebox-backup.service
sudo journalctl -u pastebox-backup.service -f

Verify backup integrity

restic -r /var/backups/pastebox check
restic -r /var/backups/pastebox snapshots --tag pastebox

Restore procedure

# 1. List available snapshots
restic -r /var/backups/pastebox snapshots --tag pastebox

# 2. Restore to a temporary directory
restic -r /var/backups/pastebox restore latest --tag pastebox --target /tmp/restore

# 3. Stop the application
pm2 stop all

# 4. Restore the database
pg_restore -U pastebox -d pastebox -c /tmp/restore/tmp/pastebox-pgdump/pastebox.dump

# 5. Restore .env if needed
cp /tmp/restore/opt/pastebox/.env /opt/pastebox/.env

# 6. Restart the application
pm2 start all

# 7. Verify
curl http://localhost:3000/readyz

# 8. Clean up
rm -rf /tmp/restore

Critical: The .env file contains MASTER_KEY. Without it, all Mode A pastes are permanently unrecoverable. Ensure .env is included in backups and stored securely.

Monitoring and Alerting

Recommended checks

Check	Method	Frequency	Alert condition
API liveness	GET /healthz	30 s	Non-200 for 2+ consecutive checks
API readiness	GET /readyz	60 s	Non-200 (indicates DB connectivity loss)
PM2 process status	pm2 jlist	60 s	Any process in errored or stopped state
Disk usage	Standard	5 min	> 85% on data or log partitions
PostgreSQL connections	pg_stat_activity	60 s	Near max_connections limit
Webhook delivery failures	Query webhook_deliveries	5 min	Rising count of status = 'failed'
Backup freshness	restic snapshots --latest 1	Daily	Latest snapshot older than 26 hours

PM2 monitoring

PM2 can report metrics to external services:

# Built-in web dashboard
pm2 plus

# Or export metrics for Prometheus
# Install pm2-prometheus-exporter module
pm2 install pm2-prometheus-exporter

Log-based alerting

Since logs are structured JSON (Pino), they can be ingested by log aggregation tools (Loki, ELK, Datadog) and used for alerting on:

• Error-level log entries ("level":50)
• Repeated webhook delivery failures
• Database connection errors
• Rate limit hits (potential abuse)

TTL Cleanup Worker

The TtlCleanupWorker runs inside the worker process and handles automatic paste expiration.

Behavior:

• Polls every WORKER_TTL_CLEANUP_INTERVAL_MS (default: 60 seconds).
• Deletes all rows from pastes where expires_at IS NOT NULL AND expires_at < NOW().
• Also cleans up expired, unused Telegram linking tokens.
• Logs the count of deleted pastes and tokens per cycle (only when count > 0).

Timing: A paste with ttlSeconds: 3600 may persist up to ~60 seconds past its expiry time, depending on the cleanup interval. This is acceptable for a pastebin use case.

Scaling: Only one worker instance should run the TTL cleanup to avoid duplicate deletes. The PM2 config enforces this with instances: 1, exec_mode: 'fork'.

Webhook Delivery Worker

The WebhookDeliveryWorker processes the webhook_deliveries queue.

Behavior:

• Polls every WORKER_POLL_INTERVAL_MS (default: 5 seconds).
• Picks up to 10 deliveries per tick where status IN ('pending', 'retrying') AND next_retry_at <= NOW().
• For each delivery, looks up the webhook endpoint and signs the payload with HMAC-SHA256.
• Sends an HTTP POST with a 10-second timeout.
• On success (2xx response): marks as delivered.
• On failure (non-2xx or network error): retries with exponential backoff.
• If the webhook endpoint has been deactivated, the delivery is marked as cancelled.

Exponential backoff schedule:

Attempt	Backoff delay	Cumulative wait
1	20 s	20 s
2	40 s	1 min
3	80 s	~2.3 min
4	160 s	~5 min
5 (final)	Marked failed	--

Formula: 2^attempts * 10,000 ms

After WORKER_WEBHOOK_MAX_RETRIES (default: 5) failed attempts, the delivery is permanently marked as failed. Response status codes and truncated response bodies (max 1,000 chars) are stored for debugging.

Webhook HTTP headers sent:

Header	Value
Content-Type	application/json
X-Pastebox-Event	Event name (e.g., paste.create)
X-Pastebox-Signature-256	sha256=<hex-hmac>
X-Pastebox-Delivery	Delivery ID

The Telegram notification worker (TelegramWorker) follows the same polling, retry, and backoff pattern but delivers via the Telegram Bot API instead of HTTP webhooks.