Production Issue Debugging

A practical, step-by-step guide to identifying, investigating, troubleshooting, and resolving production issues in Spring Boot-based distributed microservices architectures.

1. Observability – Core Pillars

Observability helps understand the internal state of an application by analyzing its outputs and behaviour.

1. Logs – What happened?

  • Application events, exceptions, stack traces, errors, and business transactions.

2. Metrics – How frequently and how badly is it happening?

  • Request count, error rate, response time, CPU, memory, garbage collection, threads, and database connections.

3. Distributed Tracing – Where did the request spend its time, and which service failed?

  • Follow a request across API Gateway, microservices, databases, Kafka, and external APIs.

4. Profiling – What is consuming application resources?

  • CPU-intensive methods, memory allocation, thread contention, deadlocks, and JVM bottlenecks.

5. Events – What changed in the environment?

  • Application deployments, configuration changes, infrastructure changes, scaling events, and database migrations.

2. Correlation ID & Distributed Tracing

Objective: Track a single request as it travels across multiple microservices and identify where failures or delays occur.

What to implement:

1. Generate Correlation ID

  • Generate a unique Correlation ID at the API Gateway or first application entry point.

  • Pass it to every downstream microservice using the X-Correlation-ID HTTP header.

2. Include IDs in Application Logs

  • Use SLF4J + Logback + MDC.

  • MDC allows Correlation ID, Trace ID, and Span ID to be automatically included in application log entries.

3. Implement Distributed Tracing

  • Use Micrometer Tracing + OpenTelemetry to track requests across Spring Boot microservices.

  • Capture Trace ID and Span ID for individual operations.

  • Use Grafana Tempo, Jaeger, or Zipkin to visualize the complete request journey.

4. Propagate Tracing Context

  • For HTTP communication, propagate tracing information through HTTP headers.

  • For Kafka communication, propagate tracing information through Kafka message headers.

  • Use W3C Trace Context (traceparent) for standardized trace propagation.

Recommended technology combination:

  • Spring Boot: Micrometer Tracing + OpenTelemetry.

  • Application logging: SLF4J + Logback + MDC.

  • HTTP and Kafka: Automatic trace-context propagation through instrumentation.

  • Tracing visualization: Grafana Tempo, Jaeger, Zipkin.

Example request flow:

API Gateway → Order Service → Payment Service → Inventory Service → Database

A distributed tracing platform helps visualize the complete request journey, individual operations, and execution time.

Example log entry:

2026-11-02 10:15:30 INFO
service=order-service
correlationId=CORR-12345
traceId=abc123
spanId=span001
Order created successfully

3. Application Logging & Centralized Log Management

Objective: Collect application logs from multiple microservices in one centralized platform so that errors and request execution can be investigated easily.

What to implement:

1. Application Logging

  • Use SLF4J as the logging API and Logback as the default logging implementation in Spring Boot.

  • Configure appropriate log levels: TRACE, DEBUG, INFO, WARN, ERROR.

  • Implement structured JSON logging.

  • Include timestamp, service name, log level, Correlation ID, Trace ID, Span ID, and exception details.

2. Centralized Log Management

Choose one centralized logging solution based on your infrastructure:

  • ELK Stack: Elasticsearch + Logstash + Kibana.

  • Grafana Loki: Loki + Grafana Alloy + Grafana.

  • OpenSearch: OpenSearch + Data Prepper + OpenSearch Dashboards.

  • Splunk: Splunk Enterprise / Splunk Cloud.

  • AWS CloudWatch: CloudWatch Logs + CloudWatch Agent.

3. Log Collection

  • Use Grafana Alloy, Fluent Bit, or Logstash to collect and forward application logs.

  • Configure log retention, access control, and sensitive-data masking.

  • Search logs using Correlation ID, Trace ID, service name, or transaction ID.

Note: SLF4J + Logback generate application logs. Centralized logging platforms collect, store, and search logs from multiple applications.

4. Metrics & Application Monitoring

Objective: Monitor application health, performance, resource utilization, and service availability to identify performance degradation before it becomes a major production issue.

A. JVM Monitoring

What to monitor:

  • Heap and non-heap memory utilization.

  • Garbage collection frequency and duration.

  • CPU utilization.

  • Thread count, blocked threads, and deadlocks.

  • Memory pressure and JVM health.

Recommended combination: Spring Boot Actuator + Micrometer + Prometheus + Grafana.

Deep JVM investigation: Java Flight Recorder (JFR) + Java Mission Control (JMC).

APM alternatives: Dynatrace, Datadog APM, New Relic.

B. HTTP/API Monitoring

What to monitor:

  • Request rate and throughput.

  • HTTP 4xx/5xx error rates.

  • P95/P99 response latency.

  • API timeouts and endpoint-level performance.

Recommended combination: Micrometer + Prometheus + Grafana.

APM alternatives: Datadog APM, Dynatrace, New Relic.

C. Database Monitoring

What to monitor:

  • Active and idle database connections.

  • HikariCP connection pool usage.

  • Slow SQL queries and query execution plans.

  • Database locks and deadlocks.

  • Transaction latency and connection acquisition timeouts.

Application connection pool monitoring: HikariCP metrics + Micrometer + Prometheus + Grafana.

PostgreSQL – useful commands and statistics:

Check active and idle connections:

SELECT
    pid,
    usename,
    application_name,
    state,
    wait_event_type,
    wait_event,
    now() - state_change AS state_duration,
    query
FROM pg_stat_activity
WHERE datname = current_database()
ORDER BY state, state_duration DESC;

Count connections by state:

SELECT state, COUNT(*)
FROM pg_stat_activity
WHERE datname = current_database()
GROUP BY state;

Identify slow and expensive queries:

  • pg_stat_statements – query execution statistics.

  • EXPLAIN (ANALYZE, BUFFERS) – query execution plan and performance analysis.

  • log_min_duration_statement – log queries exceeding a configured duration.

MySQL – useful commands and monitoring:

SHOW FULL PROCESSLIST;

SHOW STATUS LIKE 'Threads_connected';
SHOW STATUS LIKE 'Threads_running';
SHOW STATUS LIKE 'Max_used_connections';

Identify slow queries and database bottlenecks:

  • Slow Query Log – identify slow SQL queries.

  • Performance Schema – investigate query execution, waits, and database activity.

  • EXPLAIN ANALYZE – analyze query execution.

Database monitoring alternatives: pgAdmin, MySQL Workbench, Datadog Database Monitoring, Dynatrace, AWS CloudWatch Database Insights.

Note: EXPLAIN ANALYZE executes the query. Use caution with expensive or write queries in production.

D. Kafka Monitoring

What to monitor:

  • Consumer lag.

  • Message throughput.

  • Failed message processing.

  • Broker health and partition availability.

  • Under-replicated partitions and dead-letter topic volume.

Recommended combination: Kafka JMX Exporter + Prometheus + Grafana.

Alternatives: Confluent Control Center, Datadog.

Useful commands:

kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe --group my-consumer-group
kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --describe

E. Redis Monitoring

What to monitor:

  • Memory utilization.

  • Cache hit/miss ratio.

  • Command latency.

  • Connected clients.

  • Evictions and cache performance.

Tools: Redis CLI + Redis Insight + Prometheus + Grafana.

Useful commands:

redis-cli INFO
redis-cli INFO memory
redis-cli INFO clients
redis-cli INFO stats
redis-cli SLOWLOG GET 10

F. Kubernetes Monitoring

What to monitor:

  • Pod CPU and memory.

  • Pod restarts and OOMKilled events.

  • Node health.

  • CPU throttling.

  • Deployment status and resource limits.

Recommended combination: Prometheus + Grafana + kube-state-metrics + Metrics Server.

Alternatives: Datadog, Dynatrace.

5. APM – Application Performance Monitoring

Objective: Monitor application performance across services and identify bottlenecks at the application and transaction level.

What to investigate:

  • Slow API endpoints and database queries.

  • External API latency.

  • Distributed transaction bottlenecks.

  • Memory leaks and thread blocking.

  • Application-level performance degradation.

APM platforms – choose one:

  • Dynatrace.

  • Datadog APM.

  • New Relic.

  • AppDynamics.

  • Elastic APM.

These platforms provide application-level visibility, transaction tracing, service dependency analysis, and performance diagnostics.

6. JVM Profiling & Troubleshooting

Objective: Investigate JVM-level problems such as high CPU utilization, memory leaks, thread contention, deadlocks, and garbage collection issues.

What to investigate:

  • Capture thread dumps to identify blocked threads and deadlocks.

  • Capture heap dumps to investigate memory leaks.

  • Analyze garbage collection behaviour.

  • Identify CPU-intensive methods.

  • Investigate thread pool exhaustion.

  • Analyze object allocation and memory retention.

Tools:

  • Java Flight Recorder (JFR) + Java Mission Control (JMC).

  • Eclipse MAT – heap dump analysis.

  • VisualVM – JVM monitoring and profiling.

  • async-profiler – CPU and allocation profiling.

Useful commands:

# Find Java processes
jps -lv

# Capture thread dump
jcmd <pid> Thread.print

# Check heap information
jcmd <pid> GC.heap_info

# Start Java Flight Recorder
jcmd <pid> JFR.start

# Capture heap dump
jcmd <pid> GC.heap_dump /tmp/heap.hprof

Note: Heap dumps may contain sensitive application data. Handle them securely.

7. Production Debugging – Step-by-Step Approach

Follow this sequence when a production issue occurs.

Step 1 – Understand the Issue

  • Identify the affected API, microservice, and business transaction.

  • Check whether the issue affects all users or specific users.

  • Identify when it started and its business impact.

  • Check HTTP 500, 502, 503, and timeout errors.

Tools: Grafana, Prometheus, APM dashboards.

Step 2 – Identify the Request

  • Collect Correlation ID, Trace ID, timestamp, and transaction ID.

  • Identify the affected service and application instance.

  • Verify request headers and downstream propagation.

Tools: API Gateway logs, Grafana Loki, Kibana, Splunk.

Step 3 – Analyze Application Logs

  • Search using Correlation ID or Trace ID.

  • Identify exceptions, stack traces, and recurring error patterns.

  • Check recent application errors and failed transactions.

Tools: Grafana Loki, Kibana, Splunk, CloudWatch Logs Insights.

Step 4 – Analyze Distributed Traces

  • Search the Trace ID.

  • Visualize the complete request journey.

  • Identify slow services and failed spans.

  • Check database calls, Kafka interactions, and external APIs.

Tools: Grafana Tempo, Jaeger, Zipkin, Dynatrace.

Step 5 – Check Application and JVM Health

  • Investigate heap/non-heap memory, CPU, and GC.

  • Check thread count and thread pool exhaustion.

  • Capture thread dumps or JFR recordings if required.

Tools: Prometheus + Grafana, JFR + JMC.

Step 6 – Investigate Dependencies

  • Check database connections, slow queries, and locks.

  • Investigate Kafka consumer lag and failed messages.

  • Check Redis latency and cache behaviour.

  • Verify external API response times and connectivity.

Tools: PostgreSQL/MySQL native monitoring, Kafka JMX Exporter, Redis Insight, APM.

Step 7 – Investigate Infrastructure

  • Check CPU, memory, disk, network, and process health.

  • Check application instance and container health.

  • Investigate Kubernetes events and resource limits where applicable.

Tools: Linux commands, Prometheus, Grafana, kubectl.

Step 8 – Check Recent Changes

  • Review recent application deployments.

  • Verify configuration and environment variable changes.

  • Check database migrations and infrastructure changes.

  • Review feature flags and dependency upgrades.

Tools: Jenkins, GitHub Actions, Argo CD, AWS CloudTrail.

Step 9 – Identify Root Cause and Fix

  • Identify the actual failing component.

  • Fix code, configuration, database queries, or infrastructure.

  • Roll back the deployment if necessary.

  • Validate the fix in a controlled manner.

Step 10 – Validate and Prevent Recurrence

  • Monitor application health after deployment.

  • Verify error rates and latency return to normal.

  • Configure missing alerts and dashboards.

  • Add regression tests and document the root cause.

8. Debugging Applications Deployed on Linux VM

When Spring Boot applications run directly on a virtual machine, investigate the process, system resources, logs, JVM, and network.

A. Process and System Monitoring

Identify the running Java process and check CPU, memory, disk, and system load.

Tools: ps, top, htop, free, df, uptime.

ps -ef | grep java
top
htop
free -m
df -h
uptime

B. Application Logs

Inspect application logs and systemd service logs.

Tools: tail, grep, journalctl.

tail -f application.log
grep -i "ERROR" application.log

systemctl status order-service
journalctl -u order-service -f

C. JVM Investigation

Capture thread dumps, heap information, and JFR recordings.

Tools: JFR, JMC, jcmd.

jcmd <pid> Thread.print
jcmd <pid> GC.heap_info
jcmd <pid> JFR.start

D. Network Investigation

Check listening ports, connectivity, DNS, and application health.

Tools: ss, curl, nslookup, dig, tcpdump.

ss -tulpn
curl -v http://localhost:8080/actuator/health
nslookup example.com
dig example.com

For deeper network investigation:

sudo tcpdump -i any port 8080

E. Monitoring When the VM Runs on AWS

  • CloudWatch Agent – system and custom JVM metrics.

  • CloudWatch Logs – application and system logs.

  • AWS Systems Manager Session Manager – secure VM access.

  • AWS CloudTrail – infrastructure and configuration changes.

  • AWS X-Ray – distributed tracing when integrated.

9. Debugging Containerized Applications – Docker / Kubernetes

A. Docker Container Debugging

Objective: Investigate container crashes, resource utilization, application logs, and networking.

Check running containers and resource usage:

docker ps
docker stats
docker inspect <container-id>
docker logs -f <container-id>
docker logs --since 30m <container-id>

Investigate:

  • Container exits and restart behaviour.

  • CPU and memory consumption.

  • Application logs and environment configuration.

  • Container networking and connectivity.

B. Kubernetes Pod and Deployment Investigation

Objective: Identify pod failures, deployment issues, resource constraints, and cluster-level problems.

Check pods, deployments, services, and nodes:

kubectl get pods -n <namespace>
kubectl get deployments -n <namespace>
kubectl get services -n <namespace>
kubectl get nodes

Investigate pod failures:

kubectl describe pod <pod-name> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Look for CrashLoopBackOff, OOMKilled, ImagePullBackOff, failed probes, and scheduling failures.

C. Container Logs

kubectl logs <pod-name> -n <namespace>

# Logs from the previous container instance
kubectl logs <pod-name> -n <namespace> --previous

# Logs from a specific container
kubectl logs <pod-name> -c <container-name> -n <namespace> -f

D. Resource Utilization

kubectl top pods -n <namespace>
kubectl top nodes

Check CPU throttling, memory limits, and resource requests.

Note: kubectl top requires Metrics Server.

E. Container Access and Advanced Debugging

kubectl exec -it <pod-name> -n <namespace> -- /bin/sh

For minimal or distroless images without a shell, use ephemeral debugging containers:

kubectl debug <pod-name> -it \
  --image=<debug-image> -n <namespace>

For node-level investigation:

kubectl debug node/<node-name> -it --image=ubuntu

F. Kubernetes Observability Stack

  • Metrics: Prometheus + Grafana.

  • Logs: Grafana Loki + Alloy or Fluent Bit.

  • Traces: Grafana Tempo, Jaeger, Zipkin.

  • Telemetry collection: OpenTelemetry Collector.

  • Kubernetes state: kube-state-metrics.

  • Basic resource metrics: Metrics Server.

10. Resilience & Fault Tolerance

Objective: Prevent cascading failures and keep services functional when downstream services or dependencies become unavailable.

What to implement:

  • Circuit Breaker – Resilience4j: Stop repeated calls to an unhealthy downstream service and allow recovery after a configured wait period.

  • Retry – Resilience4j: Retry transient failures with limited attempts and exponential backoff.

  • Bulkhead – Resilience4j: Isolate resources so one slow dependency does not exhaust all application threads or connections.

  • Rate Limiting – Resilience4j RateLimiter, Bucket4j: Restrict request rates to protect services from excessive traffic.

  • Timeout Management – HTTP client and service-level configuration: Set connection and response timeouts to prevent requests from waiting indefinitely.

  • Fallback – Resilience4j: Return a default response, cached data, or graceful degradation when a dependency fails.

  • Dead Letter Queue – Kafka DLQ/DLT: Route repeatedly failed messages to a separate topic for investigation and controlled reprocessing.

  • Idempotency – Idempotency keys + database constraints: Prevent duplicate processing of the same business operation.

11. Additional Production Best Practices

  • Centralized configuration management.

  • Proper health checks and readiness/liveness probes.

  • Alerts based on SLIs, SLOs, and error budgets.

  • Avoid high-cardinality Prometheus labels.

  • Appropriate distributed trace sampling.

  • Context propagation through asynchronous processing.

  • Secure Actuator endpoints and monitoring infrastructure.

  • Never log passwords, access tokens, or sensitive customer information.

  • Monitor certificates, DNS, network connectivity, and disk utilization.

  • Maintain incident response procedures and post-incident reviews.

12. Recommended Modern Observability Stack

For a Java 21 / Spring Boot microservices architecture, one practical combination is:

  • Application: Java 21 + Spring Boot.

  • Application Logging: SLF4J + Logback + MDC.

  • Instrumentation: Micrometer + OpenTelemetry.

  • Request Tracking: Correlation ID + Trace ID + Span ID.

  • Centralized Logging: Grafana Loki + Grafana Alloy.

  • Metrics: Prometheus.

  • Visualization: Grafana.

  • Distributed Tracing: Grafana Tempo.

  • Telemetry Collection: Grafana Alloy or OpenTelemetry Collector.

  • JVM Profiling: JFR + JMC.

  • Database Monitoring: HikariCP metrics + PostgreSQL native statistics / MySQL Performance Schema.

  • Messaging: Kafka JMX Exporter + Prometheus + Grafana.

  • Resilience: Resilience4j.

  • Infrastructure: Linux VM + Docker + Kubernetes.

  • Alerting: Prometheus Alertmanager.

This is a suggested stack, not a requirement to deploy every alternative. Choose the logging, monitoring, and tracing platforms that fit your infrastructure and operational requirements.

Key Takeaway:

Production debugging is not just about finding exceptions in application logs. It is about correlating requests, analyzing metrics, tracing service interactions, investigating JVM and infrastructure health, identifying the actual root cause, implementing a fix, and preventing recurrence.

By