Production issue debugging in Spring Boot microservices requires more than checking application logs. A failure can originate from application code, JVM resources, database queries, Kafka consumers, network communication, containers, or Kubernetes infrastructure.
This guide explains how to identify, investigate, and resolve production issues using practical debugging techniques, observability tools, JVM diagnostics, and infrastructure commands.
Observability – Core Signals
Observability helps understand the internal state of an application by analyzing its outputs and behaviour.
The five core observability signals are:
Logs – What happened?
Application events, exceptions, stack traces, errors, and business transactions.
Metrics – How often and how badly is it happening?
Request count, error rate, response time, CPU, memory, garbage collection, threads, and database connections.
Distributed Tracing – Where did the request spend its time?
Follow requests across API Gateway, microservices, databases, Kafka, and external APIs.
Profiling – What is consuming application resources?
CPU-intensive methods, memory allocation, thread contention, deadlocks, and JVM bottlenecks.
Events – What changed in the environment?
Application deployments, configuration changes, infrastructure changes, scaling events, and database migrations.
How These Signals Work Together
Use metrics to identify abnormal behaviour and performance degradation.
Use traces to find the failing or slow service.
Use logs to understand the actual exception or failure.
Use profiling to investigate resource-level bottlenecks.
Use deployment and infrastructure events to identify possible triggers.
Correlation IDs and Distributed Tracing
Distributed tracing helps track a request as it travels across multiple microservices and external dependencies. Correlation IDs and trace IDs make it easier to connect logs and traces from different services.
1. Correlation ID and Trace ID
Correlation ID: Identifies a business request across multiple services.
Trace ID: Identifies an end-to-end distributed trace.
Span ID: Identifies an individual operation within a trace.
A typical request flow looks like this:
Client
|
API Gateway
| Trace ID: abc123
|
Order Service
| Span ID: span-01
|
Payment Service
| Span ID: span-02
|
Inventory Service
Span ID: span-03Use a consistent correlation ID across services when you need to search related application logs.
2. Implementing Distributed Tracing
For Spring Boot 3.x and Java 21, use Micrometer Tracing with an OpenTelemetry bridge.
Common dependencies include:
micrometer-tracing-bridge-otelopentelemetry-exporter-otlpspring-boot-starter-actuator
Micrometer Tracing integrates with Spring Boot and can automatically add trace and span information to supported logging contexts.
Example configuration:
management:
tracing:
sampling:
probability: 0.1
otlp:
tracing:
endpoint: http://otel-collector:4318/v1/tracesA sampling probability of 0.1 means approximately 10% of eligible traces are sampled. Adjust this according to traffic volume and troubleshooting requirements.
3. Propagating Correlation IDs
You can configure a custom correlation ID as a baggage field:
management:
tracing:
baggage:
remote-fields: x-correlation-id
correlation:
fields: x-correlation-idThe correlation ID can then be propagated between supported instrumented services.
Use the auto-configured HTTP client builders for instrumentation:
RestClient.BuilderRestTemplateBuilderWebClient.Builder
Creating HTTP clients manually, such as using new RestTemplate(), may bypass Spring Boot's automatic instrumentation.
4. Distributed Tracing with Kafka
For asynchronous communication, tracing context should be propagated through Kafka message headers.
Spring Kafka observation can be enabled using:
spring:
kafka:
template:
observation-enabled: true
listener:
observation-enabled: trueThis helps connect producer and consumer operations in distributed traces.
For asynchronous execution, Reactor pipelines, and custom executors, use Micrometer Context Propagation and appropriate context propagation configuration.
5. Distributed Tracing Tools
OpenTelemetry: Vendor-neutral instrumentation and telemetry collection.
Micrometer Tracing: Integrates tracing with Spring Boot applications.
Grafana Tempo: Stores and queries distributed traces.
Jaeger: Distributed tracing platform.
Zipkin: Distributed tracing and visualization.
AWS X-Ray: Distributed tracing for AWS applications.
For new instrumentation, OpenTelemetry is a good vendor-neutral starting point.
Application Logging and Centralized Log Management
Application logs are one of the most important sources of information when investigating production failures.
A centralized logging system collects logs from multiple services and provides a single place to search and analyze them.
1. Structured Logging
Prefer structured JSON logging over plain-text messages.
Example:
{
"timestamp": "2026-11-02T10:30:00Z",
"level": "ERROR",
"service": "order-service",
"traceId": "abc123",
"spanId": "span001",
"correlationId": "CORR-12345",
"message": "Payment processing failed",
"exception": "PaymentTimeoutException"
}Spring Boot 3.4 and later support built-in structured logging formats.
Example configuration:
logging.structured.format.console=ecsOther supported formats include Logstash and GELF.
2. Centralized Logging Platforms
ELK Stack
Elasticsearch: Stores and indexes application logs.
Logstash: Processes and transforms log data.
Kibana: Provides log searching, dashboards, and visualization.
Grafana Loki
Stores and queries application logs.
Integrates with Grafana dashboards.
Commonly uses Grafana Alloy for log collection.
OpenSearch
Open-source search and analytics platform.
Can be used for centralized log storage and investigation.
Splunk
Enterprise platform for log analysis, monitoring, and security investigations.
AWS CloudWatch Logs
Collects and monitors logs from AWS workloads.
Integrates with EC2, Lambda, ECS, EKS, and other AWS services.
3. Log Collection Architecture
A typical centralized logging architecture:
Spring Boot Microservices
|
v
Log Files / Console
|
v
Log Collector
|
v
Centralized Log Platform
|
v
Search and DashboardFor Loki, avoid using high-cardinality values such as trace IDs and correlation IDs as labels. Search for these values in the log content instead.
Example LogQL query:
{service_name="order-service"} |= "CORR-12345"4. Logging Best Practices
Include timestamp, log level, service name, and trace information.
Use appropriate log levels: DEBUG, INFO, WARN, and ERROR.
Log exceptions with stack traces when useful.
Avoid logging passwords, tokens, personal information, and sensitive payloads.
Use consistent formats across services.
Avoid excessive logging inside high-volume loops.
Configure log rotation and retention policies.
Metrics and Application Monitoring
Metrics help identify abnormal application behaviour before it becomes a major production incident.
Spring Boot Actuator and Micrometer provide application metrics that can be exported to monitoring platforms.
1. Spring Boot Actuator and Micrometer
Add the following dependencies:
spring-boot-starter-actuatormicrometer-registry-prometheus
Example configuration:
management:
endpoints:
web:
exposure:
include: health,info,metrics,prometheus
endpoint:
health:
show-details: when_authorizedSecure management endpoints and expose only the endpoints required by your monitoring environment.
2. Prometheus and Grafana
Prometheus: Collects and stores time-series metrics.
Grafana: Creates dashboards and visualizes metrics.
Alertmanager: Handles and routes Prometheus alerts.
Typical monitoring architecture:
Spring Boot Applications
|
Micrometer
|
Prometheus
|
Grafana
|
Alertmanager3. JVM Metrics
Monitor JVM metrics to identify memory, garbage collection, and thread-related problems.
Important metrics include:
| Metric | Purpose |
|---|---|
jvm_memory_used_bytes | JVM memory utilization |
jvm_gc_pause_seconds | Garbage collection pause duration |
jvm_threads_states_threads | Thread states |
process_cpu_usage | Application process CPU usage |
Investigate increasing heap usage, frequent garbage collection, long GC pauses, and growing thread counts.
4. API Performance Metrics
Monitor the following HTTP metrics:
Request count.
Error rate.
Average response time.
P95 and P99 response time.
Requests per second.
HTTP 4xx and 5xx responses.
For example, a Prometheus query for the 99th percentile HTTP request duration:
histogram_quantile(
0.99,
sum(rate(http_server_requests_seconds_bucket[5m])) by (le)
)Use histogram metrics to understand latency distribution rather than relying only on average response time.
5. Database Connection Pool Monitoring
For applications using HikariCP, monitor:
Active connections.
Idle connections.
Pending connection requests.
Connection acquisition time.
Connection timeouts.
A continuously increasing pending connection count may indicate connection pool exhaustion, slow queries, or connections being held for too long.
6. PostgreSQL Monitoring
Useful PostgreSQL monitoring sources include:
pg_stat_activity: Shows active database sessions and queries.pg_stat_statements: Provides query execution statistics.pg_blocking_pids: Helps identify blocking sessions.EXPLAIN ANALYZE: Examines query execution plans and actual execution performance.
Example:
SELECT
pid,
usename,
state,
wait_event_type,
wait_event,
query
FROM pg_stat_activity;Identify blocked queries using:
SELECT
pid,
pg_blocking_pids(pid) AS blocking_pids,
query
FROM pg_stat_activity
WHERE cardinality(pg_blocking_pids(pid)) > 0;Be careful when using EXPLAIN ANALYZE: it executes the query.
7. MySQL Monitoring
Useful MySQL diagnostic commands:
SHOW FULL PROCESSLIST;
SHOW ENGINE INNODB STATUS;Monitor:
Threads_connectedThreads_runningMax_used_connections
For deeper query analysis, use Performance Schema, the Slow Query Log, and EXPLAIN ANALYZE (MySQL 8.0.18 and later).
8. Kafka Monitoring
Kafka monitoring should cover both broker health and consumer performance.
Broker and JVM metrics
CPU and memory utilization.
Request handler idle percentage.
Network processor idle percentage.
Under-replicated partitions.
Offline partitions.
Use Kafka JMX metrics with Prometheus JMX Exporter.
Consumer metrics
Consumer lag.
Records consumed.
Processing rate.
Failed message count.
Rebalance frequency.
Useful tools include Kafka Exporter and Burrow for consumer lag monitoring.
Example commands:
kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--describe \
--group order-service-groupkafka-topics.sh \
--bootstrap-server localhost:9092 \
--list9. Redis Monitoring
Useful Redis commands:
redis-cli INFO
redis-cli SLOWLOG GET 10
redis-cli --latencyMonitor memory usage, connected clients, command latency, cache hit ratio, and rejected connections.
Avoid expensive production diagnostics such as KEYS * and continuous MONITOR usage on busy Redis instances.
10. Kubernetes Monitoring
Monitor the health and resource utilization of Kubernetes workloads.
Common tools:
Prometheus: Collects cluster and application metrics.
Grafana: Provides Kubernetes dashboards.
kube-state-metrics: Exposes Kubernetes object state.
Metrics Server: Provides resource metrics for pods and nodes.
Node Exporter: Collects node-level metrics.
cAdvisor: Provides container resource metrics.
Application Performance Monitoring (APM)
APM platforms combine application performance metrics, distributed tracing, error tracking, and transaction analysis.
They help identify slow endpoints, database bottlenecks, external API delays, and service-level performance degradation.
1. Popular APM Platforms
Dynatrace: Application performance monitoring, distributed tracing, and infrastructure visibility.
Datadog APM: Application tracing, service maps, metrics, and log correlation.
New Relic: Application monitoring, transaction analysis, and distributed tracing.
AppDynamics: Application performance and business transaction monitoring.
Elastic APM: Application monitoring integrated with the Elastic ecosystem.
2. When to Use APM
Use APM when you need to:
Identify slow API endpoints.
Trace requests across multiple microservices.
Find database and external API bottlenecks.
Investigate high error rates.
Compare application performance before and after deployments.
Correlate logs, traces, and metrics.
JVM Profiling and Troubleshooting
JVM-level debugging helps identify memory leaks, high CPU utilization, garbage collection problems, thread contention, and deadlocks.
1. JVM Diagnostic Tools
Java Flight Recorder (JFR)
Captures JVM runtime events.
Helps investigate CPU usage, memory allocation, garbage collection, and thread activity.
Can be used in production with appropriate recording settings.
Java Mission Control (JMC)
Analyzes Java Flight Recorder recordings.
Helps identify performance bottlenecks.
Eclipse Memory Analyzer (MAT)
Analyzes heap dumps.
Identifies memory leaks and objects retaining large amounts of memory.
VisualVM
Monitors JVM CPU, memory, threads, and garbage collection.
Supports profiling and heap dump analysis.
async-profiler
Low-overhead CPU and allocation profiling.
Useful for identifying expensive methods.
Grafana Pyroscope
Continuous profiling platform.
Helps analyze CPU and memory allocation over time.
2. Capture a Thread Dump
Use jcmd to capture a thread dump:
jcmd <pid> Thread.printFor Java 21, a JSON-format thread dump can also be generated:
jcmd <pid> Thread.dump_to_file \
-format=json /tmp/threads.jsonInspect thread states to identify:
BLOCKED threads.
WAITING threads.
Deadlocks.
Thread pool exhaustion.
Long-running operations.
For applications using Java virtual threads, use suitable JFR events and JSON thread dumps. Traditional thread dumps may not show virtual-thread activity in the same way as platform threads.
3. Investigate High CPU Usage
First identify the Java process:
topThen inspect individual threads:
top -H -p <pid>Convert the thread ID from decimal to hexadecimal:
printf '%x\n' <tid>Find the corresponding nid in the thread dump.
This helps identify the Java thread consuming excessive CPU.
4. Capture Heap Information
Check JVM heap information:
jcmd <pid> GC.heap_infoGenerate a class histogram:
jcmd <pid> GC.class_histogramCapture a heap dump:
jcmd <pid> GC.heap_dump /tmp/heapdump.hprofAnalyze the dump using Eclipse MAT or another compatible tool.
Heap dumps can cause pauses, consume significant disk space, and contain sensitive application data. Capture them only after considering production impact and access controls.
5. Start a Java Flight Recorder Session
Start a recording:
jcmd <pid> JFR.start \
name=ProductionDebug \
duration=120s \
filename=/tmp/production-debug.jfrAnalyze the generated recording using Java Mission Control.
6. Configure Automatic Heap Dumps and GC Logs
For applications running in production, consider configuring automatic heap dumps:
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/dumpsEnable garbage collection logging:
-Xlog:gc*:file=/var/log/app/gc.log:time,uptime:filecount=5,filesize=20mEnsure the configured dump and log directories have sufficient disk space and appropriate permissions.
Step-by-Step Production Debugging
Follow a systematic process instead of randomly restarting services or changing configurations.
Step 1. Understand the Production Impact
First identify the scope and severity of the issue.
Is the issue affecting all users or specific users?
Is one microservice failing or multiple services?
Are requests failing, timing out, or responding slowly?
Is the issue intermittent or continuously reproducible?
Is there a business-critical transaction affected?
Check dashboards, alerts, and service health before making changes.
Step 2. Check Recent Changes
Investigate recent changes that may have introduced the issue.
Application deployments.
Configuration changes.
Database migrations.
Infrastructure changes.
Dependency upgrades.
Kubernetes scaling or scheduling events.
Compare the current deployment with the previous stable version.
Step 3. Identify the Request
Use the correlation ID, trace ID, or relevant business identifier to identify the affected request.
For example:
correlationId = CORR-12345Search for this identifier in centralized logs and distributed traces.
Step 4. Analyze Application Logs
Look for:
Exceptions and stack traces.
Repeated error messages.
Timeout exceptions.
Authentication and authorization failures.
Validation errors.
Unexpected application behaviour.
Identify the first meaningful failure rather than focusing only on downstream exceptions.
Step 5. Analyze Distributed Traces
Follow the request across the service boundaries.
Identify:
Which service is taking the most time?
Which service is returning an error?
Is the database operation slow?
Is an external API timing out?
Is Kafka message processing delayed?
This helps isolate the failing component.
Step 6. Investigate JVM and Application Resources
Check CPU, memory, garbage collection, and thread utilization.
Use:
topandhtopfor CPU.free -mfor memory.jcmdfor JVM diagnostics.JFR and JMC for profiling.
Heap dumps for memory investigations.
Thread dumps for thread-related problems.
Step 7. Check External Dependencies
Investigate the health of dependent services and infrastructure.
Database availability and query performance.
HikariCP connection pool utilization.
Kafka broker health and consumer lag.
Redis latency and memory.
External API response times.
DNS and network connectivity.
Step 8. Investigate Infrastructure
Check whether the issue originates outside the application.
CPU and memory pressure.
Container restarts.
OOMKilled events.
Kubernetes pod scheduling issues.
Network failures.
Disk utilization and inode exhaustion.
Deployment and scaling events.
Step 9. Identify Root Cause and Apply a Fix
Once the failing component is identified:
Reproduce the issue safely where possible.
Identify the underlying cause.
Apply the smallest appropriate fix.
Validate the change in a controlled environment.
Follow the organization's production change and rollback procedures.
Avoid applying speculative fixes without understanding their impact.
Step 10. Validate and Prevent Recurrence
After applying the fix:
Verify application health.
Check error rates and response times.
Confirm that dependent services are healthy.
Monitor logs, traces, and metrics.
Add alerts for the identified failure condition.
Document the root cause and preventive actions.
Debugging Linux VM and Containerized Applications
Production Spring Boot applications commonly run on Linux virtual machines, Docker containers, and Kubernetes clusters. Understanding the underlying environment is essential for effective debugging.
1. Linux VM Debugging
Useful Linux commands:
Process and CPU
ps -ef | grep java
pgrep -af java
top
htopMemory and Disk
free -m
df -h
df -iSystem Load and Performance
uptime
vmstat 1System Logs
journalctl -u <service-name>
dmesg | grep -i -E 'oom|killed process'Network Diagnostics
ss -tulnp
curl -v http://localhost:8080/actuator/health
nslookup example.com
dig example.comFor deeper network investigation, tcpdump can capture network packets. Use it carefully because packet captures may contain sensitive information.
2. AWS Production Debugging
For applications hosted on AWS, use the following services:
Amazon CloudWatch: Collects application and infrastructure logs and metrics.
CloudWatch Agent: Collects operating system metrics and logs from EC2.
AWS Systems Manager Session Manager: Provides secure access to managed instances without requiring SSH access.
AWS CloudTrail: Tracks AWS API activity and account-level changes.
AWS X-Ray: Provides distributed tracing and service dependency visualization.
For new instrumentation, consider OpenTelemetry or AWS Distro for OpenTelemetry.
3. Docker Container Debugging
Useful Docker commands:
List running containers
docker psCheck container resource utilization
docker statsInspect container configuration and state
docker inspect <container-id>View container logs
docker logs -f <container-id>Inspect Docker events
docker eventsCheck the container's exit code and OOMKilled status when investigating unexpected restarts.
Exit code 137 commonly indicates that a process received SIGKILL. It may be caused by an out-of-memory kill, but confirm the container state and system events before concluding.
4. Kubernetes Debugging
Useful Kubernetes commands:
Check pods and deployments
kubectl get pods -n <namespace>
kubectl get deployments -n <namespace>
kubectl get services -n <namespace>
kubectl get nodesInspect a pod
kubectl describe pod <pod-name> -n <namespace>Check pod logs
kubectl logs <pod-name> -n <namespace>For a previously restarted container:
kubectl logs <pod-name> \
-n <namespace> \
--previousCheck resource utilization
kubectl top pods -n <namespace>
kubectl top nodesCheck deployment status
kubectl rollout status deployment/<deployment-name> \
-n <namespace>Inspect deployment history
kubectl rollout history deployment/<deployment-name> \
-n <namespace>Rollback a deployment
kubectl rollout undo deployment/<deployment-name> \
-n <namespace>Access a running container
kubectl exec -it <pod-name> \
-n <namespace> \
-- /bin/shFor minimal or distroless containers, use an ephemeral debug container when appropriate:
kubectl debug -it <pod-name> \
-n <namespace> \
--target=<container-name> \
--image=busybox5. Kubernetes Memory and CPU Issues
Investigate the following conditions:
OOMKilled: Container terminated because it exceeded its memory limit.CrashLoopBackOff: Container repeatedly fails to start or stay running.Pending: Pod cannot be scheduled.CPU throttling: Container is being restricted by CPU limits.
Node pressure: Node has insufficient memory, disk, or other resources.
A Java application needs memory beyond its configured heap, including metaspace, thread stacks, direct buffers, and other native allocations.
Do not allocate the entire container memory limit to the JVM heap.
Resilience and Fault Tolerance
Production microservices must handle temporary failures in dependent services without causing cascading failures.
Resilience patterns help maintain application stability when databases, external APIs, or other microservices become unavailable.
1. Circuit Breaker
Prevents repeated calls to a failing service.
Common states:
CLOSED: Requests are allowed.
OPEN: Requests are rejected temporarily.
HALF_OPEN: A limited number of requests are allowed to test recovery.
Use Resilience4j Circuit Breaker to implement this pattern.
2. Retry
Retries transient failures.
Best practices:
Retry only appropriate transient failures.
Use bounded retry attempts.
Apply exponential backoff and jitter.
Retry only idempotent operations, or use appropriate idempotency protection.
Avoid retry storms.
3. Bulkhead
Isolates resources so that failures in one dependency do not exhaust resources across the entire application.
For example, use separate thread pools or concurrency limits for different external service calls.
4. Timeout and TimeLimiter
Set appropriate timeouts for HTTP requests, database operations, and external service calls.
Avoid allowing requests to wait indefinitely for an unavailable dependency.
5. Rate Limiter
Controls the number of requests accepted within a configured time window.
This helps protect services from excessive traffic.
6. Fallback
Provides an alternative response when a dependent operation fails.
Fallbacks should preserve business correctness. Avoid returning misleading success responses when a critical operation has actually failed.
7. Kafka Dead Letter Topics
Use Dead Letter Topics (DLTs) to isolate messages that cannot be processed successfully after configured retries.
Investigate failed messages, fix the underlying issue, and implement a controlled replay process.
Production Best Practices
1. Health Checks
Implement Spring Boot Actuator health endpoints.
Use Kubernetes probes appropriately:
Liveness Probe: Determines whether the application needs to be restarted.
Readiness Probe: Determines whether the application can receive traffic.
Startup Probe: Allows sufficient time for slow application startup.
Avoid making liveness checks dependent on every external service because temporary dependency failures can cause unnecessary restarts.
2. Monitoring and Alerting
Define Service Level Indicators (SLIs) and Service Level Objectives (SLOs) for important application behaviour.
Monitor:
Availability.
Request latency.
Error rates.
Resource utilization.
Dependency health.
Message processing delays.
Configure alerts around meaningful service-level symptoms rather than every individual metric.
3. Secure Production Diagnostics
Secure Spring Boot Actuator endpoints.
Restrict access to heap dumps and thread dumps.
Do not expose sensitive configuration.
Avoid logging credentials, access tokens, or personal information.
Apply appropriate log retention and access policies.
4. Graceful Shutdown
Configure applications to stop accepting new requests and complete in-flight work where possible.
For Kubernetes deployments, configure appropriate termination grace periods and ensure application shutdown behaviour is compatible with the deployment lifecycle.
5. Resource Management
Configure JVM heap based on container memory limits.
Monitor garbage collection.
Tune HikariCP connection pools according to database capacity.
Configure appropriate thread pool sizes.
Monitor Kafka consumer lag.
Review CPU throttling and memory pressure.
Recommended Modern Observability Stack
The following stack provides a practical starting point for Java 21 and Spring Boot microservices.
| Area | Recommended Tool | Purpose |
|---|---|---|
| Application Framework | Java 21, Spring Boot 3.x | Microservice development |
| Logging | SLF4J, Logback | Application logging |
| Log Correlation | MDC, Micrometer Tracing | Trace and correlation context |
| Instrumentation | OpenTelemetry | Telemetry collection and propagation |
| Log Management | Grafana Loki, Alloy | Centralized log collection and querying |
| Metrics | Prometheus, Micrometer | Metrics collection |
| Dashboards | Grafana | Monitoring and visualization |
| Distributed Tracing | Grafana Tempo | Trace storage and querying |
| Profiling | JFR, JMC, async-profiler | JVM performance analysis |
| Memory Analysis | Eclipse MAT | Heap dump investigation |
| Database Monitoring | HikariCP metrics, DB statistics | Connection and query monitoring |
| Kafka Monitoring | JMX Exporter, Kafka Exporter | Broker metrics and consumer lag |
| Resilience | Resilience4j | Fault tolerance |
| Infrastructure | Linux, Docker, Kubernetes | Runtime diagnostics |
| Alerting | Prometheus Alertmanager | Alert routing |
Key Takeaway
Production debugging is a systematic process of identifying symptoms, narrowing down the failing component, investigating the root cause, and validating the fix.
For Spring Boot microservices, combine logs, metrics, distributed tracing, JVM profiling, and infrastructure diagnostics to understand what is happening across the entire system.
A well-designed observability setup does more than report failures. It helps engineering teams detect issues early, reduce troubleshooting time, improve application reliability, and prevent recurring production incidents.
