API Performance Optimization Guide

You are a backend performance engineer who specializes in identifying and resolving API latency and throughput bottlenecks. Create a comprehensive performance optimization plan for the following API system.

System Context:
API Framework: [EXPRESS/FASTIFY/DJANGO/RAILS/SPRING/GO/DOTNET]
Database: [POSTGRESQL/MYSQL/MONGODB/MULTI-DATABASE]
Current P99 Latency: [CURRENT LATENCY]
Target P99 Latency: [TARGET LATENCY]
Current Throughput: [REQUESTS PER SECOND]
Target Throughput: [TARGET RPS]

Section 1 - Performance Profiling and Baseline:
Define the performance baseline measurement methodology including which endpoints to profile, the traffic pattern to simulate, and the metrics to capture including P50, P95, P99, and P99.9 latency distributions. Design the profiling toolkit setup including application profiler for CPU and memory hotspots, database query analyzer for slow queries, network analyzer for external call latency, and distributed tracing for request flow visualization. Create the performance budget allocation that breaks the total latency target into component budgets for request parsing, authentication, business logic, database queries, serialization, and response transmission. Specify the continuous performance monitoring that tracks latency trends over time and automatically detects regressions. Define the performance regression testing that runs on every deployment comparing against the established baseline. Create the prioritized bottleneck identification process using Amdahl's law to determine which optimizations will have the greatest impact.

Section 2 - Database Query Optimization:
Define the slow query identification process using database query logs, ORM query analysis, and distributed tracing to find the highest-impact queries. Create the query optimization playbook including adding appropriate indexes, rewriting N+1 queries to use joins or batch loading, replacing correlated subqueries with Common Table Expressions, and optimizing pagination from offset to cursor-based. Specify the database connection pool tuning including pool size calculation based on available connections and worker concurrency, connection lifetime management, and idle connection cleanup. Design the read replica routing strategy that offloads read-heavy queries from the primary database. Define the query result caching strategy that stores frequently accessed and rarely changing query results in Redis or application memory. Create the database monitoring integration that tracks query execution time, connection utilization, and lock contention in real time.

Section 3 - Serialization and Payload Optimization:
Define the response payload optimization strategy including field selection to return only requested fields, lazy loading of nested relationships, and pagination for collection endpoints. Design the serialization performance tuning including choosing between JSON libraries with different performance profiles, implementing streaming serialization for large responses, and using binary protocols for internal service calls. Specify the compression strategy including response compression with gzip or brotli, minimum payload size thresholds, and content type-based compression rules. Create the partial response implementation that allows clients to specify exactly which fields they need reducing serialization and transfer time. Define the response format standardization that eliminates unnecessary wrapper objects and redundant data. Design the batch API endpoint strategy that reduces HTTP overhead by combining multiple operations into a single request.

Section 4 - Connection and Resource Management:
Define the HTTP connection management strategy including keep-alive configuration, connection pooling for upstream services, and DNS caching for external dependencies. Design the connection reuse optimization for database connections, Redis connections, and HTTP clients to eliminate connection setup overhead. Specify the resource leak detection mechanisms including connection pool monitoring, file descriptor tracking, and memory growth alerting. Create the graceful degradation strategy that sheds load when approaching resource limits including request queuing, circuit breaking, and priority-based request handling. Define the memory optimization approach including object pooling for frequently allocated structures, garbage collection tuning for the runtime, and buffer reuse for I/O operations. Design the asynchronous processing strategy that moves non-critical work out of the request path including background jobs, event emission, and fire-and-forget logging.

Section 5 - Caching and Precomputation:
Define the multi-level caching strategy from CDN through application cache to database query cache with appropriate TTLs and invalidation for each level. Design the request-level caching for authenticated endpoints using cache keys that incorporate user identity and permission scope. Specify the precomputation strategy for expensive calculations that can be computed ahead of time including aggregate statistics, recommendation scores, and derived data. Create the cache warming pipeline that prepopulates caches before traffic hits including predictive warming based on traffic patterns and scheduled warming for known high-traffic events. Define the edge caching strategy using CDN or edge computing for API responses that vary infrequently. Design the stale-while-revalidate implementation that serves slightly stale data while refreshing the cache asynchronously to eliminate cache miss latency.

Section 6 - Load Testing and Capacity Planning:
Define the load testing strategy including realistic traffic patterns, ramp-up sequences, sustained load tests, and spike tests that simulate traffic surges. Create the load testing infrastructure including test environment configuration, test data generation, and result analysis dashboards. Specify the performance acceptance criteria that must be met before each deployment including latency thresholds, error rate limits, and resource utilization caps. Design the capacity planning model that projects infrastructure needs based on traffic growth forecasts and performance characteristics. Define the auto-scaling configuration that responds to real-time traffic patterns including scaling triggers, cooldown periods, and maximum instance limits. Create the performance optimization backlog process that continuously identifies, prioritizes, and tracks optimization opportunities based on profiling data and business impact.