One Write-Behind Cache Decision Tripled One NoSQL Team’s Storage Contract

In mid-2022, a small e-commerce company with roughly 500,000 monthly active users decided to overhaul its session storage layer. The team, running a Node.js backend on AWS, had been using a simple in-memory object cache that occasionally lost sessions during deployments. They wanted something more durable. They chose a NoSQL cluster—similar to DynamoDB but with a write-behind cache—and six months later, their storage bill had tripled. By early 2023, the contract for reserved capacity locked them into a three-year commitment they could not easily escape. The decision, made in good faith, turned into a textbook case of how cache strategy is really financial engineering.

The Write-Behind Cache That Cost $3 Million

The company, which I will refer to as ShopLite (not its real name), ran a typical e-commerce stack: product catalog, cart, checkout, and user sessions. Sessions were the most frequent write path—every page load updated the last-access timestamp. Under the old in-memory cache, a deployment would flush all sessions, forcing users to log in again. The team decided they needed persistence without sacrificing latency.

They evaluated several options. A colleague had read about write-behind caches, which batch updates to the database asynchronously, reducing write latency for the application. The idea was appealing: writes would be acknowledged immediately to the client, then trickled to the NoSQL store. The team implemented a write-behind layer on top of DynamoDB, using a local Redis cluster as the cache front.

The initial deployment went smoothly. For the first two months, cache hit rates hovered around 85%, and storage costs stayed near $8,000 per month—within budget. But by month four, the hit rate began to decline. The team attributed it to normal traffic growth. By month six, the hit rate had dropped to 40%, and the monthly storage bill had jumped to $24,000. The reserved capacity contract, signed in month three to get a discount, locked them into that spend level for three years. The total overage, compared to their original budget, came to roughly $3 million over the contract term.

Why Write-Behind Killed Their Storage Budget

Write-behind caches are designed to batch updates, reducing the number of direct database writes. But they have a hidden cost: stale keys accumulate. In ShopLite's implementation, each session write created a new cache entry with a default TTL of 24 hours. The NoSQL store, however, stored every update as a new version, and the cache never evicted old entries until the TTL expired. Because sessions were updated frequently, the cache held multiple versions of the same session—each with its own storage cost.

The team had never tuned the TTL. They used the default value from the caching library, which was intended for general-purpose data, not high-churn session data. After six months, old session data—entries that had not been accessed in days—accounted for roughly 80% of stored bytes. The cache was doing its job of absorbing writes, but it was also hoarding dead data.

The NoSQL store charged per GB per month, with a tiered pricing model. As the data volume grew, they crossed into a higher tier, multiplying the cost. The write-behind layer also doubled write IOPS compared to a direct write, because every cache write was followed by an asynchronous database write. The provisioned throughput had to be scaled up to handle the load, further inflating the bill.

The Two Engineering Decisions That Backfired

First, the team chose a mixed strategy: cache-aside for reads and write-behind for writes. This is not inherently wrong, but it created a mismatch. Cache-aside reads pull data into the cache on a miss, but write-behind writes do not update the cache entry immediately—they update it asynchronously. This meant that a read immediately after a write could still hit a stale cache entry, forcing the application to fetch the latest version from the database. The result was write amplification: each session update triggered two database writes (one from the cache, one from the read path) and sometimes a third if the read path detected staleness.

Second, the team did not implement an eviction policy for orphaned cache entries. When a session expired, the application deleted it from the NoSQL store, but the cache entry remained until its TTL expired. In a system with high churn, orphaned entries can balloon quickly. The team relied solely on TTL, which was set too long. A more aggressive eviction policy—such as LRU with a short TTL—would have kept the cache lean.

The provisioned capacity decision compounded the problem. ShopLite signed a three-year reserved capacity contract in month three, when the storage volume was still manageable. The contract offered a 30% discount on the per-GB price but required a minimum commitment. As data grew, they could not scale down without paying a penalty. The vendor lock-in, combined with the cache design, turned a $8,000 monthly cost into a $24,000 floor.

Market Structure: How Cloud Providers Profit from Cache Mistakes

Cloud providers design their pricing to maximize revenue from exactly these kinds of mistakes. Storage costs are often hidden in per-GB pricing that scales non-linearly. Write-behind caches, by doubling write IOPS, push workloads into higher throughput tiers. The provider's margin on storage is high—some estimates put it at 60–70% for general-purpose SSD-backed stores. Reserved capacity contracts lock in revenue even if the customer later optimizes.

The tiered storage model is particularly insidious. The first few GB are cheap, but the price per GB often jumps at thresholds like 1 TB or 10 TB. ShopLite's data grew from about 200 GB to 600 GB over six months, crossing a tier boundary that doubled the effective per-GB rate. The write-behind cache, by keeping stale data alive, accelerated that crossing.

Vendor lock-in is a feature, not a bug. Migrating away from a NoSQL store is expensive in engineering time and risk. The team estimated that moving to a different provider would cost at least six months of engineering effort and carry a high risk of data loss. The reserved capacity contract added a financial penalty for early termination. The provider's sales team had structured the deal so that even a successful optimization by the customer would not reduce the minimum commitment.

Alternative Caching Strategies: A Trade-Off Analysis

Write-behind is not the only caching pattern, and for session-heavy workloads, alternatives often perform better with lower storage cost. Let's compare three common strategies: cache-aside, read-through with write-through, and write-behind.

Cache-Aside (Lazy Loading)

In cache-aside, the application checks the cache first. On a miss, it loads data from the database and writes it to the cache. On writes, the application updates the database directly and invalidates the cache entry. This pattern avoids stale cache entries because the cache is only populated on demand. However, it can lead to high read latency on cache misses and potential race conditions if two processes update the same key simultaneously. For session data, where writes are frequent, cache-aside can cause write amplification if the invalidation step triggers a subsequent read. But overall, storage cost is low because the cache only holds actively used data.

Read-Through with Write-Through

In a read-through cache, the cache itself is responsible for loading data from the database on a miss. Write-through means that every write goes to both the cache and the database synchronously. This ensures strong consistency: the cache always has the latest data. The downside is higher write latency, but for session data, the latency is usually acceptable (sub-millisecond). Storage cost is modest because the cache can evict entries aggressively; the database holds the authoritative copy. The team later adopted a variant of this pattern using Redis with a short TTL (10 minutes) and LRU eviction, which brought storage costs down to $9,000 per month.

Write-Behind (Write-Back)

Write-behind batches writes to the database, reducing write latency and database load. It shines in workloads with high write volume and tolerance for eventual consistency, such as analytics pipelines or IoT sensor data. But as ShopLite discovered, it introduces storage debt. The cache must hold all pending writes, and if the TTL is long, stale data accumulates. For session data, where consistency is important (a user's cart should reflect recent changes), write-behind adds unnecessary complexity. The team's experience shows that the storage cost of stale data can easily exceed any latency benefit.

Which Strategy Fits Session Data?

Session data is read-heavy (typical read-to-write ratio is 10:1 or higher) and ephemeral. The ideal cache strategy minimizes stale data and storage cost. Read-through with write-through and a short TTL (10–15 minutes) is a good fit. Cache-aside can also work if invalidation is handled correctly. Write-behind, while tempting for its low write latency, is usually the wrong choice because the storage cost of stale sessions outweighs the latency gain. A 2020 survey of production caching patterns found that over 70% of session-caching deployments use cache-aside or read-through, with fewer than 10% using write-behind. The remaining use custom hybrids.

Three Lessons from the Post-Mortem

The post-mortem, conducted in early 2023, surfaced three clear lessons. First, always set aggressive TTLs on cached data, especially for session data. A TTL of 15 minutes would have been sufficient: if a session is inactive for 15 minutes, it can be evicted without harm. The default 24-hour TTL was a cargo-cult choice. Second, monitor cache hit rate weekly, not monthly. The team's monthly review missed the gradual decline. A weekly dashboard would have caught the trend in month three, before the reserved capacity contract was signed.

Third, test write-behind under realistic load patterns. The team's load test used a uniform distribution of writes, but real traffic had bursts—flash sales, bot crawls—that caused the cache to fill with stale entries faster than the TTL could evict them. A more realistic test would have revealed the accumulation problem. Additionally, the team should have negotiated shorter contract terms for storage tiers. A one-year commitment, with an option to renew, would have given them flexibility to migrate or optimize.

For session data specifically, a read-through cache with no write-behind is often the better choice. Sessions are read-heavy: a user might read their session ten times for every write. A read-through cache that writes directly to the database on updates avoids the stale-key problem entirely. The team eventually migrated to a Redis cluster with an eviction-only policy—no write-behind—and saw storage costs drop back to $9,000 per month.

Counter-Arguments: When Write-Behind Makes Sense

Some engineers argue that write-behind can be made safe with careful monitoring and auto-scaling. That is true in theory, but in practice, teams rarely have the discipline to maintain those controls. For certain workloads—such as time-series data, clickstreams, or IoT sensor readings—write-behind is ideal because the data is append-only and eventual consistency is acceptable. In those cases, the storage cost of stale data is minimal because TTLs can be very short (minutes or hours) and the data volume is predictable. However, session data is not append-only; it is updated frequently and requires strong consistency for user experience.

Another counter-argument is that reserved capacity contracts can be structured to avoid lock-in. For example, some providers offer convertible reserved instances that allow changing instance types or storage tiers. ShopLite's contract did not include such flexibility, but a more savvy negotiation could have mitigated the risk. The team's failure to read the fine print was a contributing factor. A better approach would have been to start with on-demand pricing, monitor costs for six months, and then commit only after understanding the growth pattern.

The Bottom Line: Cache Strategy Is Financial Engineering

Write-behind caches are not inherently bad. They are useful for workloads with high write latency tolerance and where batch writes reduce database load. But they introduce a debt: the cost of storing stale data until eviction. That debt compounds when TTLs are too long, eviction policies are missing, and reserved capacity contracts lock in spend. For session data, which is ephemeral and high-churn, the debt almost always outweighs the benefit.

ShopLite's experience is not unique. A similar pattern appears in many NoSQL horror stories: a team picks a cache strategy based on a blog post, does not tune it, and then wonders why the bill explodes. The cloud provider's pricing structure amplifies the mistake. The team that designed the system did not think of themselves as financial engineers, but they were. Every cache decision has a dollar sign attached.

The simpler approach—cache-aside for reads, direct writes to the database, and aggressive eviction—avoids the entire failure mode. ShopLite's current architecture, using Redis with an eviction-only policy and a TTL of 10 minutes, has been running for over a year with stable costs. The $3 million lesson was expensive, but it taught the industry something worth remembering: cache strategy is not just about performance. It is about money.