How We Replaced Our Pricing System While It Was Running

Clipboard is a marketplace where workplaces post shifts, and workers browse and book them. The shift's price determines whether it fills, whether a workplace posts again, and whether the marketplace makes money.

In mid-2023, if you asked, "Why does this shift have this price?" the answer lived in at least four places. One service priced shifts when events happened (a shift was created, a booked worker canceled late). A Lambda function repriced each market every ten minutes by querying the monolith's database directly, so a schema change elsewhere could silently break pricing. Two more pricing flows lived inside the monolith itself. Prices were mutable fields on the shift record, auditable only through a shared log collection, and some pricing logic worked by searching that log for magic strings. Yes, really.

The operational cost was constant: an average of five pricing alerts a day, recurring incidents, and a steady stream of "is this a pricing bug?" investigations that consumed engineering and ops time whether or not a bug existed. Worse, every marketplace feature we shipped was a variation on the same two mechanics: shift visibility and pay rates, and each one got built as a separate special case in a different part of the system. Real-time pricing was structurally impossible because the batch system could update prices only once every ten minutes.

Eleven months later, one service priced every shift in real time, the moment a worker requested it, through a rules engine where each price is the output of named, composable, individually testable rules. The batch Lambda and the old event-based path were gone. We never paused feature work, never ran late-night migrations, and business metrics held steady throughout.

This post is about how. The short version: we treated milestones as decision points instead of deadlines, we wrapped every migration in the same three-state feature flag, and when it came time to swap pricing systems, we didn't flip a switch. We turned a dial.

State the end goal, then iterate

The tempting move was a clean-slate pricing service: work in private for months, then switch over. We explicitly rejected that as a strategic mistake. Instead, we wrote down the end goal and the problems (impeded experimentation, insufficient reliability and auditability, no isolation between markets), then committed only to the first milestone in detail. Each milestone got its own design doc; later milestones stayed intentionally vague so we could fold in what we learned along the way.

One architectural decision did get made up front, because everything else depended on it. Should consumers get prices from events (price-updated messages they store locally) or synchronously (call the pricing service when you need a price)? Event-driven wins on availability and read performance. But it means pricing logic leaks into every consumer, and the records that guarantee a worker gets paid the price they saw would live outside the pricing service too. Synchronous is slower and forces more change through the system, but it puts all pricing logic and data in one place. Since real-time pricing and consolidation were the point of the whole project, we chose synchronous and accepted the cost.

The project also grew a second half early on. Pricing controls one lever; shift visibility controls the other, and every marketplace feature pulls both. So the vision became a single service owning both shift visibility and pricing. Our product manager made the case for why this mattered more than any individual feature:

A unified layer where any engineer can see all of the pricing/shift visibility logic in one place enables launching marketplace features 10x faster. This is the highest leverage thing we can do.

A milestone designed to answer questions

While designing the first big milestone, we hit two unknowns large enough to threaten the timeline: where the new service should live, and whether we should replace the legacy pricing service outright or keep it as a data API underneath the new one. Rather than guess, we carved out a smaller milestone, M0.5, whose explicit job was to answer those two questions while shipping pricing through a new rules engine.

The rules themselves are pure functions. Given the same inputs, they produce the same outputs, with no side effects, which makes them trivial to test and, more importantly, trivial to explain after the fact.

The rollout set the pattern every subsequent migration reused. One feature flag, three states:

  • off: old code only, the safe fallback.
  • dry-run: run both systems, compare results, log and alert on discrepancies, keep using the old result.
  • on: same as dry-run, but use the new result.

We dry-ran for a week, launched to ten markets before the holidays, went fully live in January, and deleted the flag two weeks later. The retro answered both questions, and one answer was "keep the legacy service for now": replacing it would add scope and coordination risk for no near-term benefit. Neither answer was obvious when we started, and we would have paid for guessing wrong.

Swapping the milestones

The original plan tackled the batch repricing system first and real-time pricing later. In late December, we swapped them, and this was probably the most consequential decision of the project.

The reasoning: without real-time pricing, you price shifts the same for the entire marketplace, and that pushes you toward predictive complexity. The existing system priced shifts in a workplace's time slot differently to avoid over-subsidizing, then normalized across the market; all state that has to be computed ahead of time and kept consistent. Rebuilding that system first, even cleanly, would have reproduced its shape, and that shape is hard to refactor into a real-time one. From the decision message:

By doing real-time first, we're able to be more reactive, which tends toward statelessness and simplicity. On each request and based on current market conditions, we can decide whether a shift should be visible and whether it should be subsidized.

A price computed at request time doesn't need to predict anything; it can just look at the market as it is right now. Real-time first also unlocked the visibility lever, not just the price lever. This is the payoff of milestone-based planning: the milestones gave us a defined point to regroup, and the first milestone gave us the information to know the original order was wrong.

Two migrations, same playbook

Real-time pricing needed two things moved into the new service: shift search (so we control visibility) and shift offers (so we control the price a worker is quoted).

Shift search meant migrating the service behind every open-shifts view in the worker app. A proof of concept moved all the endpoints in about a day; making the move safe took six weeks. We mirrored production traffic to both implementations and diffed the responses, shift IDs, order, and all, before any customer traffic switched, then ran a percentage rollout behind the standard three-state flag.

Shift offers are our what-you-see-is-what-you-get pricing guarantee: when a worker views a shift, we record the price they saw so they're paid that amount even if the price moves before they book. Offer creation moved out of the monolith with both systems running in parallel behind the flag, and the mobile API never changed.

Neither migration changed a single price. That was the point: get the request path through the new service while pricing behavior stays provably identical, so that when prices do change, only one variable moved.

Turning the dial

Now the real swap: replace a ten-minute batch repricing system with rules that run at request time. We started embarrassingly simple. If a market's fill targets were being met, pay the market-clearing base rate, clamped between a pay floor and a margin-derived ceiling; otherwise fall back to whatever the old system said. Within a month, this priced about a third of shifts across five metro areas in real time.

Going global from there is where the project's best mechanism appeared. Instead of trusting the new system, we made distrust a rule in the engine: a discrepancy rule that compared the new price to the old one and, if they differed by more than a threshold set in the feature flag, served the old price and logged the full context for investigation.

The threshold started at 2%, meaning the new engine could barely disagree with the old one. Every discrepancy became a debugging session rather than a production incident: a fill-probability bug here, an outlier that needed clamping there, a few missing database indexes. As the discrepancies got boring, we turned the dial: 10%, then 50%, then 90%, each turn letting the new system win more often while fill and margin metrics told us whether it deserved to. A few weeks later we rolled out globally. The launch was an anticlimax, which is what a launch should be: by then the dial had already done the work.

Everything is a rule

By the end, everything was a rule: base pay, margin, holiday boosts, urgent shifts, late cancellations, no-shows, fixed-rate contracts, guardrails, and the discrepancy check itself. We store every rule's output on every offer. "Why is this shift priced this way?", the question that used to eat org-wide time, is now a lookup. Our service tests assert on the full rules output per shift, so pricing behavior is pinned down in CI, not just in production dashboards.

Then we deleted things: the v1 rules engine, the old event-based path, the batch repricing Lambda's remaining consumers, one by one. Eleven months after the first design doc, we marked the project complete.

What it cost

Performance churn ate about two weeks, including a self-inflicted classic: we bumped the new service from 2 to 4 vCPUs to fix latency. We made it worse because Node is single-threaded and our autoscaling triggered at 75% CPU, so the extra cores meant we never scaled out. We ended up at 1 vCPU. I also sequenced the migrations wrong: I assumed real-time pricing would attach to the shift search code and moved it first, but the pricing actually attached to shift offers. The search migration was the right long-term call, and I'd still have made it, just second, with real-time prices shipping sooner.

What it bought

Reliability first: a single, stateless path sets every pay rate, so the class of incident where background jobs fight over prices is gone, and every price explains itself. But the compounding return is the one predicted in that quote up top. While the migration was still in flight, other teams built urgent-shift pricing and multi-shift block pricing on the new engine, and quality features on its visibility rules. Features that used to mean carving a new special case into three services now mean writing a rule.

And the door the old architecture kept shut is now open: prices can respond to current marketplace conditions at request time. Repricing urgent shifts as marketplace demand changes and running auctions for hard-to-fill shifts are now experiments, not architecture projects.

The takeaway, if you're staring at a legacy system everyone is afraid to touch: you don't need a rewrite, and you don't need to flip a switch. State the end goal. Cut milestones small enough that each one answers a question, and be willing to reorder them when the answer surprises you. Run old and new side by side and log every disagreement. Then put your rollout on a dial, and only turn it when the evidence says so.

Rocky Warren
Rocky Warren
Senior Staff Software Engineer
More from Rocky