Using a backtest to choose Dynamic Pricing software | Perfect Price

Dynamic Pricing software takes data and turns it into actionable real-time prices. At Perfect Price, we’ve learned a lot about how executives assess and decide on using our software. We think this process could work well for evaluating any software that an enterprise is trusting to make decisions on its behalf.

The problem
As a CFO or CMO deciding to present a new decision-making process to your executive peers, you need confidence that it will work. There is inherent risk in adopting any new business method, especially enterprise software. Will the new pricing system outperform your current strategy? Will it throw operations into disarray? This is just as true for, say, an anti-fraud system. Will it prevent as much fraud as the existing method? Will it reject more legitimate customers? Answers to questions like these are a critical step in the decision process.

Vendor risk is real, as well. Many pre-internet vendors promise rapid timelines–but can take years to deliver. PROS, for example, took 7 years to deliver an upgrade for Avis. Hertz has been working on a similar upgrade for at least 4 years, according to annual reports, with nothing to show for it.

A CEO of a public US hotel company told us that when he chose a pre-internet vendor, he had no idea that implementation would take 4 years, cost $20 million, and require a team of 30 new pricing hires to actually use the software. “If their sales guy told me that up front, I never would have signed up,” he said.

How do you mitigate decision risk and vendor risk before you have committed to a seven-figure agreement?

Backtests and their side effects
When data scientists at Google, Amazon, Uber and other leading companies try something new, they backtest it. That means they run simulations using the new approach and the old approach, comparing the results. Where did the new method show improvements? Where was it inferior?

This is called a backtest, and it is the only way to have confidence in a new model or solution before deploying it to real customers.

A backtest consists of several phases, and as you’ll see momentarily, there are very positive side effects of this process. (This assumes you have already aggregated your data into a data warehouse of some kind–which most large enterprises have already done.)

Data Onboarding and Integration

Data Analysis

AI Training and Modeling

Review

Presentation