Introducing LarkBench

LarkBench tests how well AI models handle a simulated distributor buyer's job: what to order, how much, and when, week after week, while demand moves around and vendors ship late. We want to know two things: how much profit today's models leave on the table, and whether they beat representative ordering rules.

1Motivation

At a distributor, the buyer decides what's on the shelves. Margins are thin and stock ties up a lot of cash, so mistakes cost real money either way: buy too little and customers go elsewhere, buy too much and the money sits in the warehouse. Many distributors let their ERP suggest orders with a reorder point, which just says "when stock drops below this level, order more." A good buyer earns their keep on everything that rule misses: a truck that's late or never comes, demand that shifts for good, a big one-off job, an item the vendor is dropping, a rep's tip that turns out to be wrong.

LarkBench tests how AI models handle realistic purchasing scenarios across a vendor’s full product line, including business constraints and edge cases such as order minimums, free-freight thresholds, pack sizes, spending limits, late deliveries and bad data. Each model manages 15 to 30 products at a simulated distributor. We measure how much profit it keeps after costs and compare its results with rule-based buyers.

2Benchmark design

2.1 Environment

A book is one vendor's product line at a simulated distributor: 15 or 30 items, each with at least a year of sales history, normal starting stock and some open POs. A season is 26 weeks, with one ordering decision a week. Customers order every week. If something is out of stock the order waits, it may get cancelled, and a customer who gets shorted too often stops ordering. The vendor has an order minimum, a free-freight threshold, pack sizes and lead times, and shipments can come late, come short, or not come at all. The 30-item books also have a weekly spending limit.

The benchmark covers a broad set of real-world purchasing problems and edge cases. The scenarios draw on public statistics, research papers and trade reporting, trends we have observed among our customers, and conversations with more than 50 distributors. We set how often they occur in the simulator using published numbers where available and stated assumptions otherwise.

Each scheduled disruption leaves a clue in the information the buyer can access, such as a change in sales, a vendor notice, or an unconfirmed purchase order. Across the benchmark, products go through a full 26-week season without a scheduled disruption about half the time.

2.2 Model interface

Each week, the model reviews a simulated ERP screen showing inventory, open orders, recent sales and suggested purchases. It can investigate with look-ups and send vendor questions, answered the following week. It then places one PO and writes a note for next week. All models have the same interface and limits within each setup.

We compare three setups:

• With look-ups: the model decides what to order using the screen and tools.
• Formula → AI: it also gets a suggested order from our rule-based buyer.
• One screen: current records and four weeks of notes are supplied up front. There are no look-ups, but vendor questions remain available.

2.3 Scoring

Demand margin is the margin on all customer orders at list prices and costs. Net profit is earnings after costs, including an adjustment for the inventory, open orders and waiting customers left at week 26.

The adjustment uses another 26 weeks of simulated trading to account for future overstock and shortages. Losses count in full, and scores can exceed 100%.

All 160 books contribute to our difficulty-weighted score. We give 40 harder books three times the weight, selected using separate calibration runs. Books with larger demand margins also contribute more.

2.4 Baselines

We compare the models with three non-AI purchasing strategies:

• ERP reorder point. Automatically reorders based on stock, recent sales and delivery times.
• Careful rule. A hand-written buyer built from inventory formulas and purchasing rules. It estimates demand, sets a reorder point with a safety buffer, and orders toward a target stock level. We tuned it on separate practice scenarios.
  Show formula

  C = max(1, L) + 7

  s = μC + min(1.65σ√C, 21μ)

  Q = √(2K × 365μ / (c × h))

  S₀ = s + max(Q, M)

  q₀ = max(S₀ − IP, 0) if IP ≤ s; otherwise 0

  This gives the starting order. The rule then applies stock limits and a check on potential write-offs, and adjusts for freight, vendor minimums, promotions and pack sizes. It also handles stale orders and recorded substitutes.

  What the symbols mean

  μ, σ

  Estimated daily demand and its variability.

  L, C

  Delivery time in days, and delivery time plus the weekly review period.

  s, S₀

  Reorder point and initial target stock level.

  Q

  Economic order quantity, balancing ordering and holding costs.

  K, c, h

  Cost per order, cost per unit, and annual inventory carrying-cost rate.

  M

  Minimum quantity for the item, in stocking units.

  IP

  Stock on hand + incoming orders − allocations − backorders, after data clean-up.

  q₀

  Initial order quantity, before the adjustments above.

• Screen-only planner. Simulates possible futures using demand and delivery estimates from the ERP screen, without look-ups or notes.

The screen-only planner provides a reference score; it is not a proven optimum.

3Results

Scores show how much of the margin on all customer orders the buyer keeps after costs. 100% means keeping the full margin at the listed buying and selling prices; 63.6% means $63.60 in net profit for every $100 of that margin. Discounts can push scores above 100%.

Table 1. Difficulty-weighted scores on the same 160 books × 3 draws. Each AI model completed one full run using look-ups.

Sol leads the other models and the careful rule under the current scoring method. Opus and Gemini score close to the careful rule, with no clear difference in this evaluation. All three models outperform the ERP reorder rule.

Figure 2 compares three setups on the same 50 books. Rule-based suggestions raised the recorded scores modestly. Showing records up front helped Opus in this run and made little difference for Sol and Gemini. The setups supply different information, so this comparison does not isolate the value of tools.

Run-to-run variation

Table 2. Each model played the same 20 books × 3 draws three times, with Run 1 taken from the main evaluation. Scores use the same difficulty weighting. These books are easier than the full set, so compare scores within this table.

4Where the profit goes

After the runs, we analyzed where the models lost sales and incurred costs that reduced their profit. We split what each model didn't keep into five parts: sales missed because the shelf was empty, the cost of holding stock, customers kept waiting, other operating costs (ordering, freight and fees, net of discounts), and the end-of-season adjustment from section 2.3, which charges for excess stock and for a short shelf alike.

Per $100 of margin on customer orders, all three models lose about $10–$12 to missed sales. Holding stock costs Sol $10.00, Opus $12.90 and Gemini $13.50. End-of-season adjustments cost another $4.60 for Sol, $8.00 for Opus and $10.30 for Gemini, reflecting both excess stock and shortages. The logs show several recurring mistakes:

1. Buying years of stock at once. In one season a model meant to buy about 64 fittings, ordered 64 cases of 12, and lost money on the season. Orders with more than a year's supply of an item come up 3.4 times a season for Gemini, 1.4 for Opus and 1.3 for Sol; about a third of them were ordered in cases.
2. Cutting back too little when demand drops. Starting four weeks after a lasting demand drop, the median ordering pace is 30% of its old level for Opus, 42% for Gemini and 55% for Sol. The careful rule also stays at 55%. Of the 129 drops eligible for this comparison, 59 (46%) have a linked note that correctly describes the drop; notes can also be absent or wrong.
3. Ordering the same item over and over. Big reorders while there's still plenty on the shelf or on the way. One model ordered the same item five times in one season, 720 units in total. Ordering one item 4 times in 5 weeks happens 1.4 times a season for Opus, 0.9 for Gemini and 0.3 for Sol.

6Limitations

• Scope. Each simulation covers one vendor, 15–30 items and 26 weeks. Performance in a live distributor has not been tested.
• Coverage. We tested three models at one reasoning setting, with one full run each. Only 20 of the 160 books had repeat runs; the reported intervals do not capture run-to-run variation.
• Assumptions. Results depend on the simulated economics and scoring rules. Some comparisons change under alternative scoring, and we have not tested other plausible event rates.
• Known issue. Some substitute descriptions contain catalogue-number markers that could help models identify them. We have not measured the effect on scores.

Next, we plan to evaluate the 52-week version already built and test more models.

• Test a model: send it to us and we'll run it on the secret books. Write to jason@larkagents.com.