Methodology

The model estimates the probability that a restaurant concept at a given location survives to 4 years, against a citywide base rate of 57.6%. It is an L2-regularised (ridge) logistic regression fit on 15,173 matured location attempts (each kept only once its 4-year outcome is actually observable) with a post-hoc calibrator that adjusts the probability scale without touching the ranking, and that ships only because it improved calibration on held-out years it was never fit on (see below). Every figure in this section is read from the shipped model artifact: the exact file the product scores with.

Read this honestly. An AUC of 0.649 is modest discrimination. This model is directional, not deterministic, and should never be the only input to a lease decision. What it does well is separate the extremes: the gap between top-decile and bottom-decile survival is 41 points, which is a large and actionable difference even when any single prediction carries real uncertainty.

• Out-of-time holdout. The evaluation set is split by time, not at random. Random splits leak the future into training and flatter the model; a time split does not. 2,691 attempts are held out.
• Bootstrap confidence intervals on every reported figure, so the precision of each estimate is visible rather than implied.
• Holm-Bonferroni correction across all 28 candidate hypotheses. Testing many features and reporting the winners produces false discoveries; correcting for the number of tests is what prevents that.
• Every shipped feature is ablated. Each of the 5 features is removed, the model refit, and the difference measured on the held-out years with a paired bootstrap interval. One of them, serviceSig, the concept's own track record, carries the model: removing it costs 0.030 AUC. The others are kept on a different basis. Two of them contribute nothing measurable to ranking but hold the probability scale straight at the weak end, and removing either makes the estimate for a marginal location several points more optimistic than it should be. One is simply too small an effect for a holdout this size to resolve, which is a statement about the test rather than the feature, and is recorded as such.
• The calibrator is kept on evidence, not taste. It was fitted on one held-out slice and evaluated on a later one, where it cut the average gap between stated probability and outcome from 2.9 to 3.8 points across 1,057 attempts. A retrain where that evidence does not hold ships without one.
• Rejected variants are recorded. A conformal prediction interval on the outcome was built and benched. Verdict: correct and useless: for a single binary outcome the 90% prediction set is {closed, open}. Shipped the rate interval instead. And an entire second model class, fit on roughly four times the training signal, failed to beat this one out-of-time; that negative result gets its own section below rather than a footnote.

The Bureau of Labor Statistics measures the same thing by other means. Its Business Employment Dynamics series tracks establishment survival by opening year from payroll tax records, and publishes it for Accommodation and Food Services. Averaged over every opening cohort from 2003, which is also the first year this record is usable, it gives an independent read on the number the score is built on.

At four years, the horizon the model is trained at and the one every number in this product quotes, the two land within about a point of each other.

* The short horizons are ours, not Chicago's. A closure is only counted here once the licence has been lapsed 270 days, and closure recall does not reach certainty until roughly 3 years out. Survival at two years is therefore overstated by construction on our side. The federal series has no such lag. The row is shown rather than hidden, because a gap that size with no explanation invites the flattering reading.

The two series are not the same measurement, and the ten-year divergence is expected. BLS counts employer establishments with a payroll tax account, nationally, across all of Accommodation and Food Services: hotels, bars, caterers and franchised fast food included, all of which outlive independent restaurants. An establishment continuing under new ownership survives in their series. Here a rename or a re-registration reads as a closure, which is worth about 1.4 points on its own. Both differences push this record down relative to theirs and both compound with the horizon. We have not decomposed the ten-year gap and do not claim to know which share is method and which is Chicago.

The count is checked the same way. Survival is half of what this product asserts about Chicago; the other half is how many restaurants there are, and the filter that separates a restaurant from the groceries, pharmacies and school cafeterias sharing its licence category has been missing from a surface before. The Quarterly Census of Employment and Wages counts establishments from payroll tax filings. For Cook County, the smallest federal geography containing Chicago, it records 9,681 restaurant establishments in 2024. We count 7,647 open in the city, or 79% of the county. That is a plausibility band and not an identity, for three reasons the artifact states: the county is larger than the city, payroll records miss a restaurant with no employees, and the federal category excludes the bars and caterers a licence record does not cleanly separate. It is still decisive against the failure it guards. Without the filter, the city alone would report 11,887 open, which is more than the federal government counts in the whole county.

Every feature considered for the model, whether or not it worked. 8 of 28 hypotheses survive Holm correction at α = 0.05, and 6 of those 8 are negative results: features that were significant precisely because they made out-of-time performance worse. Only 2 helped, the largest by 0.0029 AUC.

Restaurant density and transit proximity are the clearest examples. Both are intuitively appealing, both were tested properly, and both reduced predictive accuracy. They are not in the model.

1 of the 2 that helped is not in the model yet: it has cleared validation but not a retrain. This table records what was tested, not what ships. The Shipped marker is read from the production model file, so it cannot drift from what is actually running.

Δ AUC is the change in out-of-time AUC from adding the feature. Negative means the feature made the model worse. “Not significant” means the result did not survive correction for the number of hypotheses tested. It is not evidence the feature has no effect, only that this data cannot establish one.

One caveat about this table itself. Each row is a measurement taken on a particular day against a particular snapshot of the licence record, and until August 2026 those snapshots were overwritten rather than kept, so older rows could not be re-derived from the data that produced them. Re-running everything on current data in August 2026 reproduced every verdict except one, where the original result was significant and the re-run was not. The effect size held; the precision did not. Snapshots are now retained and every run records the date and inputs it used, so a figure here can be checked against the thing it was measured on rather than taken on trust.

This is the most consequential result on the page, and it is a negative one.

The shipped model trains on 15,173 matured attempts: one row per restaurant, kept only once its four-year outcome is actually observable. That throws away every restaurant too young to have reached the horizon, which is a lot of them. So a second model was built to stop throwing them away: a discrete-time hazard model that expands each restaurant into yearly observations, letting a still-open two-year-old contribute real evidence instead of being dropped. It trains on 47,655 attempt-year rows from 14,691 attempts, roughly four times the training signal.

It did not do better. Out-of-time AUC 0.634 against the binary model's 0.635 measured at the same time: a dead heat, correctly benched. The shipped model has since improved to 0.649 on other grounds, which only widens the gap.

Quadrupling the training data moved nothing. That is strong evidence the limit here is not how much history has been collected. It is how much the available signals can say. No larger sample and no fancier model closes that gap, because the gap is not made of data volume. The only routes past it are genuinely new information about a location or an operator, or a second market to test whether any of this generalises at all.

It is published here rather than left in a file because a negative result of this size changes what is worth building next, and because a research programme that only reports what worked is not one whose positive results should be trusted.

The score predicts restaurant failure by location, in a city with a documented history of redlining. So we measured whether it is harsher on poorer or majority-Black neighbourhoods than what actually happened in them, across 57 Chicago ZIP codes with a matured four-year cohort, scoring every ZIP on the same fixed basket of eight common concepts, so that differences in what people open could not masquerade as differences in place.

It is not harsher. It is kinder. In the lowest-income quartile the model predicts survival about 11 points above what attempts there actually achieved. In majority-Black ZIP codes it predicts about 9.5 points above. In majority-white ZIP codes it predicts about 8 points below what actually happened there.

The reason is that location contributes very little to the score. Observed survival tracks median household income closely (r = 0.81) and the share of residents who are Black inversely (r = −0.73); the model’s predictions track those far more weakly (0.30 and −0.49). What it mostly reads is the concept and the operator’s track record, not the neighbourhood.

We also checked whether the result was an artefact of the comparison itself. Holding concept constant instead of matching it inflated the gap by about a third, so the smaller like-for-like figures are the ones quoted above. Operator experience, the other candidate explanation, was ruled out: majority-Black ZIP codes average slightly more prior openings per operator than majority-white ones, not fewer.

That is also an accuracy finding, and we would rather say so: there is real variation in Chicago restaurant survival that this model does not capture. Fitting it would mean scoring restaurants in poor and majority-Black neighbourhoods lower, which is the harm the audit exists to guard against. Our position is that we will not close that gap with features whose main effect is to price neighbourhoods rather than businesses, and that a future change which improves accuracy by widening these gaps should be rejected on that basis rather than shipped on an accuracy score.

The score describes what has happened at a place. It is not a verdict on a neighbourhood or on anyone trading there, it is not a credit model, and it must not be used to decline anyone a lease, a loan or a licence. The full write-up, including the limits of banding ZIP codes by their largest group, is in docs/fairness.md.

• Chicago only. The model is fit entirely on City of Chicago business-license data and has not been validated in any other market.
• License data lags reality. Chicago renewals routinely lag six to twelve months, so a business that has closed can still appear active. Active status carries a 365-day grace window for exactly this reason, which means recent closures are systematically under-counted.
• Cuisine is inferred, not declared. It is classified from business names, license activity codes, and Google categories. It is accurate in aggregate and wrong for individual records.
• The score is optimistic at the weak end, by about 6 points. AUC measures ordering and cannot tell you whether an absolute probability is correct, so the estimate is checked separately against what actually happened on the held-out years. Across all bins the average gap between estimate and outcome is 3.0 points, but it is not evenly spread: the lowest-scoring locations survive somewhat less often than the number suggests, while the highest-scoring ones are accurate to within a point or two. Treat a low score as at least as bad as it reads, never better.
• Modest discrimination. An AUC near 0.649 supports ranking and comparison. It does not support confident single-location verdicts.
• The revenue range holds the real figure four times in ten, not nine. The paid report forecasts revenue as a low-mid-high. We checked how often that range is right, by matching 1,422 Chicago restaurants to their federal payroll-loan records and estimating each one's sales from what it paid its staff. The range contained that estimate 39% of the time, against a ceiling of 50%: the width is built from median errors, so even a perfectly set range would be a coin flip by construction, and the published one is slightly narrower than that. Three things follow. A revenue range here is where the number tends to land, not a bound it stays inside. It cannot be fixed by widening, because reaching nine times in ten would take a range running from below zero to more than double the estimate, so making it tighter means making the estimate better, which is open work rather than a claim. And payroll is a proxy rather than a filed sales figure, which carries error of its own and can only widen the misses we counted, so treat 39% as a floor on how often the range is right rather than a precise score.
• Having a Google listing is partly a consequence of having survived. The concept cohorts behind the model's strongest feature are built partly from Google listings, and that snapshot is current: a restaurant that closed years ago is not on the map today. Among the 20,959 attempts the model is fit from, only 27.5% carry a Google match, and those that do survived four years 96.0% of the time against 49.2% for those that do not. That gap is mostly the listing following the survival, not predicting it, which makes the held-out evaluation easier than reality. Withholding Google-derived classification entirely and re-fitting costs close to half the model's whole margin over a coin flip. That is a floor rather than an estimate, because you tell us your cuisine and format directly so your own concept is never guessed at; but the headline AUC above was measured under better conditions than a first-time operator at an unlisted address enjoys.
• Chains are better documented than independents, and it tilts the score. Google records chain locations thoroughly and informal independents patchily. Across the 44 concept cohorts large enough to measure, coverage runs from 10.2% in the worst-documented quartile (Hot Dog / Chicago Classic and restaurant (unspecified) places) to 57.3% in the best, which is almost entirely fast-food formats. Because a cohort's survival rate is assembled from whoever is documented, well-covered concepts are lifted relative to poorly-covered ones: about 3.4 score points between those quartiles. We publish this rather than correct it, deliberately: every available correction costs more accuracy on the strongest feature than the tilt is worth, and the measurement is re-run whenever coverage changes. If you are opening an independent in a format Google barely records, read your score as slightly conservative.
• Survivorship in the source. Attempts are reconstructed from license records, so operators who never obtained a license are invisible to the model.
• An era assumption sits under every score. The fit includes an opening-year term so a 2003 opening is compared fairly against a 2021 one. At prediction time that term is pinned to a single year: 2021, the most recent opening the model can score an outcome for. The choice matters: across the plausible range of that one constant the estimate for an average concept moves from 58% to 58%. It is pinned inside the range the model was trained on rather than extrapolated past it, and the most recent matured cohorts are checked against their own predictions. The largest gap is 4.8pp. This assumption is invisible to AUC, which is why it is stated here rather than left to a validation figure to catch.

Fee, investment and unit-economics figures for 75 franchise brands are read from each brand's own current Franchise Disclosure Document, as registered with the Wisconsin Department of Financial Institutions, a franchise registration state that publishes complete FDDs publicly. Every figure is read against a cited page of the filing rather than parsed, because FDD table layouts vary enough between franchisors that a parser cannot be trusted with a number that ends up in a cost estimate.

Item 19 is optional, and 26 of these brands decline it entirely. Where it does exist there is no required format, so the disclosed population differs sharply between brands: franchised outlets versus company-operated, one build format versus the whole system, captive venues versus street-front, and samples ranging from two outlets to twelve thousand. Every figure therefore carries a description of what it covers, and that description travels with the number wherever it is shown. Payback estimates are refused outright when a brand's Item 7 and Item 19 describe different populations, since dividing one into the other yields a confident-looking number with no meaning.

Chicago outcomes for these brands are computed from the same licence history and the same corrected closure labels as the survival model above, so they are directly comparable to every other survival figure in the product. They are descriptive: a brand's Chicago record reflects where it chose to open and who operated those locations, and is not evidence that the brand causes survival.

Registry-backed franchise detection was tested as a model improvement and rejected. See the hypothesis ledger above. It corrects a real labelling gap but does not improve out-of-time accuracy.

License and inspection data refreshes daily; Google verification and rent bands refresh monthly, on a separate cadence. Both timestamps are shown so neither is mistaken for the other. Franchise registrations renew annually and are re-swept quarterly.

Chicago licenses

never

Google enrichment

never

Every shipped model is identified by a content hash, 7fc31479116f1865 for the model serving today, and that hash is recorded alongside every scored decision, so any score a customer acted on can be traced to the exact model that produced it.

The model serving today was fitted on attempts through 2026-08-29. The distance between that date and the day a report is read is drift?: the population moving while the coefficients stay fixed. It is watched by re-scoring a fixed reference cohort against the current licence record: because the cohort is held constant, any movement is the world changing rather than the code.

The model retrains quarterly, after the monthly data refreshes have settled. A retrain ships only with a published validation delta against the outgoing version. If accuracy does not hold out-of-time, the old model keeps serving.

Issued reports are never retroactively changed. A report is a record of what a specific model version said on a specific date; when the model updates, existing reports stand as issued, and re-running the same address produces a new report under the new version stamp.