What We Measure in a Talk (and What We Don’t)
By Noam Shemla · Data & Method
Any system that scores a speech, ours included, is making a set of choices about what's measurable and what isn't, and a fair version of "we're built on data" includes being specific about which parts of a talk that data covers.
What's directly measurable from audio
Several dimensions of delivery can be measured from a recording with reasonable reliability, because they're acoustic properties rather than judgment calls: speaking rate (words per minute, and its variation across the talk), filler-word frequency (a countable event, once a fixed list of filler words and hedging phrases is defined for the language being spoken), pause frequency and duration, and pitch variation (measurable directly from the audio's fundamental frequency track). These are the metrics with the most direct line from "what happened in the recording" to "the number reported", with comparatively little inference in the middle.
Estimates that look like measurements
Other dimensions coaching systems commonly report ("confidence," "clarity," "energy") are not directly observable properties of a sound wave. They're inferences built from a combination of the directly measurable acoustic features above plus, where available, the transcript's language and structure. This is not unique to automated systems: a human coach forms the same kind of inference when they say a speaker "sounded confident," built from the same underlying cues (steady pace, controlled pitch, minimal filler, resolved sentences) rather than from any single directly observable signal called "confidence." The difference to admit is that a number presented as a score can look more objective than it is, because it's a number. The underlying judgment call doesn't disappear just because it's expressed numerically.
Where a metric can mislead if read alone
Filler-word rate is a good example of a metric that's easy to measure and easy to over-interpret. A low filler rate correlates with perceived fluency, but a speaker who eliminates filler words by replacing every hesitation with a long, awkward pause hasn't necessarily improved. They've traded one disfluency for a different one that the filler-rate metric alone won't catch, since it wasn't designed to. Similarly, speaking rate in isolation says nothing about whether the rate was well-matched to content difficulty. A uniformly fast talk and a uniformly slow one can both score as "consistent" on rate variation while missing the real skill of matching pace to content.
Where we do exactly what this article warns about
We should name our own case, not only the industry's. Two of our five scored axes are about what you said rather than how: Structure, which judges whether the talk was built, and Intellect, which the scoring instruction defines as depth of ideas and argument sophistication.
Those are content judgements. They have no acoustic instrument behind them at all. A language model reads the transcript and assigns a number. We ship them because the alternative was to score only the easy things and let a speaker conclude that delivery is what speaking is. But they are the two numbers on the page furthest from anything measured, and they should be read as a well-informed opinion rather than a reading off an instrument.
Intellect has one deterministic half (a count of how many distinct words you used against how many you spoke), and that measure has a known flaw: it falls as a talk gets longer, and it marks down deliberate repetition, which is one of the oldest devices in rhetoric.
Out of scope, on purpose
Whether a joke landed, whether the specific content was true, whether the talk was appropriate for its real audience, whether the room was persuaded. Nothing in the pipeline attempts any of these, and a system that implied it could would be overclaiming in exactly the way the power-posing story and the 93% myth show research claims can drift past what the evidence supports. These dimensions matter as much or more than the measurable ones, and they're better served by peer feedback and human coaching than by a number.
Why we publish this
A methodology is only credible if it's willing to say what it can't measure as well as what it can. That list is as much a part of "built on data" as the metrics themselves.