Where Our Instrument Broke

By Noam Shemla ·

By Noam Shemla · Speeches That Lasted

TL;DR We measured the speech President Obama gave at Selma on 7 March 2015 with the same instrument our members record into. It returns 115 words a minute, which our pace band scores at 50 out of 100, the lowest mark the scale can give. That score says more about our scale than about the speech.

Building a tool that scores speaking carries a quiet risk. The tool can be wrong, of course, but the bigger danger is that nobody ever tests it against something already known to be good.

So we did. We took an address widely held to be among the finest of a modern presidency, ran it through the instrument our members’ recordings go through, and committed the output to our own repository before writing a word of this. Every figure below is read from that file. Where the instrument produced nothing, we say so instead of reaching for an estimate.

Why this speech, and why we can quote it

Two reasons, and the second is the duller one that decided it.

The interesting reason: this speech is long, deliberate, and delivered to a crowd that interrupts. Nothing about it resembles the three-minute recording our tool was designed around. If our numbers were going to break, they would break here.

The dull reason: a speech given by a sitting US president in his official capacity is a work of the federal government, and so carries no copyright. We can quote it in full, at length, without asking anyone. That is not true of most of the great speeches of the last century. A good deal of the canon is still owned, some of it until the 2030s, and that shapes what an article like this can show. Where we cannot quote, we will publish structure and measurement only, and say plainly why the words are missing. Here we can quote freely, so we will.

The number that comes out wrong

The instrument reports 3,748 words across 32 minutes and 43 seconds. Divide one by the other and you get 115 words a minute.

Here is what our own scale does with that.

Fifty out of a hundred, for the Selma address. If a member saw that on their own recording they would reasonably conclude they had done something wrong.

What the clock is measuring

The arithmetic is fine. The problem sits in the denominator.

Words a minute divides words by elapsed time. It has no concept of whether anyone was speaking during that time. The official transcript of this speech (anyone can count for themselves at the link at the end) marks applause thirty separate times. Some of those ovations are long. All of them are counted by our clock as though the speaker were still going.

So 115 words a minute is a true statement about the recording and a false statement about the speaker. He was not speaking slowly. He was speaking, and then waiting, and then speaking again. A room of that size does this to anyone worth listening to. Our instrument, built for someone alone at a laptop with no audience to wait for, has no way to tell those two situations apart.

The fix here is not obvious. Excluding applause would need the tool to distinguish a crowd from a voice, which it does not attempt. What we can do, and have done, is stop presenting the resulting score as though the scale applied.

You do not have to take our word for what thirty ovations inside one speech sound like. The recording is below.

What the instrument reads well

Sentence length needs no recording, no timings and no audio. It is simply how the speech is built, and it is where the measurement gets interesting.

The shape of that chart is the argument against judging a speech by its average anything. The 64-word sentence at the centre is a single sustained question:

What greater expression of faith in the American experiment than this, what greater form of patriotism is there than the belief that America is not yet finished, that we are strong enough to be self-critical…

And the shortest sentence in the entire address is two words long:

That’s America.

A tool reporting "mean sentence length 17.6" has described neither of them. The variance is the craft. The average is what is left after you throw the craft away.

The litany

Structure also exposes repetition. Of all the rhetorical devices, repetition is the one you can locate without quoting anything, and that is what makes this series possible even for speeches we are not allowed to reproduce.

Look at where the marks cluster. Almost nothing in the first half; then, from about sentence 145 onward, the lanes stack up on top of each other. That is a peroration being assembled in plain sight, and the eight consecutive "We’re the" openings between sentences 160 and 170 are the centre of it:

We’re the immigrants who stowed away on ships to reach these shores… We’re the slaves who built the White House and the economy of the South… We’re the fresh-faced GIs who fought to liberate a continent.

Nine such openings clear our threshold of three repetitions. We show six, because a seventh lane makes the figure taller than the column it sits in. We would rather say so than quietly show you six and let you assume there were only six.

What we could not measure, and why we are not filling it in

Our instrument measures pitch and volume four times a second. On this recording it took 7,853 readings, and the whole timeline sits in the committed file.

And yet this article contains no claim about how Obama’s voice moved. Not one.

The reason is a real limitation, and it would have been easy to hide. Turning readings into a statement (average pitch, how much it varied, whether a delivery reads as monotone, how many pauses were deliberate) requires knowing when each word was spoken, and the official transcript is prose with no timings in it. Our recording pipeline gets word-level timings from speech recognition; a historical transcript has none. So the readings exist and the conclusions do not.

We could have averaged the 7,853 numbers and printed a figure. It would have looked authoritative. It would also have silently included every second of applause, every introduction, and every gap, as though they were the speaker’s voice. The honest output is a gap, so the gap is what we are publishing.

  • 3,748 words. Counted by the same tokeniser that counts a member’s practice recording.
  • 32m 43s elapsed. Against a tool designed for recordings of up to 3 minutes.
  • 210 sentences. Mean 17.6 words; range 2 to 64.
  • 38 filler words. One percent of everything he said. "So" accounts for 16 of them.
  • 3 hedges. A tenth of one percent. Almost nothing is qualified away.
  • 7,853 sound readings. Of which our member pipeline would have kept the first 1,000.

The full set of numbers this recording produced, including the one the instrument would have thrown away.

Two of those deserve a note.

Thirty-eight filler words in 3,748 is one percent, and it produces a clarity-from-fillers score of 93.7 out of 100. Sixteen of the thirty-eight are the word "so", which in this speech is mostly doing real work as a connective rather than buying time. Our filler list cannot tell those apart either, and that caveat applies just as much to a member’s recording as to this one.

Three hedges in the whole address is the number that impressed us most, and it is the one most transferable to anyone reading this. Across thirty-two minutes on a contested subject, almost nothing is softened with "sort of" or "I think maybe". That is a choice, it is visible in the measurement, and it is available to anyone.

Where the instrument is out of its range

Every corpus entry we generate carries a plain-language list of the places its own thresholds do not apply. For this speech that list came out as three items, and we would rather print them than bury them:

That third line is the whole article in one sentence, written by the tool about itself before we had drawn a single chart.

Three more recordings, and the one everybody remembers

We later measured three more Obama recordings: the 2004 convention keynote, the 2008 address on race, and the Charleston eulogy of June 2015. He covers the widest range of occasion types of anyone in this corpus, and the instrument responds to it sharply.

The eulogy is the one where he paused for a long moment and then began to sing. By most accounts it is the most extraordinary thing he did with a microphone, and our instrument scores it his lowest of the four. The measurement is doing its job here, because a eulogy really is built differently from an argument. We are back at the lesson this article opened with, arriving from the other side: a score tells you how well an instrument fits an occasion.

If you are practising your own talk

Keep your pace score. On a two-minute practice recording with no audience, that score measures what it claims to measure, and 115 words a minute really is slow for that setting.

Take this instead: a score is a measurement inside a context, and it stops meaning anything the moment the context changes. The most admired speech we could find scores 50 on our pace scale. If your talk has an audience that reacts, pauses that you are holding on purpose, or a length past a few minutes, the number on your dashboard is answering a question you did not ask.

The parts that survived the change of context were the ones about structure: sentence variance, repetition, how little he hedged. Those measure the same thing at two minutes and at thirty-two. They are also the parts you can practise.

The same pitch and volume analysis our members record into, ported to run on an audio file, and verified to return identical values on identical input.

Measured

  • Word count, and words per minute against elapsed time
  • Filler and hedging counts and rates, from the text
  • Vocabulary diversity: 38.8 percent of words distinct
  • Sentence count and the length of every sentence
  • Positions of every opening that repeats three times or more
  • Pitch and volume readings across the full recording

Inferred, not measured

  • Nothing about how the voice moved: average pitch, variation, monotone and pause counts all need word-level timings, and a prose transcript has none
  • Sentence boundaries, which follow the transcript’s punctuation rather than the speaker’s breath
  • Applause is not separated from speech, so elapsed time, and therefore pace, includes every ovation

Sources

There is a further reason to distrust the pace number here. The persuasion research our band comes from found faster speech more persuasive in a specific setting (a recorded message, a listener with no stake, a topic they had not thought about)1, and later work showed the effect flips depending on whether the audience already agrees2. A crowd on a bridge in Selma is not that listener.

References

  1. Miller, N., Maruyama, G., Beaber, R. J., & Valone, K. (1976). Speed of speech and persuasion. Journal of Personality and Social Psychology, 34(4), 615–624. https://doi.org/10.1037/0022-3514.34.4.615
  2. Smith, S. M., & Shaffer, D. R. (1995). Speed of speech and persuasion: evidence for multiple effects. Personality and Social Psychology Bulletin, 21(10), 1051–1060. https://doi.org/10.1177/01461672952110006