Understanding Sound from First Principles: Part 3

Introduction

Welcome back! If you have followed us this far, congratulations, because we have already travelled quite a distance. In Part 1, we started with a physical system that vibrates and gradually developed the intuition that sound is a mechanical wave carrying energy through a medium. In Part 2, we moved from the physical world into the digital one and saw how an analog sound signal can be sampled and quantized into a sequence of numbers that a computer can understand.

But there is still a rather fundamental question left unanswered. We have a sequence of numbers representing sound, so what exactly can we learn from those numbers? Looking at the waveform in the time domain tells us how the amplitude changes over time, but it does not always give us the most intuitive picture of what is happening inside the sound. What if we wanted to know which frequencies are present? What if we wanted to distinguish a low-pitched sound from a high-pitched one? What if we wanted to understand why a violin and a flute playing the same musical note do not sound the same?

This is where one of the most beautiful ideas in applied mathematics enters the picture: the Fourier Transform. It gives us another way of looking at a signal. Instead of asking how the signal changes over time, we can ask what frequencies make up the signal. It is almost as if we change the lens through which we observe the same physical phenomenon.

Enough suspense. Let's transform some signals!

The Time Domain

Let us first remind ourselves where we currently are. After sampling and quantization, a digital audio signal can be represented as a sequence of values

Each value represents the measured amplitude of the signal at a particular point in time. If we plot these values against time, we obtain what is commonly called the waveform. This is the time-domain representation of the signal.

The time domain is incredibly useful. We can see when a sound starts and ends, observe sudden changes in amplitude, identify silence, inspect transients and even get a rough idea of the energy contained in different parts of the signal. For speech, we can observe pauses and changes in amplitude as we speak. For music, we can observe the attack and decay of notes.

However, the waveform does not always make the frequency content immediately obvious. Consider two different signals that have exactly the same fundamental frequency but very different shapes. They may correspond to two different musical instruments playing the same note. Looking at their waveforms tells us that something is different, but it may not immediately tell us what is different.

This is where we need another perspective.

From Time to Frequency

Imagine that instead of looking at a complicated waveform directly, we try to describe it using simpler building blocks. What if we could take a complicated signal and decompose it into a collection of sinusoidal waves, each with its own frequency, amplitude and phase?

This is precisely the intuition behind Fourier analysis. A sufficiently well-behaved signal can be represented, or approximated, as a combination of sinusoidal components. In other words, instead of describing a signal only by asking when something happens, we can also describe it by asking which frequencies are present.

This is a profound idea when you think about it. A complicated waveform that looks almost impossible to understand can be represented as a collection of much simpler waves. And this is not merely a mathematical trick. It turns out that the frequency content of a sound tells us a tremendous amount about how that sound was produced and how we perceive it.

Let us return to our good old friend, the sinusoid:

We already know that $A$ describes the amplitude, $f$ the frequency and $\varphi$ the phase. The Fourier perspective asks us to imagine a complicated signal as being composed of many such sinusoidal components. Each component contributes some amount of energy to the final waveform.

Fourier Series: Building a Signal from Sinusoids

For periodic signals, this idea can be expressed mathematically using the Fourier Series. A periodic signal can be represented as a sum of sinusoidal components whose frequencies are integer multiples of a fundamental frequency $f_0$.

Do not let the equation intimidate you. The idea is actually quite simple. We are saying that a complicated periodic waveform can be constructed by adding together sinusoidal waves of different frequencies, amplitudes and phases. The frequency $f_0$ is the fundamental frequency, while $2f_0$, $3f_0$, $4f_0$ and so on are called harmonics.

This is particularly interesting for musical sounds. Suppose a musical instrument produces a fundamental frequency of 440 Hz. The sound may also contain components at 880 Hz, 1320 Hz, 1760 Hz and many other frequencies. The exact combination and relative strength of these components contributes to the characteristic sound, or timbre, of the instrument.

This explains something we intuitively experience every day. A flute and a violin can play the same musical note, meaning they can have the same fundamental frequency, yet they sound completely different. The difference is partly explained by the distribution of energy among the harmonics and other spectral components.

So frequency is not merely about pitch. It can tell us something about the structure and origin of a sound.

The Fourier Transform

The Fourier Series is particularly useful for periodic signals. But most real-world signals are not perfectly periodic and, more importantly for us, audio recordings have finite durations. We therefore need a more general mathematical tool. Enter the Fourier Transform.

The Fourier Transform takes a signal represented in the time domain and transforms it into a representation in the frequency domain. For a continuous-time signal $x(t)$, the Fourier Transform is defined as

The equation looks a little more serious than our sinusoid from Part 1, but the intuition remains approachable. We are essentially measuring how much of each frequency $f$ is present in the signal. The result is $X(f)$, a representation of the signal in the frequency domain.

The mysterious-looking exponential term $e^{-j2\pi ft}$ is not there merely to make the equation look sophisticated. Through Euler's identity,

the complex exponential gives us a convenient mathematical way of representing sinusoidal oscillations. The Fourier Transform therefore uses these complex sinusoids as a basis for analysing the signal.

And here is where things become really interesting. The Fourier Transform does not simply give us a list of frequencies. Its output is generally complex-valued. The magnitude tells us the strength of a frequency component, while the phase tells us how that component is positioned relative to the reference sinusoid.

If we are mainly interested in knowing which frequencies are present, we can look at the magnitude spectrum:

This gives us a frequency-domain representation that is often much easier to interpret for audio analysis.

The Discrete Fourier Transform

There is, however, a small problem. Our computers do not usually receive an infinite continuous-time signal. They receive a finite sequence of sampled values. This brings us to the Discrete Fourier Transform (DFT).

Given a finite sequence of $N$ samples $x[n]$, the DFT is defined as

Again, don't panic. The DFT is essentially doing the same conceptual job as the Fourier Transform, but now it operates on a finite collection of discrete samples. Instead of asking about every possible continuous frequency, we evaluate the signal at a finite set of frequency bins.

The frequency associated with the $k$-th DFT bin is

This equation gives us a very useful intuition. The sampling frequency $f_s$ determines the overall frequency range that can be represented, while the number of samples $N$ determines how finely we divide that range into frequency bins. The frequency spacing between adjacent bins is

Therefore, increasing the number of samples while keeping the sampling frequency fixed gives us finer frequency spacing. But, as we will discover shortly, there is always a trade-off between how precisely we can localize something in time and how precisely we can localize it in frequency. Signal processing is full of these beautiful little compromises.

The Fast Fourier Transform

Now you may be wondering: if the DFT is so useful, why do we keep hearing about something called the Fast Fourier Transform, or FFT? Is it another type of Fourier Transform?

Not quite. The FFT is an efficient algorithm for computing the DFT. The mathematical result is the same; the difference is how efficiently we calculate it.

A straightforward implementation of the DFT requires roughly $O(N^2)$ operations. For a large number of samples, this quickly becomes computationally expensive. The FFT exploits mathematical structure within the DFT to reduce the computational complexity to approximately $O(N\log N)$ for common FFT algorithms.

That difference may not sound dramatic when $N$ is small. But imagine processing thousands or millions of audio signals. Suddenly, computational efficiency becomes very important. This is one of those moments where a clever mathematical insight becomes a practical engineering advantage.

But Audio Changes Over Time

We have now transformed our signal from the time domain into the frequency domain. Wonderful. But there is another problem lurking around the corner.

Real-world sounds change over time.

If I record a person saying a sentence, the frequency content at the beginning of the sentence will not necessarily be the same as the frequency content at the end. A musical performance is even more obvious. Notes begin, evolve and disappear. Instruments enter and leave. Speech moves continuously between voiced and unvoiced sounds.

If we take the Fourier Transform of the entire recording, we obtain information about the frequencies present across the whole recording, but we lose information about when those frequencies occurred.

This is a very important point. The ordinary Fourier Transform tells us what frequencies exist, but it does not directly tell us when those frequencies occur.

And if we are studying speech, "when" matters enormously.

Short-Time Fourier Transform

The obvious idea is: why don't we break the signal into small pieces and perform the Fourier Transform on each piece separately?

Exactly.

This is the intuition behind the Short-Time Fourier Transform (STFT). Instead of analysing the entire signal at once, we take a short window of the signal, compute its Fourier Transform, move the window slightly forward, and repeat the process.

Mathematically, we can express the STFT as

where $w[n-m]$ represents a window centred around a particular position $m$. The result is now a function of both time and frequency.

This is exactly what we wanted. We can now ask not only which frequencies are present, but approximately when they occur.

The Spectrogram

If we take the magnitude of the STFT and display it visually, we obtain what is commonly known as a spectrogram.

A spectrogram is essentially a visual map of the frequency content of a signal as it changes over time. The horizontal axis represents time, the vertical axis represents frequency, and the intensity of each point represents the strength of the corresponding frequency component.

This representation is incredibly powerful for audio. Looking at a waveform, we see amplitude changing over time. Looking at a spectrogram, we can see the evolution of frequency content over time.

For speech, spectrograms reveal rich structures such as harmonics, formants and transitions between phonemes. For music, they reveal notes, harmonics, onsets and other spectral patterns. For environmental sounds, they can reveal characteristic frequency structures associated with different sound sources.

In fact, if you have ever seen an image of a colourful-looking audio waveform where time is on the horizontal axis and frequency is on the vertical axis, there is a good chance you were looking at some form of spectrogram.

The Time-Frequency Trade-off

But wait. We have just introduced another little complication.

Remember our window in the STFT? How large should it be?

If we use a very short window, we get excellent information about when something happens because the window covers only a small period of time. However, we get poorer frequency resolution because there are fewer samples available to estimate the frequencies.

If we use a longer window, the opposite happens. We obtain better frequency resolution because we have more samples to analyse, but we lose temporal precision because the window covers a longer period of time.

This is the famous time-frequency trade-off. We cannot arbitrarily have perfect resolution in both dimensions simultaneously using a fixed-size STFT window.

This is not a failure of signal processing. It is a fundamental property of analysing signals that vary over time. In fact, this limitation is one of the reasons why so many different time-frequency representations and transforms have been developed.

Why Does Any of This Matter for AI?

Now we are finally getting closer to the motivation that started this series. Why should someone interested in machine learning care about Fourier analysis, frequency spectra and spectrograms?

Because machine learning models do not magically understand sound. They operate on representations of sound. The choice of representation can have a profound effect on what information is made available to the model.

We could feed a raw waveform directly into a neural network. This is increasingly common in modern deep learning systems. Alternatively, we could transform the waveform into a spectrogram or another time-frequency representation and allow a model to learn from that representation.

Neither approach is automatically "better." They encode different inductive biases and expose different structures to the learning algorithm. A spectrogram makes frequency-local patterns explicit, while a raw waveform leaves the model to discover those structures itself.

This brings us back to the philosophical point from Part 2. End-to-end learning is powerful, but it does not eliminate the underlying structure of the physical problem. The sound still has to obey the laws of physics. The question is simply whether we explicitly encode some of that knowledge into our representation or ask the model to learn it from data.

Putting It All Together

Let's pause and look at the journey once again.

We started with a physical vibration. The vibration created pressure variations in the air, which propagated as a mechanical wave. A microphone converted those pressure variations into an electrical signal. An analog-to-digital converter sampled and quantized that signal, turning it into a sequence of numbers.

We then looked at those numbers in the time domain. But we realized that a waveform does not always reveal the complete structure of a sound. Fourier analysis gave us another perspective by decomposing the signal into its frequency components.

The Fourier Transform gives us the frequency-domain representation. The DFT makes this idea computationally applicable to finite digital signals, while the FFT provides an efficient way of calculating the DFT. Finally, because real-world audio changes over time, the STFT allows us to examine how frequency content evolves through time, giving rise to the spectrogram.

So our current pipeline looks something like this:

Physical vibration $\rightarrow$ Sound wave $\rightarrow$ Microphone $\rightarrow$ Analog signal $\rightarrow$ Sampling $\rightarrow$ Quantization $\rightarrow$ Digital waveform $\rightarrow$ Fourier analysis $\rightarrow$ Spectrogram

We have now crossed another important abstraction boundary. We are no longer simply looking at sound as a physical phenomenon. We are looking at it as information that can be represented mathematically and computationally.

And this is where things become even more interesting.

A spectrogram is still not the final representation used in many audio systems. We can manipulate it, transform it, compress it, extract perceptually motivated features from it, and eventually use it as an input to machine learning models. One of the most important examples is the Mel spectrogram, which incorporates a frequency scale motivated by characteristics of human auditory perception.

That will be our next rabbit hole.

In Part 4, we will explore how humans perceive frequency, why the linear Hertz scale does not perfectly describe our perception of pitch, and how the Mel scale, Mel spectrograms and MFCCs became fundamental representations in speech and audio processing.

Until then, keep listening. There is considerably more information hidden inside those little waves than meets the eye.