Understanding Sound from First Principles: Part 5

Introduction

We have finally arrived at the part where things start getting particularly interesting for me. If you have been following this series from Part 1, we have travelled from the physics of sound all the way to computational representations of audio. We started with vibration and pressure waves, moved through sampling and quantization, explored the Fourier Transform and spectrograms, and eventually introduced perceptually motivated representations such as Mel spectrograms and MFCCs in Part 4.

Now we have a digital representation of sound. But what do we actually do with it?

Suppose I give you thousands of short recordings and tell you that some contain a dog barking, some contain a piano, some contain human speech, and others contain rain. Could we build a machine that learns to distinguish between them?

Absolutely.

This is where machine learning enters our story.

But before we jump headfirst into neural networks, we need to understand how an audio problem becomes a machine learning problem. The machinery of machine learning is not particularly interested in whether the numbers originated from air pressure, pixels, financial markets or brain recordings. It sees vectors, matrices, tensors and probability distributions. Our job is to construct a meaningful bridge between the physical phenomenon and those mathematical objects.

Let's build that bridge.

From Audio to a Machine Learning Problem

Let us begin with a simple example. Imagine that we want to build an instrument classifier. We have recordings of a piano, violin, flute and guitar, and we want our machine learning system to predict which instrument produced a new recording.

Conceptually, our dataset might look something like this:

Audio Recording $\rightarrow$ Representation $\rightarrow$ Label

For example, one recording may be represented by a waveform or a spectrogram and associated with the label "piano." Another may be associated with "violin." The machine learning algorithm's task is to learn a mapping from the input representation to the corresponding label.

Mathematically, we can think of this as learning a function

where $X$ represents our input data, $Y$ represents the target labels, and $\theta$ represents the parameters learned by the model.

During training, the model sees examples of $X$ together with their corresponding $Y$ and attempts to learn parameters $\theta$ that allow it to make accurate predictions on examples it has never seen before.

That last part is extremely important.

The goal of machine learning is not to memorize the training data. The goal is to generalize.

Features, Labels and Examples

Traditionally, one of the first things we would do when applying machine learning to audio is to extract a collection of meaningful features from the signal. In Part 4, we encountered MFCCs. These are a perfect example.

Suppose an audio recording is converted into a vector of features:

Each $x_i$ represents some measurable property of the audio. Depending on the problem, these could include MFCCs, spectral centroid, spectral bandwidth, zero-crossing rate, chroma features, energy, pitch-related features or other descriptors.

The corresponding target might be

where $C$ is the number of classes.

For our instrument classifier, $C=4$ if our classes are piano, violin, flute and guitar.

The dataset therefore becomes a collection of examples

where $N$ is the number of examples in the dataset.

This is the basic language of supervised machine learning. We provide examples of inputs and their corresponding answers, and the algorithm tries to learn the relationship between them.

But Where Do Features Come From?

This is where our previous four posts suddenly become relevant.

A feature does not magically appear from nowhere. It is extracted from some representation of the original physical signal. Consider the journey we have already travelled:

Sound $\rightarrow$ Waveform $\rightarrow$ STFT $\rightarrow$ Spectrogram $\rightarrow$ Mel Spectrogram $\rightarrow$ MFCC $\rightarrow$ Feature Vector $\rightarrow$ Machine Learning Model

Each arrow represents an operation that changes the representation of the information.

This is why understanding signal processing is so important when working with machine learning for audio. If you do not understand how the representation was created, it becomes difficult to understand what information the machine learning model is actually receiving.

For example, an MFCC is not simply "some numbers." It is the result of a sequence of assumptions and transformations involving framing, windowing, Fourier analysis, perceptual frequency scaling, logarithmic compression and the DCT.

Once you understand that, an MFCC becomes considerably less mysterious.

The Classical Machine Learning Approach

Historically, audio classification systems often followed a pipeline that looked approximately like this:

Audio $\rightarrow$ Preprocessing $\rightarrow$ Feature Extraction $\rightarrow$ Feature Selection $\rightarrow$ Classifier $\rightarrow$ Prediction

Suppose we want to classify spoken words. We might first divide the recording into short frames, compute MFCCs for each frame, summarize the resulting sequence using statistical features, and then train a conventional classifier such as logistic regression, a support vector machine, a random forest or a Gaussian mixture model.

The interesting thing here is that humans are doing a considerable amount of the representation design.

We decide what properties of the signal might be useful. We decide how to transform the waveform. We decide which features to retain. Then we give those features to the machine learning algorithm.

This approach can work extremely well, particularly when the dataset is relatively small and the domain knowledge is strong. But it also has an obvious limitation: we may throw away information before the learning algorithm ever sees it.

Representation Learning

This is where the idea of representation learning becomes important.

Instead of manually specifying all the features that a model should use, we can ask the model to learn useful representations directly from the data.

Raw Input $\rightarrow$ Learned Representation $\rightarrow$ Prediction

This is one of the central ideas behind deep learning. A neural network contains layers of transformations that can progressively convert a low-level input representation into increasingly abstract representations.

For an audio signal, the early layers may learn relatively simple patterns. Later layers can combine those patterns into increasingly complex structures. In speech, this could eventually correspond to phonetic or linguistic information. In music, it could correspond to harmonic structure, rhythm or instrument characteristics.

The remarkable thing is that we do not necessarily have to explicitly tell the model what a phoneme, harmonic or spectral pattern looks like. The model can discover useful representations through optimization.

Of course, this does not mean that the model has suddenly developed an understanding of sound in the human sense. It has learned statistical representations that are useful for the task it was trained to perform.

Why Spectrograms Became So Popular

Now we have a particularly interesting problem. We know that a spectrogram is a two-dimensional representation: time along one axis and frequency along the other. This makes it look surprisingly similar to an image.

And researchers noticed something important.

If a spectrogram looks like an image, perhaps image-processing techniques can be used to process it.

This is one of the reasons convolutional neural networks became so popular in audio classification.

A convolutional neural network, or CNN, was originally developed with image-like data in mind. It learns local patterns using convolutional filters. A filter may respond strongly to a particular spatial pattern in an image. When applied to a spectrogram, those spatial patterns become time-frequency patterns.

A CNN can therefore learn filters that respond to local structures in an audio representation. A particular filter might respond to a harmonic pattern, a transient, a frequency transition or some other structure that is useful for the task.

This does not mean that the CNN "sees" a spectrogram exactly as a human sees an image. The important point is that the mathematical structure of a spectrogram makes convolution a useful operation.

Convolution: A Very Simple Intuition

Let us simplify convolution as much as possible.

Imagine placing a small matrix, called a kernel or filter, over a larger matrix. We multiply corresponding values, add them together, and then move the filter to another location. Repeating this process produces a new representation called a feature map.

In a neural network, the values inside these filters are learned from data rather than manually specified.

For a two-dimensional input $X$ and a kernel $K$, a simplified discrete convolution-like operation can be written as

The exact mathematical operation used in most deep learning libraries is technically a cross-correlation rather than the strict mathematical definition of convolution, but the term convolution is so deeply established in deep learning that we will happily continue using it.

The important idea is not the equation itself. It is that the same small set of learned parameters can be applied across different locations. This gives CNNs an ability to detect local patterns regardless of where those patterns occur in the input.

From Spectrograms to CNNs

Suppose we have a Mel spectrogram represented as a matrix:

where $H$ represents the number of frequency bands and $W$ represents the number of time frames.

A CNN can process this matrix using learned filters. The first layers may detect relatively simple local structures. Deeper layers combine these structures into more complex patterns.

For example, imagine a short sound containing a harmonic structure. The spectrogram may show several horizontal bands corresponding to related frequencies. A convolutional filter can learn to detect this arrangement. Another filter might learn to detect a rapidly changing frequency pattern. Another might respond to a transient.

The network does not need us to manually define these filters. During training, the filters are adjusted so that the resulting representation becomes useful for predicting the target labels.

The Learning Part

But how does the network actually learn these filters?

The answer takes us to one of the most important ideas in modern machine learning: optimization.

Suppose our model receives an input $x$ and produces a prediction $\hat{y}$. We compare this prediction with the true label $y$ using a loss function.

For a classification problem, one common choice is categorical cross-entropy:

where $C$ is the number of classes, $y_c$ represents the target indicator for class $c$, and $\hat{y}_c$ is the model's predicted probability for that class.

The objective of training is to find model parameters that minimize the loss over the training data.

This is where gradient-based optimization enters the story. The model calculates how changing each parameter would affect the loss and then updates its parameters in a direction that tends to reduce that loss.

The process is repeated over many examples and many iterations. Eventually, if everything goes well, the model learns parameters that allow it to recognize useful patterns in the data.

Training, Validation and Test Data

There is a trap here, however. A model can become extremely good at predicting the examples it has already seen without actually learning a general rule.

This phenomenon is called overfitting.

Imagine showing a student the exact questions that will appear on an exam and allowing them to memorize the answers. They may achieve an impressive score on those questions but perform terribly when faced with slightly different questions.

Machine learning models can do essentially the same thing.

This is why datasets are commonly divided into separate subsets. The training set is used to learn the model parameters. The validation set is used during model development to make decisions such as selecting hyperparameters or comparing models. The test set is held aside for the final evaluation.

Training Data $\rightarrow$ Learn

Validation Data $\rightarrow$ Tune

Test Data $\rightarrow$ Evaluate

The distinction is important because the test set should provide an estimate of how the model performs on previously unseen data. If we repeatedly optimize our model based on the test set, it is no longer truly an independent evaluation set.

Accuracy Can Be Deceptive

Now suppose we have a dataset where 95% of the recordings belong to one class and only 5% belong to another. A model that always predicts the majority class would achieve 95% accuracy.

Sounds impressive, right?

It is actually useless for identifying the minority class.

This simple example illustrates why evaluation metrics matter. Depending on the application, we may care about precision, recall, F1-score, sensitivity, specificity, balanced accuracy, area under the ROC curve or other measures.

The choice of metric should reflect the scientific or practical question we are trying to answer. There is no universally superior metric.

This becomes particularly important when we move beyond toy audio classification problems into domains such as medical speech analysis, clinical decision support or other safety-critical applications. A model that achieves a high overall accuracy can still fail catastrophically for the cases that matter most.

What About Raw Waveforms?

At this point, someone might reasonably ask: why bother with Mel spectrograms, MFCCs and all these transformations? Why not just give the raw waveform to a neural network and let it figure everything out?

This is a very reasonable question, and modern deep learning has increasingly demonstrated that it is possible.

A raw waveform is simply a sequence of amplitude values:

A neural network can operate directly on this sequence using one-dimensional convolutions, recurrent architectures, transformers or other mechanisms.

The advantage is that we do not explicitly impose the entire traditional signal-processing pipeline. The model has the opportunity to learn useful representations directly from the waveform.

The disadvantage is that the learning problem can become considerably more difficult. The network may need to discover low-level structures that signal-processing theory already tells us something about. It may also require substantially more data and computation to learn useful representations.

Again, there is no free lunch.

A spectrogram gives the model a strong inductive bias about time-frequency structure. A raw waveform gives the model greater freedom but potentially a harder learning problem. The best choice depends on the task, dataset, computational budget and desired properties of the system.

End-to-End Learning

This brings us back to something I briefly mentioned in Part 2: end-to-end learning.

In a traditional audio system, we might explicitly design the entire pipeline:

Waveform $\rightarrow$ MFCCs $\rightarrow$ Classifier $\rightarrow$ Prediction

In an end-to-end deep learning system, we might instead have

Waveform $\rightarrow$ Neural Network $\rightarrow$ Prediction

The distinction is not that one approach uses mathematics and the other does not. Both are mathematical. The difference is largely in where we place the representation engineering.

In the classical pipeline, humans explicitly construct more of the representation. In an end-to-end system, the model learns more of the representation jointly with the task.

And this is one of the reasons deep learning became so powerful. Instead of asking a domain expert to specify every useful feature, we can provide enough data and an appropriate learning architecture and allow the model to discover representations that minimize the task objective.

But Does the Model Understand Sound?

This is where I think we should slow down for a moment.

We often use phrases such as "the model understands speech" or "the network understands music." These statements are convenient, but they can be misleading.

A trained model has learned statistical relationships in its training data. It has discovered patterns that allow it to make useful predictions. Whether that constitutes "understanding" depends heavily on what we mean by understanding.

This distinction becomes increasingly important as models become larger and more capable. A model may recognize a word without understanding its meaning. It may identify an instrument without knowing what an instrument physically is. It may classify a cough without possessing any human concept of illness.

The machine is operating on representations.

And this brings us back to the first principle of this entire series: what exactly are those representations?

From Features to Learned Representations

The classical approach might give the model a carefully engineered vector of features. Deep learning instead allows successive layers of the network to transform the input into increasingly useful representations.

We can think of a neural network abstractly as a sequence of functions:

and eventually

Each intermediate representation $\mathbf{h}^{(l)}$ is learned as part of the overall task. Earlier representations can capture relatively local structures, while deeper representations can combine these structures into more abstract patterns.

This is the essence of deep representation learning.

And suddenly our journey from Part 1 starts to make even more sense. We have been progressively changing representations all along. The difference is that now one of those transformations is being learned from data rather than explicitly designed by us.

Putting It All Together

Let's take one final look at the journey.

We started with physics: a vibrating physical system produces pressure variations that propagate through a medium. A microphone converts those pressure variations into an electrical signal. Sampling and quantization convert the analog signal into digital numbers.

We then explored different ways of representing those numbers. The waveform describes how amplitude changes over time. Fourier analysis describes the frequency content. The STFT gives us a time-frequency representation. A spectrogram visualizes that representation. The Mel scale incorporates a perceptual perspective, and MFCCs provide a compact representation that has historically been extremely useful for speech processing.

Now we have connected those representations to machine learning.

Sound $\rightarrow$ Waveform $\rightarrow$ Spectrogram $\rightarrow$ Features $\rightarrow$ Machine Learning $\rightarrow$ Prediction

Or, in a more modern deep learning setting:

Sound $\rightarrow$ Waveform / Spectrogram $\rightarrow$ Learned Representation $\rightarrow$ Prediction

This is where our little first-principles journey starts transitioning from signal processing into artificial intelligence. But we should not forget everything that came before. The neural network did not make the physics disappear. The waveform still came from a physical process. The spectrogram still comes from Fourier analysis. The features still encode assumptions about the signal.

Deep learning simply gives us another mechanism for discovering useful representations.

And this raises a much more interesting question than "Can a neural network classify audio?"

What exactly does the neural network learn from the sound?

Does it learn pitch? Timbre? Harmonics? Temporal patterns? Phonetic structure? Speaker identity? Or some complicated combination of all of these things that we cannot easily name?

This question takes us into the fascinating world of representation learning and interpretability.

In the next part, we will take a closer look inside the machine learning model itself. We will explore neural network representations, embeddings, convolutional filters and what it actually means for a model to "learn" a representation of sound.

The rabbit hole is getting deeper.

See you in Part 6.