Understanding Sound from First Principles: Part 6

Introduction

Here we are. The final part of our little journey into the world of sound.

If you have made it this far, congratulations. We have travelled quite a distance. What started as a simple question — what exactly is sound? — has taken us through physics, signal processing, digital systems, Fourier analysis, psychoacoustics, feature engineering and finally machine learning.

In Part 1, we started with vibration and pressure waves. In Part 2, we explored how these physical phenomena become digital data through sampling and quantization. In Part 3, we moved from the time domain to the frequency domain and introduced the spectrogram. In Part 4, we brought human perception into the picture through the Mel scale, Mel spectrograms and MFCCs. Finally, in Part 5, we connected these representations to machine learning and deep learning.

But there is one question I have deliberately postponed until the very end.

What does a machine actually learn from sound?

This is where things become philosophical, mathematical and, frankly, a little mysterious.

Let's finish the journey.

What Is a Representation?

Before talking about neural networks, we need to understand the word representation. It is one of those words that appears everywhere in machine learning and is often used without much explanation.

At its simplest, a representation is a way of describing something.

Consider the number of ways we can describe a person. We could describe their height, weight and age. We could describe their face using an image. We could describe their voice using a recording. We could describe their professional experience using a CV.

All of these are representations of the same underlying entity, but they preserve different information.

The same thing happens with sound.

A waveform represents the changing amplitude of a signal over time. A spectrum represents its frequency content. A spectrogram represents how that frequency content changes over time. A Mel spectrogram transforms that representation according to a perceptual frequency scale. MFCCs provide yet another compact representation.

The representation is not the phenomenon.

This distinction is perhaps the most important idea in the entire series.

Every Representation Makes a Choice

There is no such thing as a completely neutral representation.

When we sample an analog signal, we make choices about the sampling rate. When we quantize it, we choose the resolution. When we compute an STFT, we choose a window size and hop length. When we construct a Mel spectrogram, we choose the number of Mel bands and the frequency range. When we calculate MFCCs, we choose how many coefficients to retain.

Every one of these decisions changes what information is preserved and what information is discarded.

This is not necessarily a bad thing. In fact, it is often the whole point.

A useful representation should make the information relevant to our problem easier to work with.

This is why signal processing is so powerful. We are not merely manipulating numbers. We are choosing mathematical descriptions that expose particular properties of a physical phenomenon.

Representation Learning

Deep learning changes one important part of this process.

Instead of explicitly designing every useful representation ourselves, we can construct a model whose parameters are learned from data.

Suppose a neural network receives an input $\mathbf{x}$. The first layer transforms it into a new representation:

The next layer transforms that representation again:

And the process continues:

Eventually, the model produces a prediction.

The intermediate variables $\mathbf{h}_1,\mathbf{h}_2,\ldots,\mathbf{h}_L$ are learned representations.

The interesting part is that we do not usually tell the network what each representation should mean. Instead, the parameters of the network are optimized so that the final representation becomes useful for the task.

What Might a Neural Network Learn from Sound?

Imagine feeding thousands of speech recordings into a neural network and asking it to recognize spoken words.

The first layers might respond to relatively simple acoustic structures. Perhaps they capture local changes in amplitude or frequency. Deeper layers can combine these patterns into more complex temporal structures. Eventually, the network may learn representations that are useful for distinguishing phonetic or linguistic patterns.

For music, the story could be different. The network might discover representations related to harmonic structure, rhythm, timbre, instrumentation or other properties that help solve the particular task.

But here is the important caveat:

We should not automatically assume that a learned feature corresponds to a human-interpretable concept.

A neural network does not necessarily have a neuron labelled "this is a violin" or "this is a phoneme." Its internal representation may distribute information across many dimensions and many layers.

The representation can be useful without being easily interpretable.

Embeddings

One particularly useful concept in modern machine learning is the embedding.

An embedding is essentially a vector representation learned by a model. Instead of representing an audio recording as millions of raw samples, we can map it into a lower-dimensional vector:

The vector $\mathbf{z}$ can be thought of as a compact representation of the information that the model considers useful.

For example, two recordings that are acoustically or semantically similar may produce embeddings that are close together in the learned representation space.

This gives us a fascinating way to think about machine learning.

Instead of asking the model to explicitly describe what a sound is, we can ask whether sounds occupy meaningful locations in a learned space.

The geometry of that space can reveal relationships that are difficult to observe directly in the waveform.

The Geometry of Sound

Suppose every audio recording is represented by a vector $\mathbf{z}$. We can measure the distance between two recordings using some function $d(\mathbf{z}_i,\mathbf{z}_j)$.

If the model has learned a useful representation, recordings that share relevant characteristics may tend to be closer together than recordings that are unrelated.

This is one reason embeddings are so powerful. We can move from asking whether two recordings have identical waveforms to asking whether they are similar in a learned representation space.

Consider two people saying the same word. Their waveforms will obviously not be identical. Their voices, accents, speaking rates and recording environments may all be different. Yet a good representation may capture the underlying similarity between the utterances while being relatively insensitive to some of those irrelevant variations.

This is what we often mean when we say that a representation captures invariant structure.

The model is attempting to retain what matters for the task while becoming less sensitive to what does not.

But What Does the Model Ignore?

This is an equally important question.

If a representation preserves some information, it must inevitably discard or suppress some other information. The question is whether the discarded information is actually irrelevant.

Imagine training a speech-recognition system on recordings collected mostly from one group of speakers, using relatively clean recording conditions. The model may learn representations that work exceptionally well under those conditions.

But introduce a different accent, microphone, acoustic environment, age group or speaking style and the performance may deteriorate.

The model has not necessarily "failed to learn speech." It may have learned a representation that is too closely tied to the distribution of its training data.

This is one of the fundamental challenges of machine learning:

Learning useful structure without accidentally learning the wrong structure.

The Dataset Is Part of the Model

This brings us to another lesson that is easy to overlook.

When we say that a neural network learns from data, the data is not simply a container of examples. It determines which patterns are available for the model to learn.

If the training data contains strong correlations between the target and some unintended characteristic, the model may exploit that characteristic.

For example, imagine an animal-sound classifier where recordings of one class were mostly collected outdoors and recordings of another class were mostly collected indoors. The model could potentially learn environmental acoustic cues rather than the animal characteristics we actually care about.

The model may achieve excellent accuracy while solving the wrong problem.

This is why understanding the data-generating process is just as important as understanding the neural network.

And this is another reason I insisted on starting this series from first principles. If we understand where the signal comes from, we have a much better chance of understanding what the model might eventually learn from it.

Signal Processing and Deep Learning Are Not Enemies

There is sometimes an unnecessary tension between traditional signal processing and deep learning. One camp says that carefully engineered features are essential. Another says that neural networks should learn everything directly from raw data.

I think the more interesting perspective lies somewhere in between.

Signal processing gives us decades of knowledge about the structure of physical signals. It tells us about frequency, periodicity, filtering, sampling, aliasing, resonance, harmonics and many other properties.

Deep learning gives us powerful machinery for learning complicated representations from data.

These ideas can complement one another.

Domain Knowledge + Data + Learning = Better Scientific Questions

We can use signal processing to construct informative representations, use neural networks to learn representations, or combine both approaches depending on the problem.

The goal should not be to use the newest method simply because it is new. The goal should be to understand the problem well enough to choose an appropriate representation and learning strategy.

The Danger of End-to-End Learning

This brings us back to something I mentioned near the beginning of this series: the excitement around end-to-end learning.

There is something undeniably attractive about the idea of taking a raw waveform, throwing it into a gigantic neural network and letting the machine discover everything.

Sometimes this works remarkably well.

But "end-to-end" does not mean "assumption-free."

The sampling rate is an assumption. The input duration is an assumption. The architecture is an assumption. The optimization objective is an assumption. The training distribution is an assumption. Data augmentation is an assumption. Even the decision about what constitutes a useful label is an assumption.

The abstractions have not disappeared. We have simply moved some of them from explicit feature engineering into the architecture, data and learning procedure.

End-to-end learning does not remove the need for understanding.

Back to First Principles

And now, after all this technical discussion, let us return to where we started.

Imagine a person playing a violin.

The bow causes the strings to vibrate. The instrument's body responds to those vibrations and interacts with the surrounding air. Pressure variations propagate through the medium. The microphone converts those variations into an electrical signal. The analog signal is sampled and quantized. We obtain a sequence of numbers.

We can look at those numbers as a waveform.

We can transform them into the frequency domain.

We can construct a spectrogram and observe how frequency content evolves over time.

We can transform the representation into the Mel scale to obtain a perceptually motivated description.

We can calculate MFCCs or other features.

Or we can give a representation to a neural network and allow it to learn its own internal representation.

One physical event.
Many representations.

The Layers of Abstraction

This is perhaps the biggest lesson I take away from this journey.

At the beginning, sound looked simple. Something vibrates and we hear it.

But as we zoom in, the layers begin to appear.

Physical World
$\downarrow$
Mechanical Vibration
$\downarrow$
Pressure Wave
$\downarrow$
Analog Electrical Signal
$\downarrow$
Digital Waveform
$\downarrow$
Frequency Representation
$\downarrow$
Time-Frequency Representation
$\downarrow$
Perceptual Representation
$\downarrow$
Engineered Features
$\downarrow$
Learned Representation
$\downarrow$
Prediction

Every layer gives us a different way of asking questions about the same underlying phenomenon.

Physics asks what caused the vibration.

Signal processing asks how the signal behaves.

Psychoacoustics asks how we perceive it.

Machine learning asks what statistical structure can be learned from it.

And artificial intelligence asks whether those learned representations can be used to perform useful tasks.

So, What Is Sound?

We started this series with this question, and perhaps it is appropriate to end with it.

What is sound?

It depends on where you stand.

To a physicist, sound is a mechanical wave propagating through a medium.

To an engineer, it can be an electrical signal measured by a microphone.

To a signal-processing researcher, it can be a time series, a spectrum or a time-frequency representation.

To a neuroscientist, it can be a pattern of neural activity produced by the auditory system.

To a machine learning researcher, it can be a high-dimensional signal from which useful representations can be learned.

And to a musician, perhaps it is something much harder to put into words.

All of these perspectives can be correct at the same time.

One Final Thought

When I started this series, my intention was not to provide an exhaustive textbook on audio signal processing. There are people far more qualified than I am who have written entire books on these subjects. The goal was much simpler: to build intuition by moving slowly through the layers of abstraction.

I wanted to understand how something as apparently ordinary as sound could eventually become the input to an artificial intelligence system.

The answer turned out to be surprisingly beautiful.

We do not simply "put sound into a computer." We progressively construct representations of the phenomenon. Some are dictated by physics. Some are motivated by perception. Some are engineered by humans. Others are learned by machines.

And at every stage, something is gained and something is lost.

This is perhaps the deeper lesson beyond audio itself. Whether we are studying images, speech, biological signals, financial time series or any other complex phenomenon, we are constantly moving between representations.

The challenge is not merely to build a more powerful model.

The challenge is to understand what the model is actually being given, what it is learning, and what information is being lost along the way.

That is why I believe the first principles still matter.

Before asking a machine to learn, we should first understand the thing we are asking it to learn from.

And with that, our journey comes to an end.

From vibration to representation.
From representation to learning.
From learning to understanding.

Thank you for making the journey with me.

Until the next rabbit hole.