This post is the first in a series in which I explain ML/AI concepts at an executive level.
A key element of success in ML/AI is to develop a company-wide understanding of this technology. I certainly don’t propose that everyone write code to train a neural network. But everyone should understand the mechanics behind developing and deploying ML models, as is appropriate for their function in the business, such that they can develop realistic expectations of what this technology can and cannot achieve. This goes double for executives and other stakeholders. Hence why I’ve made explaining ML/AI a pillar of my consulting work.
A common refrain I hear is: “I couldn’t possibly understand machine learning. It’s too complicated.” The person usually follows with some hand-waving about how only their data scientists need to know this sort of thing. (We’ll come back to that later.)
My response to that is: “Give me just five minutes. I have a certain way of explaining ML models that helps it stick.”
Internally, I call it high-dimensional pattern-matching. But I start with something we all recognize: a spreadsheet.
Starting with a spreadsheet
Let’s say we’re both looking at a spreadsheet. It has one hundred numbers in a single column. If I ask you to look for patterns in those numbers, where would you start? You’d likely sort the column to get a sense of the highs and lows, and the most common range of values. Great.
Now, say I add a second column to that spreadsheet. How would you find patterns in those pairs of numbers? Sorting won’t help you there, so you’d highlight the cells and click through to create an X-Y chart. And you’d tell me straight away that you’d do the same if I were to add a third column – by plotting a 3-D chart. So far, so good.
We never gave those three columns names. Maybe if we worked in real estate, we could have called them “square footage,” “number of floors,” and “last sale price.” And by plotting those numbers in a 3-D chart, you’d be able to see how the first two columns related to a property’s price tag.
Did you see that? You were doing geometry this whole time: expressing real-world concepts as points in three-dimensional space and visually comparing them to find patterns.
Let’s add some columns …
What happens if we go from three columns – three dimensions – to ten? Or ten thousand? How would you look for patterns then? As humans, we can’t plot that many dimensions in a chart. We’re stuck.
This is where the machines come in. Machines are good at finding patterns in a high-dimensional dataset – one that is both wide (lots of columns) and long (lots of rows) – because they’re able to compare the values mathematically instead of visually.
Specifically, the computers are using linear algebra to perform those comparisons. I mention linear algebra here because our “spreadsheet full of numbers,” our “list of points” is, in fact, a matrix in disguise. And if you remember your linear algebra courses, each row of that matrix represents a vector in some N-dimensional space. Our original spreadsheet represented three-dimensional space – three columns – and computers can easily compare vectors of an arbitrarily large number of dimensions. Whether it’s two vectors or two million, it’s all the same.
So, to recap: at the start of our example, you were looking at a table full of numbers, trying to find patterns. The computer is … looking at a table full of numbers, trying to find patterns. You explore it visually while the computer uses linear algebra, but you’re both doing the same thing.
The connection to machine learning
“Fine,” you say. “You can get a computer to compare lots of vectors. So how does that connect to machine learning?”
At a high level, modern data analysis – ML included – is a twist on “compare this one thing to a bunch of other things.” Which is another way of saying “look for patterns in data.” If you can compare groups of things, you can uncover patterns. Or you will determine that there are no discernible patterns, which is useful in a different way.
(So, really, “machine learning” is “applied linear algebra and matrices” with a better marketing campaign. And if you squint just right, this idea of “compare things” is also a twist on search technology: seeing how much Item A is similar to Items B, C, and D.)
Once you can get a machine to compare lots of things at scale, you can perform magic like:
- prediction (“Given the houses I’ve seen with these features, I expect this other house should cost $N.”)
- classification (“This new document looks like these other documents about Topic XYZ, so I will say that it is also about Topic XYZ.”)
- grouping things (“This customer is similar to those customers, but very different than this other group.”)
- finding weird stuff (“This transaction is very much unlike the others; it might be fraud.”)
I’ve only listed four bullet points there. But when you think about the rich set of applications of ML/AI that you’ve seen in the wild, those four bullet points cover a lot of territory.
(I’ve also left out some details on how computers perform those comparisons. I’ll cover that in the next post.)
That’s all there is to it
This short read has taken us from rows and columns of a spreadsheet to the stacked vectors we call matrices. From low-dimensional pattern matching (plotting charts) to high-dimensional pattern matching (linear algebra). And with that, you now understand the core concepts of machine learning.
In the next post, we’ll use this knowledge to explore ML at a slightly more technical level. You’ll develop a clearer picture of what your company’s data science team works on, which will improve how the two of you communicate and also improve your chances of success putting ML/AI to work.
(Would you like your stakeholders to develop a deeper
understanding of ML/AI?
Contact me to get started.
)