An algorithm is a procedure for solving a problem that is precise enough to follow without judgement: a recipe, a knitting pattern, long division. Everything a computer does is one, and so is a great deal of what a scientist does, whether or not it is written down. This site teaches programming through neuroscience, and almost every lesson quietly introduces an algorithm along the way. This page, rewritten in October 2026 from a short note posted in 2016, says what the word means, shows with real neural data why the choice of algorithm can be the difference between a second and an hour, and collects the algorithms the lessons use into one table.
What an algorithm is, and what it is not
Three properties make a procedure an algorithm. It is finite: it ends. It is definite: each step is unambiguous, so two people following it get the same result. And it is effective: each step is something that can actually be carried out. A recipe that says “season to taste” is not quite an algorithm; one that says “add 5 g of salt” is. The distinction matters in science for the reason given in the opening essay: an analysis that lives as an algorithm can be repeated, checked, and handed to someone else. One that lives as a sequence of clicks and judgement calls cannot.
An algorithm is not the same as a program. The program is one expression of the algorithm, in one language, with particular choices of data structure; the same algorithm can be written in Python, Java or on paper. And an algorithm is not the same as a model. Logistic regression is a model, a claim about how outputs depend on inputs; gradient descent is an algorithm for finding its weights, and there are others. Keeping the two apart is useful when something goes wrong, because the model can be right while the algorithm fails to find its best version, and the other way round.
Why it matters: the same answer, ten thousand times faster
Here is the question from lesson 1: how many times did a neuron fire in each second of a 45-minute recording? The data are the spike times of 734 neurons from a mouse, sorted in time. Two procedures give the same count. The first scans every spike and asks whether it falls in the window. The second uses the fact that the times are sorted: binary search finds where the window starts and ends by repeatedly halving the range, and the count is the difference between the two positions.
import numpy as np
import time
data = np.load("steinmetz_session.npz", allow_pickle=True)
spike_times = data["spike_times"] # 734 neurons, each a sorted array of spike times in seconds
duration = max(st.max() for st in spike_times)
print(len(spike_times), "neurons,", sum(len(st) for st in spike_times), "spikes in", round(duration / 60), "minutes")
def count_by_scanning(st, t1, t2):
"""Count spikes between t1 and t2 by looking at every spike."""
n = 0
for s in st:
if t1 <= s < t2:
n += 1
return n
def count_by_searching(st, t1, t2):
"""Same answer, using the fact that the times are sorted: find both ends by binary search."""
return np.searchsorted(st, t2) - np.searchsorted(st, t1)
neuron = spike_times[303]
print(count_by_scanning(neuron, 100, 101), count_by_searching(neuron, 100, 101), "spikes in the 101st second, both ways")
# 734 neurons, 7574404 spikes in 45 minutes
# 31 31 spikes in the 101st second, both ways
Both functions return 31 for the hundred-and-first second. Now do it for every second of the recording, 2,700 windows, and add a third method: np.histogram, which sorts the problem out in a single pass over the spikes and all the window edges at once.
windows = np.arange(0, duration - 1, 1.0) # every second of the recording: 2,700 windows
t = time.perf_counter()
a = [count_by_scanning(neuron, w, w + 1) for w in windows]
scan = time.perf_counter() - t
t = time.perf_counter()
b = [count_by_searching(neuron, w, w + 1) for w in windows]
search = time.perf_counter() - t
t = time.perf_counter()
c, _ = np.histogram(neuron, bins=np.append(windows, duration)) # one call that does all the windows at once
hist = time.perf_counter() - t
print(f"scanning: {scan:6.2f} s searching: {search:6.3f} s histogram: {hist:6.4f} s same answers: {a == b == list(c)}")
# scanning: 6.34 s searching: 0.040 s histogram: 0.0005 s same answers: True
Six seconds, forty milliseconds, half a millisecond. Same answers, checked. Scanning costs the number of spikes times the number of windows, about 44 million comparisons here. Binary search costs the logarithm of the number of spikes per window, about 14, so the whole job is forty thousand comparisons. The histogram does one pass over the spikes and the edges together, which is as good as it gets. For one neuron the difference is six seconds against nothing; for the 734 neurons in this file, the scanning version takes well over an hour, and that is the difference between an analysis you run and one you do not.
The word for this is complexity: how the cost of an algorithm grows with the size of its input, written with the “big O” notation you will see in any algorithms book. Scanning is O(spikes x windows); searching is O(windows x log spikes). Nobody needs the notation to see it in the figure, where the three lines are parallel on logarithmic axes but far apart. What the notation adds is the ability to predict, before writing anything, which approach will survive contact with a dataset ten times larger.
A famous one: the fast Fourier transform
Lesson 6 computed a spectrum two ways: a loop that matched the signal against a sine and cosine at every frequency, written so you could see what a Fourier transform does, and np.fft.rfft, which gives the same answer. Here is why everyone uses the second.
def dft_by_matching(x, fs):
"""Lesson 6's spectrum: match the signal against a sine and cosine at every frequency. N frequencies x N samples."""
n = len(x)
tt = np.arange(n) / fs
freqs = np.fft.rfftfreq(n, 1 / fs)
return np.array([np.hypot(np.sum(x * np.cos(2 * np.pi * f * tt)), np.sum(x * np.sin(2 * np.pi * f * tt))) for f in freqs])
rng = np.random.default_rng(0)
for n in [256, 1024, 4096, 16384]:
x = rng.standard_normal(n)
t = time.perf_counter(); slow = dft_by_matching(x, 160.0); t_slow = time.perf_counter() - t
t = time.perf_counter(); fast = np.abs(np.fft.rfft(x)); t_fast = time.perf_counter() - t
print(f"n = {n:5d}: matching {t_slow * 1000:8.1f} ms FFT {t_fast * 1000:6.3f} ms same answer: {np.allclose(slow, fast)}")
# n = 256: matching 2.4 ms FFT 0.035 ms same answer: True
# n = 1024: matching 15.5 ms FFT 0.021 ms same answer: True
# n = 4096: matching 158.0 ms FFT 0.053 ms same answer: True
# n = 16384: matching 2213.1 ms FFT 0.334 ms same answer: True
The loop is O(n²): n frequencies, each matched against n samples. The FFT is O(n log n), and the gap grows without limit: a factor of 70 at 256 samples, 6,600 at 16,384, and for a minute of 384-channel Neuropixels data, more than you would wait for. The algorithm was published by Cooley and Tukey in 1965 (and, it turned out, by Gauss in 1805), and it works by noticing that the transform of a signal can be assembled from the transforms of its even-numbered and odd-numbered samples, each of which can be assembled the same way, and so on down. That pattern, divide and conquer, is one of a handful of ideas behind most fast algorithms, and the FFT is the reason spectrograms, filters, correlations and much of signal processing are cheap enough to be routine.
Data structures are half of it
The first example turned on a fact about the data, not the code: the spike times were sorted. Binary search is useless on an unsorted list. The shape in which data are kept is called a data structure, and choosing it is usually the first algorithmic decision in an analysis. The lessons use two for spikes. Lesson 1 keeps a list of times per neuron, which is compact and exact: 7.6 million spikes in 60 MB. Lesson 2 keeps a dense array of counts, neurons by trials by time bins, which is 59 MB for a single session even though most entries are zero, and in exchange every question about a window is a slice, not a search. Neither is right in general. Sparse for storage and for anything that walks along time; dense for anything that compares across trials or neurons. Most of the speed of NumPy is the dense array, and most of the memory problems people hit with it come from using a dense array for sparse data.
The algorithms on this site
Each of the lessons is built around one, and most of them were written out in full rather than imported, so that nothing interesting is hidden. Here they are in one place, with the kind of algorithm each is and how its cost grows, because the kind tells you what to expect: a numerical integration gets slower with finer time steps, an iterative optimisation may stop at the wrong answer, a library eigendecomposition is not worth rewriting.
| Where on this site | Algorithm | What kind of thing it is | How its cost grows |
|---|---|---|---|
| Lesson 1 | Leaky integrate-and-fire simulation | Numerical integration: step time forward, update, repeat | Linear in the number of time steps |
| Lesson 3 | Gradient descent for logistic regression | Iterative optimisation: nudge the weights downhill until they stop moving | Steps x trials x features |
| Lesson 4 | Eigendecomposition for PCA | Linear algebra: a library routine (LAPACK) that is decades old and faster than anything you would write | Cubic in the number of features |
| Lesson 5 | Threshold detection; k-means | One pass over the signal; then an iterative assign-and-move loop that is not guaranteed to find the best answer | Linear; then clusters x points per iteration |
| Lesson 6 | Fast Fourier transform | Divide and conquer: split the signal in two, transform each half, combine. The reason the spectrum is cheap | n log n instead of n$^2$ |
| ICA primer | FastICA | A fixed-point iteration: apply a rule until the answer stops changing | Iterations x components$^2$ x samples |
| Lesson 5, in passing | Kilosort | Template matching with drift correction: the state of the art, and tens of thousands of lines | Hours per recording on a GPU |
Three patterns cover most of the table. One pass over the data, which is the cheapest possible and the thing to aim for. Iterate until it stops changing, which is gradient descent, k-means and FastICA, and which needs a stopping rule and a check that it stopped somewhere sensible. And divide and conquer, which is the FFT and most sorting. When you meet a new method, asking which of these it is will tell you most of what you need to know about its behaviour.
Reading further
- Algorithms by Robert Sedgewick and Kevin Wayne (4th edition): the clearest introduction, with complete implementations and a free companion site. The definition at the top of this page is theirs.
- Introduction to Algorithms by Cormen, Leiserson, Rivest and Stein: the reference, heavier and more mathematical. Good for looking things up once you know what you are looking for.
- Grokking Algorithms by Aditya Bhargava: illustrated and short, for a first pass with no computer-science background.
- For the scientific side, Numerical Recipes by Press and colleagues, whatever edition you can find, explains the numerical algorithms (integration, FFT, optimisation) that the table above leans on, with the opinions left in.
And the data to practise on are on the open data sources page. Carl Sagan said that science is a way of thinking much more than it is a body of knowledge. Algorithms are that way of thinking made precise enough for a machine, and learning to see them is most of learning to program.
Data: Steinmetz et al., Nature 2019, CC-BY 4.0, via Neuromatch Academy (the lesson 1 file). Timings are from a laptop and will differ on yours; the ratios will not.