Memory
Memory is anywhere in the computer that we can store some bits.
In our discussions of CPUs we’ve talked about RAM as the place outside the CPU where data is stored and a bit about registers and cache within the CPU.
Now we’re going to look in more detail about the differences between these different kinds of memory and add durable storage (hard drives) into the mix.
Memory hierarchy
| Level | Typical size | CPU cycles | Access time | $/GB* |
|---|---|---|---|---|
| Registers | 128 B to a few KB per core | 0–1 | < 0.3 ns | $100k–$1M+ |
| L1 cache | 32–64 KB data + 32–64 KB instruction, per core | 4–5 | 1 ns | $10k–$100k |
| L2 cache | 256 KB – 3 MB per core | 12–16 | 3–5 ns | $300–$3k |
| L3 cache | 8–64 MB consumer; up to 1.15 GB server | 30–70 | 10–20 ns | $100–$1k |
| RAM | 8–128 GB consumer; 1–64 TB server | 200–400 | 60–100 ns | $3–$25 |
| SSD | 256 GB – 8 TB consumer; up to 250 TB server | 80k-400k | 20–100 µs | $0.08–$0.16 |
| HDD | 1–24 TB consumer; up to 36 TB server | 20M–40M | 5–10 ms | $0.01–$0.05 |
* Registers and caches aren't sold separately; these are rough estimates of based on the cost of the chip area they take up. RAM, SSD and HDD figures are 2026 retail prices, which have been unusually high and volatile, especially for RAM and SSDs.
Put it on a human scale
If we put things on a human scale, where fetching from a register took one second, it would take …
- Three seconds to fetch from L1 cache
- Seventeen seconds to fetch from L2 cache
- One minute to fetch from L3 cache
- Five minutes to fetch from RAM
- Three days to fetch from an SSD
- A year to fetch from an HDD
Latency vs throughput
Remember latency vs throughput from our discussion of pipeline parallelism?
Same principle applies here for RAM, SSD, and HDDs.
The latency times are how long it takes to access a random piece of data. But if we are going to fetch a bunch of data stored sequentially there are number of reasons that fetching the next bit of data will be faster.
So data throughput can be much higher than the latency numbers suggest.
Sequential access to RAM is faster than random access because when data is fetched from RAM into cache, it fetches a whole cache line of, usually, 64 bytes containing the actual byte requested.
Thus reading the next bytes in memory will find them already in the cache.
This is why iterating through an array is usually much faster than iterating through a linked list.
Data is fetched from SSDs in bigger chunks than from RAM but it can also fetch multiple chunks at more or less the same time.
When a program reads more data than fits in one chunk, such as when it’s reading a lot of data sequentially, that can get automatically turned into multiple reads happening at the same time which means the throughput is increased.
HDDs have an even more significant difference between latency and throughput.
HDDs have to physically move a read head to the right part of the disk and then wait for a spinning platter to come around to the right place to read a piece of data.
But once the read head is positioned properly, the disk can then read data at the speed the platter spins past the head.
The odd duck: ROM
ROM stands for Read Only Memory. Which is more or less true.
ROM is also non-volatile which means, like SSDs and HDDs, it can store data without power.
ROM is useful for holding code that doesn’t need to (or shouldn’t) change and which is needed when a computer can’t access other secondary memory.
Game consoles
Old game consoles use to use ROM cartridges that were basically a bunch of memory chips that you plugged into the console where they acted like RAM containing the code of the game.
These days game cards in systems like the Nintendo Switch are ROM but they act more like hard drives, holding data that is loaded into RAM and then run.
Every computer has at least a small bit of mask ROM which is a chip that is completely hardwired to hold certain data. It cannot be changed at all, and is used to hold the very first code that runs when the computer powers on.
That used to be called the BIOS or Basic Input/Output System.
These days the mask ROM only holds enough code to verify and load another bit of code stored in rewritable flash ROM.
The main takeaway
You should definitely be able to lay these out in this order.