Okay.
So, we're, we're through three of our
seven optimizations for today.
Let's look at the next optimization.
And this, this comes back to this
fundamental problem that you can build
something both big and fast.
They fed off against each other. You can
build really big cache, you could build
big memory.
You can have, you know, have boxes of
memory that fill this entire room.
You go in some data center, people
actually build big boxes of RAM that would
fill this entire room, but it's slow, it
takes a long time to access it.
It's a bunch of multiplexers, they need to
sort of choose between all the different
racks of RAM, we'll say, or if you have a
32 megabyte cache, that's a lot of
multiplexing and stuff like that.
So, you can have something both big and
fast.
Now,
Our solution to this, our multi-level
caches is that you put progressively
larger caches as you go farther out.
So, you have a small cache which you
access very frequently, and has a very
fast cycle time.
You put a L2 which is a little farther
away but can hold more data.
And then, you have, let's say, main
memory, or maybe an L3 out here which
holds more data but takes longer to
access.
So, that's really the insight here is
that, you can have larger cache sizes as
you get farther away and it helps to have
larger caches cuz you will be able to hold
more capacity.
And you'll, sort of, gracefully decay
versus falling off a cliff,
Either you're in the cache or you're not
in the cache.
So that means multilevels of caches can,
can fix this problem.
The other thing you can do is, because
this cache is farther away, it takes more
cycles to access it.
You can afford for it to be bigger and
take multiple cycles to access it.
So, kind of it by virtue of it being
farther away, if it takes two cycles to
access versus one cycle to access.
But it already took you, lets say, a few
cycles to get out to that cache,
You know, you've got, you have the extra
time to go do that.
So, this is our multi, multilevel caches.
So you'll see, lots of processors people
will quote, the L1 cache size and the L2
cache size, and the L3 cache size.
One, one thing I wanted to say about this
was that, there's actually some, some
rules of thumb that apply to this.
And they're kind of, kind of random and
empirically found. But typically, in a
processor design, you're going to want to
make your next level of cache about eight
times bigger than the previous level of
cache to have it have some useful, effect.
And this is just an empirical sort of rule
of thumb that people typically apply, and
if you go look at almost all the
processors in the world, this almost
always holds true.
And the reasoning behind this is, you
know, it takes extra time to go access
this next level of cache. And, if you're
just going to go up by a factor of two,
it's not really worth it.
And if you, sort of, run the numbers of
back in the beginning, of the missed time
multiplied by our missed rates.
If a miss time goes up a bunch, the miss
rate needs to go down a lot for this to
actually make any, any sense to use.
And if you only go up by a factor of two
in cache size,
Let's say, you have an eight kilobyte
cache backed by a sixteen kilobyte cache,
It just sort of works out but it doesn't
actually help.
Somebody said that it is rolled into that
rule of thumb is how much time it takes to
sort of, go to next level of cache.
Because if it only takes, let's, say, one
more tick or one more cycle to go to the
next level of cache, the rule of thumb
might break down.
It might actually make sense to have a
two, a multiplicative cache, or a cache
that's only two sizes larger than the next
INAUDIBLE. But in reality, people
typically don't really build these things
unless they're eight levels up. And you
can, sort of, sit down and plot looking in
the Hennessy and Patterson book, The
Missed Rates.
And you'll, you'll see that this rule of
thumb kind of works out pretty well.
Okay.
So, now that we think we want to build
multilevel caches, how does this affect
the actual cache design?
Believe it or not, having an L2 cache
effects the L1..
Hm, that's an interesting insight.
So, a couple of things can happen.
First of all, it might actually influence
your design to even have a smaller
low-level cache, or, or smaller level one
cache.
So, you can actually think about having a
bigger cache so will give you the
aggregate readout the L1, the L2 and the
performance out of it might be a good
trade and you can actually reduce your
cycle time of your processor and that
could be a good trade-off.
And this is something like they what they
did in the payment before where the cycle
time was very important to them.
They put a much smaller L1 cache and they
had a backing L2 behind that.
Cough And, another advantage of this is
you can actually reduce you energy used to
access the cache if you have L1 and L2 and
you make your L1 smaller, cuz a lot of the
work you're doing is in L1.
So, if you make your L1 smaller, you're
going to be firing up effectively less
transistors.
Another way that having a L2 and influence
the L1 is you can potentially have the L1
not be write-back.
So, you can actually have it be write
through.
And this is a common thing.
For instance, actually, the, the Tilera
Processor, which I worked on, we do this.
We have a right through from the L1 to the
L2.
It makes the L1 a lot simpler.
It makes it so you don't have to have
somewhere on this slide here,
You don't have to have dirty data in your
L1.
And this is a big benefit.
So, not having dirty, dirty data in L1
means that you could potentially not have
to have an error checking, or error
correcting code on your L1 data because
none of the data in there's important.
It's all going out to the L2 anyway.
So, if it's all in the L2 anyway, you
could potentially just invalidate your
entire L1 and the program is still
correct.
So, to really simplify error recovery you
can sort of replace error checking with
just pari, or, EEC with just parity. So,
that's a big win of having, write through
from the L1 to the L2.
And, this is actually the reason that
people typically do this.
And we haven't talked about this yet,
we'll actually talk about it at the end of
the term cuz it actually will make
multiple processors easier to build.
Now, what do I, what do I mean by that?
Well, if you have a cache coherent
processor.
So you have multiple processors, and they
all have caches, but they're trying to use
one piece of memory, or use one big shared
memory, you have to go check everybody
else's cache if you're looking for data.
But if you have a write through from the
L1 to the L2,
Well, this is a benefit here because what
this means is you have no dirty data in
your L1, so you never have to, to check
the L1.
You only have ever have to go check the
L2.
And this is, this is a, this is a big win
for coherence perspective because it just
makes it a lot simpler.
This is, this sort of goes back to our
write buffer that we were talking about.
If you have the L1 write through to the L2
and you go to do, let's say, something
that will invalidate or create a victim in
your L1, you don't ever have to take that
victim and write it out to the L2. If all
your data's clean in your L1 at all times.
Anyway, what I'm trying to get out at here
is,
You would think that just putting like a
multilevel cache doesn't effect the,
closer to the core caches, but it does cuz
you typically design your whole cache
system together.
Another thing that, you know, affects is
the inclusion policy.
So, if we go look at sort of a Venn
diagram of memory,
And we have data which is stored in our L2
cache, and we have a smaller L1,
The data is stored in both, in a inclusive
cache.
In an exclusive cache,
We data in our L2,
And data is in our L1, and they are not
the same.
This is inclusive.
This is exclusive.,
Okay.
So,
If you go look at inclusive, you're going
to say, I'm wasting space because I have
to store, if any piece of data is in L1
has to be stored in L1 and L2, and over
here we only need to store the data in one
location.
So, you would think everyone would have
always wanted to build inclusive caches.
In reality, most people build inclusive
caches and don't build exclusive caches.
And, and, the, the reasoning behind this
is that it gets hard to build multilevel
exclusive cache.
So, let's, let's look at this a little bit
more detail here.
So,
The exclusive cache stores more data, so
that's a positive.
The negative here is that you need to
check both locations when you have to do,
let's say, a remote invalidate.
So give our cache rehearse protocol, our
multiple processor cache rehearse
protocol, and we need to go check where
data is.
Now, instead of checking let's say, the L2
and knowing the data from a time
perspective that's in the L2 and is also
in the L1.
Now, we're going to check the tags
INAUDIBLE.
Exclusive caches, as far as I know, we
could actually implement that these are
sort of AUD. and some of the complexity
involved in this, you really have to swap
the lines between the inner type of caches
where you take caches because data, which
can't be used in the, the other cache.
So, when you take cache from this from the
one, let's say, from the L1 then pick on
these to go to the L2 and when the data is
coming in, that's a transfer from the L2
to the L1.
This gets hard, hard to be correct under
all circumstances. You'll see once you get
the hang of it there.
And one little, one last little thing of
why exclusive caches are hard to do is
that if cache y size or the cache y size
is different,
You basically can't build a exclusive
cache because you're swapping sort of,
lines between these two things, and it
just doesn't make any sense, doesn't
anybody benefit.
And, lots of times people like to have
smaller block size in the L1 cache because
you get more locations, more blocks to
work than L2. And if, let's say, have
those block sizes different, you can't
necessarily represent let's say,
In the L2 cache, which has, let's say, a
64-byte block size, and the L1 which has a
32-byte block size,
Just go. You won't have all the data cuz
it takes two of the smaller cache lines to
put together one of the bigger cache line,
so you can't really sort of, swap them cuz
it's not compatible.
So, in each, exclusive cache, this forces
you to have the same cache size between
two.
But the, of course, is that you get more
data.
This, this is also the other reason why
sort of, this for inclusive caches, if you
guys are qithout the rule eight
multiplication or making this cache a
transfigure multiple cache prospective,
that, that, that step up then kind of
INAUDIBLE turn into a noise because, oh,
yes, you were duplicating something which
all you INAUDIBLE L2 cache size INAUDIBLE.
Okay.
So, a few set of examples are ubiquitous,
multiple caches, some real, real
INAUDIBLE.
Cough Here, we have,
You have the Itanium-2 this is a DYW
Architectural advance dealing on your
picture that we talked about from Intel
and HP.
And you sort of look at, they have a
sixteen kilobytes, L1, and then L2 that
should change both these things. So, it
change the size and, and the base it's,
it's bigger than any x multiplier.
And they also changed the line size. And
then,
They even have a third level here,
That is when you have two, might as well
go to three.
So they have a third level cache because
they have a space.
This is a expensive processor, they had a
lot of space, so they put a level three
cache here.
And, in the third level cache, they
actually increased the associativity quite
a bit, a 12-way set associative.
And, they made the cache a lot bigger,
three megabytes.
And the line size is the same as they're,
they're almost the right size.
Let's look at the, the latency here. They
have single cycle latency in the L1,.
That's usually kind of what you want, very
fast latency in the L1, five cycles L2,
twelve cycles L3.
It seems like it steps up as it gets
further away.
If we look at the Power seven this was the
ten ultimate power architecture from IBM,
the Power eight that is, you know, just
came out.
Cough Here, you can see they, they have
lots of cache in the middle,
And this is a muti-core.
So, eight cores here. Each of those cores
has a 5-bit L2.
They have a little bit bigger L1s that
have 30 kilobyte I and 30 kilobyte D L1s..
A little bit by INAUDIBLE L1, three
cycles, INAUDIBLE L2.
Then, the Intel processor we've seen
before, and INAUDIBLE L3.
But the L3 is huge, 32 megabytes.
It's a lot and a lot of data out here.
But it's also being shared between the
processors.
So, this will be about private L2s,
private L1s, and a shared L3.
So, interesting design.
Its multilevel caches show often in
basically all of modern processors,
Npw we have multilevel caches.
Okay.
So, let's take a look at the efficacy of
the multilevel cache.
Cough This doesn't you feel like it's sort
of just the L1 by itself.
This is, I don't know, look at the, the
miss penalty of that is going to go down.
So, just for the, just for the L1, the
miss penalty of this goes down because if
you're missing your L1, before you had to
go out to main memory, and that was
expensive, but now you can go to your L2
or L3. So, you're reducing this out.
If you draw up a box around the L1, and L2
and L3, everything together, this is both
going to reduce your missed penalty and
you're going to have more cache space to
reduce your risk rate.
So you're going to have this, this is
reduced and broken this penalty, and in
the risk rate together, dropping box for
all your cache in the structure in a the
multilevel cache system.