Lecture5

Memory

Memory Hierarchy

Registers

CS 21 - Computer Organization and Assembly Language Programming Lecture 5 Memory: Registers and Cache

University of the Philippines - Diliman College of Engineering Department of Computer Science

Cache

Memory

Memory Hierarchy

Outline

1

Memory

2

Memory Hierarchy

3

Registers

4

Cache

Registers

Cache

Memory

Memory Hierarchy

Outline

1

Memory

2

Memory Hierarchy

3

Registers

4

Cache

Registers

Cache

Memory

Memory Hierarchy

Memory

What is memory?

Registers

Cache

Memory

Memory Hierarchy

Registers

Memory

What is memory? in human terms, it is the totality of what you remember, your recollections

Cache

Memory

Memory Hierarchy

Registers

Memory

What is memory? in human terms, it is the totality of what you remember, your recollections largely a defining factor in who you are

Cache

Memory

Memory Hierarchy

Memory

What is memory in computers?

Registers

Cache

Memory

Memory Hierarchy

Registers

Memory

What is memory in computers? structure used to store data and instructions

Cache

Memory

Memory Hierarchy

Registers

Memory

What is memory in computers? structure used to store data and instructions exhibits the widest range of type, technology, organization, performance and cost in any feature of a computer system

Cache

Memory

Memory Hierarchy

Registers

Memory

What is memory in computers? structure used to store data and instructions exhibits the widest range of type, technology, organization, performance and cost in any feature of a computer system remember: no memory = no computer!

Cache

Memory

Memory Hierarchy

Registers

Memory

What are the types of computer memory, by location?

Cache

Memory

Memory Hierarchy

Registers

Memory

What are the types of computer memory, by location?

CPU memory

Cache

Memory

Memory Hierarchy

Registers

Memory

What are the types of computer memory, by location?

CPU memory Internal memory

Cache

Memory

Memory Hierarchy

Registers

Memory

What are the types of computer memory, by location?

CPU memory Internal memory External memory

Cache

Memory

Memory Hierarchy

Outline

1

Memory

2

Memory Hierarchy

3

Registers

4

Cache

Registers

Cache

Memory

Memory Hierarchy

Memory Hierarchy

Three key characteristics of memory:

Registers

Cache

Memory

Memory Hierarchy

Memory Hierarchy

Three key characteristics of memory: cost

Registers

Cache

Memory

Memory Hierarchy

Memory Hierarchy

Three key characteristics of memory: cost capacity

Registers

Cache

Memory

Memory Hierarchy

Memory Hierarchy

Three key characteristics of memory: cost capacity access time Key relationships:

Registers

Cache

Memory

Memory Hierarchy

Memory Hierarchy

Three key characteristics of memory: cost capacity access time Key relationships: smaller access time, greater cost per bit

Registers

Cache

Memory

Memory Hierarchy

Memory Hierarchy

Three key characteristics of memory: cost capacity access time Key relationships: smaller access time, greater cost per bit greater capacity, greater access time

Registers

Cache

Memory

Memory Hierarchy

Registers

Memory Hierarchy

What do we want?

In terms of capacity? In terms of cost? In terms of access time?

Cache

Memory

Memory Hierarchy

Registers

Memory Hierarchy

What do we want?

In terms of capacity? In terms of cost? In terms of access time?

Could we get all these by using a single type of memory?

Cache

Memory

Memory Hierarchy

Registers

Memory Hierarchy

What do we want?

In terms of capacity? In terms of cost? In terms of access time?

Could we get all these by using a single type of memory? No.

Cache

Memory

Memory Hierarchy

Registers

Memory Hierarchy

What do we want?

In terms of capacity? In terms of cost? In terms of access time?

Could we get all these by using a single type of memory? No. So what do we do?

Cache

Memory

Memory Hierarchy

Registers

Memory Hierarchy

What do we want?

In terms of capacity? In terms of cost? In terms of access time?

Could we get all these by using a single type of memory? No. So what do we do?

We use a memory hierarchy

Cache

Memory Hierarchy

What is Memory Hierarchy?

A memory hierarchy uses several types of memory. The amount of memory type present in the hierarchy is inversely proportional to its cost. Thus, a memory hierarchy features very few of fast, expensive memory types and lots of slow, cheap memory types.

Memory

Memory Hierarchy

Registers

Memory Hierarchy

Important Concept The hierarchy is INVISIBLE to the user.

Cache

Memory

Memory Hierarchy

Registers

Memory Hierarchy

Important Concept The hierarchy is INVISIBLE to the user.

Except register when programming in assembly language.

Cache

Memory

Memory Hierarchy

Registers

Memory Hierarchy

Important Concept In the memory hierarchy pyramid, a higher level only contains a copy of the subset of the memory below it.

Cache

Memory

Memory Hierarchy

Memory Hierarchy

What makes Memory Hierarchy possible?

Registers

Cache

Memory

Memory Hierarchy

Registers

Memory Hierarchy

What makes Memory Hierarchy possible?

Spatial and Temporal Locality

Cache

Memory

Memory Hierarchy

Registers

What is spatial locality?

Spatial locality is the tendency of programs to access items from memory that are near each other.

Examples: linear program execution(no branches), arrays

Cache

Memory

Memory Hierarchy

Registers

What is temporal locality?

Spatial locality is the tendency of programs to access certain items from memory again and again or multiple times.

Examples: loops, counters

Cache

Memory

Memory Hierarchy

Outline

1

Memory

2

Memory Hierarchy

3

Registers

4

Cache

Registers

Cache

Memory

Memory Hierarchy

Registers

Registers

What is a register?

Memory unit closest to the ALUs Registers are contained within the CPU Fastest of all memory units(implication in terms of cost?)

Grouped into register files, which are then grouped between INTEGER and FLOATING POINT

Cache

Memory

Memory Hierarchy

Registers Why do we limit the number of registers?

Registers

Cache

Memory

Memory Hierarchy

Registers

Analogy in the human mind:

Registers

Cache

Memory

Memory Hierarchy

Registers

Analogy in the human mind: Short term memory

Registers

Cache

Registers Number of registers in historical processors

Memory

Memory Hierarchy

Outline

1

Memory

2

Memory Hierarchy

3

Registers

4

Cache

Registers

Cache

Memory

Memory Hierarchy

Registers

Cache

What is a cache? intended to provide speed and size an �intermediary� between main memory and register memory usual size: between 1K and 512K words

Cache

Memory

Memory Hierarchy

Registers

Cache

Key Concept 1 In systems with cache, processor could only access cache...

Cache

Memory

Memory Hierarchy

Registers

Cache

Key Concept 1 In systems with cache, processor could only access cache... also, processor could not access main memory directly

Cache

Memory

Memory Hierarchy

Registers

Cache

Key Concept 2 Cache memory is a TEMPORARY COPY of main memory.

Cache

Memory

Memory Hierarchy

Registers

Cache

Key Concept 2 Cache memory is a TEMPORARY COPY of main memory.

And only a very SMALL PART of main memory is copied.

Cache

Memory

Memory Hierarchy

Registers

Cache Operation Example assume that we have a processor with cache

Cache

Memory

Memory Hierarchy

Cache Operation Example the processor executes a load instruction

Registers

Cache

Memory

Memory Hierarchy

Registers

Cache Operation Example the processor tries to load the data, but address location A not in the cache

Cache

Memory

Memory Hierarchy

Registers

Cache Operation Example the cache gets copies entry from main memory

Cache

Memory

Memory Hierarchy

Registers

Cache Operation Example processor finally gets the data it needs from the cache

Cache

Memory

Memory Hierarchy

Registers

Cache Operation Example the next instruction executed by the processor is another load A

Cache

Memory

Memory Hierarchy

Registers

Cache Operation Example but since address location A is already in the cache, main memory no longer is accessed

Cache

Memory

Memory Hierarchy

Registers

Cache Performance hit - memory access that is found in the cache miss - memory access that is NOT found in the cache hit rate - percentage of memory access that is found in the cache miss rate - percentage of memory access that is NOT found in the cache

Cache

Memory

Memory Hierarchy

Registers

Cache Performance hit - memory access that is found in the cache miss - memory access that is NOT found in the cache hit rate - percentage of memory access that is found in the cache miss rate - percentage of memory access that is NOT found in the cache Problem: A certain program has 100 memory accesses(all reads), of which 82 are found in the cache.

1

What is the hit rate?

2

What is the miss rate?

Cache

Memory

Memory Hierarchy

Registers

Cache Performance hit - memory access that is found in the cache miss - memory access that is NOT found in the cache hit rate - percentage of memory access that is found in the cache miss rate - percentage of memory access that is NOT found in the cache Problem: A certain program has 100 memory accesses(all reads), of which 82 are found in the cache.

1

What is the hit rate?

2

What is the miss rate?

Important Concept: The hit rate and miss rate partly depends on the program being ran.

Cache

Memory

Memory Hierarchy

Registers

Cache Performance

hit time - time needed to access data FOUND in the cache, includes time spent in checking the cache whether it’s there or not time spent getting data from cache to processor time that is paid by all accesses

miss penalty - time needed to transfer data from main memory to cache ¨ misses additional time p ¨aidby

Cache

Memory

Memory Hierarchy

Registers

Cache Performance

hit time - time needed to access data FOUND in the cache, includes time spent in checking the cache whether it’s there or not time spent getting data from cache to processor time that is paid by all accesses

miss penalty - time needed to transfer data from main memory to cache ¨ misses additional time p ¨aidby

Problem: A certain program has 100 memory accesses(all reads), of which 82 are found in the cache. Assuming that the hit time is 2ns, and the miss penalty is 7ns, what is the total time spent for memory access?

Cache

Memory

Memory Hierarchy

Registers

Cache Performance

Important Concept: The hit time and miss penalty depend on the hardware: cache architecture and the IC fabrication process.

Cache

Memory

Memory Hierarchy

Registers

Cache Performance

Important Concept: The hit time and miss penalty depend on the hardware: cache architecture and the IC fabrication process.

Point to ponder: Miss penalty + Hitrate vs Access time if there’s no cache?

Cache

Memory

Memory Hierarchy

Cache

Elements of Cache Design Cache size Mapping Function Replacement Algorithm Write Policy

Registers

Cache

Memory

Memory Hierarchy

Elements of Cache Design: Cache Size

What is an �optimal� cache size?

Registers

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Cache Size

What is an ”optimal” cache size?

Big enough that we don’t waste time accessing main memory

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Cache Size

What is an ”optimal” cache size?

Big enough that we don’t waste time accessing main memory Small enough that it is still fast and cheap.

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Cache Size

What is an ”optimal” cache size?

Big enough that we don’t waste time accessing main memory Small enough that it is still fast and cheap.

Does a definite optimal cache size exist?

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Mapping function

What is a Mapping function?

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Mapping function

What is a Mapping function?

Policy on where data from the main memory would reside on the cache.

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Mapping function

What is a Mapping function?

Policy on where data from the main memory would reside on the cache.

Three types of mapping function:

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Mapping function

What is a Mapping function?

Policy on where data from the main memory would reside on the cache.

Three types of mapping function: Direct mapping Fully Associative mapping Set Associative mapping

Cache

Memory

Memory Hierarchy

Registers

Mapping function: Direct Mapping each memory location is mapped to exactly ONE location in the cache lower bits of an address used in mapping

Cache

Memory

Memory Hierarchy

Registers

Mapping function: Fully Associative Mapping a memory location could go to ANY cache location How to know it’s in the cache? Search every valid entry!

Cache

Memory

Memory Hierarchy

Registers

Direct Mapping vs Fully Associative Mapping

Direct Mapping very high miss rate very low hit time

Fully Associative Mapping very low miss rate very high hit time

Cache

Memory

Memory Hierarchy

Analogy for Direct Mapping

Registers

Cache

Memory

Memory Hierarchy

Analogy for Fully Associative Mapping

Registers

Cache

Memory

Memory Hierarchy

Registers

Mapping function: Set Associative Mapping

hybrid between Direct Mapping and Fully Associative Mapping location in main memory could be mapped to a SELECT number of locations level of grouping determined again by lower bits when searching, you search a few tag entries if an entry could go to n locations, it’s called an n-way associative cache

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Replacement Algorithms

Deals with the questions ”Which one do we replace?” ”Which one de we kick out?”

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Replacement Algorithms

Deals with the questions ”Which one do we replace?” ”Which one de we kick out?” Question: What Replacement Algorithm is used in a Direct Mapped Cache? Is there one?

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Replacement Algorithms

Deals with the questions ”Which one do we replace?” ”Which one de we kick out?” Question: What Replacement Algorithm is used in a Direct Mapped Cache? Is there one?

FIFO

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Replacement Algorithms

Deals with the questions ”Which one do we replace?” ”Which one de we kick out?” Question: What Replacement Algorithm is used in a Direct Mapped Cache? Is there one?

FIFO LRU

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Replacement Algorithms

Deals with the questions ”Which one do we replace?” ”Which one de we kick out?” Question: What Replacement Algorithm is used in a Direct Mapped Cache? Is there one?

FIFO LRU LFU

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Replacement Algorithms

Deals with the questions ”Which one do we replace?” ”Which one de we kick out?” Question: What Replacement Algorithm is used in a Direct Mapped Cache? Is there one?

FIFO LRU LFU Random

Cache

Memory

Memory Hierarchy

Registers

Replacement Algorithms: FIFO cache entry that is oldest is replaced Assuming a 3-entry cache with FIFO as replacement algorithm:

Cache

Memory

Memory Hierarchy

Registers

Replacement Algorithms: LRU cache entry that is least recently used/accessed is replaced Assuming a 3-entry cache with LRU as replacement algorithm:

Cache

Memory

Memory Hierarchy

Registers

Replacement Algorithms: LFU cache entry that is least frequently used/accessed is replaced Assuming a 3-entry cache with LFU as replacement algorithm:

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Write policy

Deals with the question ”What happens when we modify data that’s in the cache?”

Cache

Memory

Memory Hierarchy

Registers

Elements of Cache Design: Write policy

Deals with the question ”What happens when we modify data that’s in the cache?”

Write Through Write Back

Cache

Memory

Memory Hierarchy

Registers

Write policy: Write Through Everytime there’s a write or store, we write the data to the cache AND the main memory immediately PRO: simple to implement CON: write operations are very slow workaround: write buffers write buffer - takes care of the writing operation for the processor

Cache

Memory

Memory Hierarchy

Registers

Write policy: Write Back Everytime there’s a write or store 1 2

we write the data to the cache ONLY we set dirty bit tag of entry to 1

”update” to the main memory happens only once the entry needs to be replaced writeback for entry happens only when dirty bit tag is 1

Cache

Memory

Memory Hierarchy

Registers

Multicore processors and Caches

Which write policy do you think is better suited for multicore processors?

Cache