Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
CS 21 - Computer Organization and Assembly Language Programming Lecture 17 Moore’s Law and Parallelism University of the Philippines - Diliman College of Engineering Department of Computer Science
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Outline
1
Moore’s Law
2
Two Fronts
3
The Rationale for Multicore
4
Parallel Computing
5
The Problem with Multicore
6
Importance
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Outline
1
Moore’s Law
2
Two Fronts
3
The Rationale for Multicore
4
Parallel Computing
5
The Problem with Multicore
6
Importance
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
Moore’s Law
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
The amount of transistors that could be placed on a chip doubles every 2 years.
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
The amount of transistors that could be placed on a chip doubles every 2 years. What powers Moore’s Law?
Moore’s Law
Two Fronts
The Rationale for Multicore
Transistor Feature Size
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Outline
1
Moore’s Law
2
Two Fronts
3
The Rationale for Multicore
4
Parallel Computing
5
The Problem with Multicore
6
Importance
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Interesting fact #1 about transistors
As transistors get smaller, their switching times get shorter.
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Interesting fact #1 about transistors
As transistors get smaller, their switching times get shorter. What are implications of this fact for gate delays and operating frequencies?
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Two trends due to Moore’s Law
Moore’s Law opens up two fronts of attack in our bid to improve generation after generation of processors higher frequency more or bigger components
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Higher frequency processor operating frequencies have been constantly increasing since the 1970s, taking full advantage of the highest frequencies transistor technologies could give
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Mathematically speaking...
Which terms are affected?
CPUExecutionTime =
CPUclockcycles Clockrate
CPUClockCycles = numberofinstructions ∗ CPI
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Processor Clockrate Trend
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
More or bigger components more complicated components enable us to extract Instruction Level Parallelism from programs, enabling us to work on several instructions in parallel or to take less numbers of cycles when doing certain operations Examples: register files with multiple read ports advanced adder designs dedicated multiplier
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Mathematically speaking...
Which terms are affected?
CPUExecutionTime =
CPUclockcycles Clockrate
CPUClockCycles = numberofinstructions ∗ CPI
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Taken together
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Outline
1
Moore’s Law
2
Two Fronts
3
The Rationale for Multicore
4
Parallel Computing
5
The Problem with Multicore
6
Importance
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Interesting fact #2 about transistors
As the switching rate of transistors increase, their power consumption increases.
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
The Heat/Power Limit
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
The Heat/Power Limit
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
The CPU Cooling Challenge: Heat Sink and Fan
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
The CPU Cooling Challenge: Bigger Heat Sink and Fan
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
The CPU Cooling Challenge: Water Cooling
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Two trends due to Moore’s Law Moore’s Law opens up two fronts of attack in our bid to improve generation after generation of processors higher frequency more or bigger components
CPUExecutionTime =
CPUclockcycles Clockrate
CPUClockCycles = numberofinstructions ∗ CPI We lost one of the two!
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
I do hope you’re asking...
Couldn’t we just keep on building bigger and bigger processors?
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
I do hope you’re asking...
Couldn’t we just keep on building bigger and bigger processors?
Yes, we can but...
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Interesting fact about gates and wires
As transistors get smaller, switches(hence, gates) get faster.
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Interesting fact about gates and wires
As transistors get smaller, switches(hence, gates) get faster. But the wires that connect those switches and gates, do not get faster.
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Interesting fact about gates and wires
As transistors get smaller, switches(hence, gates) get faster. But the wires that connect those switches and gates, do not get faster. In fact, at higher frequencies we spend more time waiting for the signals to arrive than actually working on the signals.
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
In other words...
More time waiting translates to more cycles, which means higher CPI.
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
In other words...
More time waiting translates to more cycles, which means higher CPI. Meaning, we have to minimize the distance that a signal travels within the processor.
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
In other words...
More time waiting translates to more cycles, which means higher CPI. Meaning, we have to minimize the distance that a signal travels within the processor. So we could not keep on simply keep on building bigger and more complex processors.
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
A review...
We could no longer ramp up the clock rate because of power consumption issues
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
A review...
We could no longer ramp up the clock rate because of power consumption issues Transistors would still get smaller, giving us more transistors to work with as time goes by
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
A review...
We could no longer ramp up the clock rate because of power consumption issues Transistors would still get smaller, giving us more transistors to work with as time goes by But we could no longer build more and more complex processors because of the disparity between gate delays and wire delays
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
A review...
We could no longer ramp up the clock rate because of power consumption issues Transistors would still get smaller, giving us more transistors to work with as time goes by But we could no longer build more and more complex processors because of the disparity between gate delays and wire delays Given these conditions, how would we improve our processors with each new transistor tecnhnology generation?
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The solution
We keep the clockrate constant...
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The solution
We keep the clockrate constant... We keep the processor design simple...
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
The solution
We keep the clockrate constant... We keep the processor design simple... We use our numerous transistors in putting multiple copies of our simple processor into our die
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
The solution
We keep the clockrate constant... We keep the processor design simple... We use our numerous transistors in putting multiple copies of our simple processor into our die This design is known as the multicore processor.
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The new trend in processor design
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Change in plans
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
The Dual Core Opteron
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Multicore processors and parallel computing
Using multicore processors belongs to the paradigm of parallel computing.
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Multicore processors and parallel computing
Using multicore processors belongs to the paradigm of parallel computing. Is parallel computing new?
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Multicore processors and parallel computing
Using multicore processors belongs to the paradigm of parallel computing. Is parallel computing new? Not really.
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Outline
1
Moore’s Law
2
Two Fronts
3
The Rationale for Multicore
4
Parallel Computing
5
The Problem with Multicore
6
Importance
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Cluster Computers Beowulf Cluster 1 commercially available computers connected via a network doing parallel computations 2 utilizes F.O.S.S. such as Linux 3 nodes pass messages containing data to one another using MPI of PVM
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Beowulf Cluster
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Beowulf Cluster
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
Beowulf Clusters in UP Diliman DOST-ASTI
UP CSRC
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Grid Computers 1
nodes are distributed over a large geographic area, over multiple administrative domains
2
nodes are usually not dedicated computers
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
Grid Computing Example: SETI@home
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Grid Computing Example: Folding@home
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Outline
1
Moore’s Law
2
Two Fronts
3
The Rationale for Multicore
4
Parallel Computing
5
The Problem with Multicore
6
Importance
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Differences
How is multicore different from previous parallel computing approaches? most of the traditional parallel computing approaches have been applied to scientific computations many scientific computations are naturally parallel in nature they have few dependencies and they are come in very regular data structures and runtime execution patterns
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Differences
How is multicore different from previous general purpose computing approaches? previous processors were programmed pretty much the same way programmers just assumed that with each new processor the performance of their programs would automatically improve
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
The Multicore Problem
What are the problems with the Multicore Approach?
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
The Multicore Problem
What are the problems with the Multicore Approach? legacy programs would not simply improve automatically with multicore processors
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
The Multicore Problem
What are the problems with the Multicore Approach? legacy programs would not simply improve automatically with multicore processors programs have to be explicitly designed to take advantage of multicore processors
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
The Multicore Problem
What are the problems with the Multicore Approach? legacy programs would not simply improve automatically with multicore processors programs have to be explicitly designed to take advantage of multicore processors
some programs and applications are simply serial in nature
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
The Multicore Problem
What are the problems with the Multicore Approach? legacy programs would not simply improve automatically with multicore processors programs have to be explicitly designed to take advantage of multicore processors
some programs and applications are simply serial in nature most of the programs that could be parallelized or have parallel segments are not yet coded for parallel execution
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
The Multicore Problem
What are the problems with the Multicore Approach? legacy programs would not simply improve automatically with multicore processors programs have to be explicitly designed to take advantage of multicore processors
some programs and applications are simply serial in nature most of the programs that could be parallelized or have parallel segments are not yet coded for parallel execution it is difficult to program and debug in parallel!
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Multicore Processors and Amdahl’s Law
You have a program which is 50% parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a dual core processor? Speedup =
1 (1 − p) +
p s
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Multicore Processors and Amdahl’s Law
You have a program which is 50% parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a dual core processor? Speedup =
Speedup =
1 (1 − p) +
1 (1 − 0.5) +
p s
0.5 2
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Multicore Processors and Amdahl’s Law
You have a program which is 50% parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a dual core processor? Speedup =
Speedup = Speedup = 1.333
1 (1 − p) +
1 (1 − 0.5) +
p s
0.5 2
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Multicore Processors and Amdahl’s Law
You have a program which is 50% parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a quad core processor? Speedup =
1 (1 − p) +
p s
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Multicore Processors and Amdahl’s Law
You have a program which is 50% parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a quad core processor? Speedup =
Speedup =
1 (1 − p) +
1 (1 − 0.5) +
p s
0.5 4
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Multicore Processors and Amdahl’s Law
You have a program which is 50% parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a quad core processor? Speedup =
Speedup = Speedup = 1.6
1 (1 − p) +
1 (1 − 0.5) +
p s
0.5 4
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Limits in parallelizability
in reality, even parallelizable sections in programs are not infinitely parallelizable degree of parallelizability is dictated by nature and design of the program, algorithm, and the size of the dataset
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Multicore Processors and Amdahl’s Law
You have a program which is 50% parallelizable. The parallelizable section however, is three-way parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a quad core processor? Speedup =
1 (1 − p) +
p s
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Multicore Processors and Amdahl’s Law
You have a program which is 50% parallelizable. The parallelizable section however, is three-way parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a quad core processor? Speedup =
Speedup =
1 (1 − p) +
1 (1 − 0.5) +
p s
0.5 3
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Multicore Processors and Amdahl’s Law
You have a program which is 50% parallelizable. The parallelizable section however, is three-way parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a quad core processor? Speedup =
Speedup = Speedup = 1.5
1 (1 − p) +
1 (1 − 0.5) +
p s
0.5 3
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Multicore Processors and Amdahl’s Law in a nutshell
The speedup caused by adding more cores to a multicore processor depends on the fraction of the workload that is parallelizable, and the degree of parallelizability of thos sections.
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
Multicore Processors and Amdahl’s Law in a nutshell
The speedup caused by adding more cores to a multicore processor depends on the fraction of the workload that is parallelizable, and the degree of parallelizability of thos sections. Adding more cores to a processor that runs mostly on codes that have limited parallelizable sections is useless and would not bring any improvement.
Moore’s Law
Two Fronts
The Rationale for Multicore
Outline
1
Moore’s Law
2
Two Fronts
3
The Rationale for Multicore
4
Parallel Computing
5
The Problem with Multicore
6
Importance
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Why is this important?
Parallel Computing
The Problem with Multicore
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Why is this important?
Because there is no turning back.
Importance
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
What do I want you to remember?
1
Technological advances in both fields would require that computer science and engineering collaborate - from this point on, neither side could go far alone. However...
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
What do I want you to remember?
1
Technological advances in both fields would require that computer science and engineering collaborate - from this point on, neither side could go far alone. However...
2
The greater burden still lies with the programmers and application designers.
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
What do I want you to remember?
1
Technological advances in both fields would require that computer science and engineering collaborate - from this point on, neither side could go far alone. However...
2
The greater burden still lies with the programmers and application designers.
3
The potential gains are immense...
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
What do I want you to remember?
1
Technological advances in both fields would require that computer science and engineering collaborate - from this point on, neither side could go far alone. However...
2
The greater burden still lies with the programmers and application designers.
3
The potential gains are immense... but so are the potential losses.
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
What do I want you to remember?
1
Technological advances in both fields would require that computer science and engineering collaborate - from this point on, neither side could go far alone. However...
2
The greater burden still lies with the programmers and application designers.
3
The potential gains are immense... but so are the potential losses.
4
The future of computing is definitely, irrevocably parallel. That much is certain. But...
Moore’s Law
Two Fronts
The Rationale for Multicore
Parallel Computing
The Problem with Multicore
Importance
What do I want you to remember?
1
Technological advances in both fields would require that computer science and engineering collaborate - from this point on, neither side could go far alone. However...
2
The greater burden still lies with the programmers and application designers.
3
The potential gains are immense... but so are the potential losses.
4
The future of computing is definitely, irrevocably parallel. That much is certain. But...a lot of details remain to be decided, so it is our duty to remain vigilant, educated, and informed.