Lecture17

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

CS 21 - Computer Organization and Assembly Language Programming Lecture 17 Moore’s Law and Parallelism University of the Philippines - Diliman College of Engineering Department of Computer Science

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Outline

1

Moore’s Law

2

Two Fronts

3

The Rationale for Multicore

4

Parallel Computing

5

The Problem with Multicore

6

Importance

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Outline

1

Moore’s Law

2

Two Fronts

3

The Rationale for Multicore

4

Parallel Computing

5

The Problem with Multicore

6

Importance

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

Moore’s Law

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

The amount of transistors that could be placed on a chip doubles every 2 years.

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

The amount of transistors that could be placed on a chip doubles every 2 years.

What powers Moore’s Law?

Moore’s Law

Two Fronts

The Rationale for Multicore

Transistor Feature Size

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Outline

1

Moore’s Law

2

Two Fronts

3

The Rationale for Multicore

4

Parallel Computing

5

The Problem with Multicore

6

Importance

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Interesting fact #1 about transistors

As transistors get smaller, their switching times get shorter.

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Interesting fact #1 about transistors

As transistors get smaller, their switching times get shorter.

What are implications of this fact for gate delays and operating frequencies?

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Two trends due to Moore’s Law

Moore’s Law opens up two fronts of attack in our bid to improve generation after generation of processors higher frequency more or bigger components

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Higher frequency processor operating frequencies have been constantly increasing since the 1970s, taking full advantage of the highest frequencies transistor technologies could give

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Mathematically speaking...

Which terms are affected?

CPUExecutionTime =

CPUclockcycles Clockrate

CPUClockCycles = numberofinstructions ∗ CPI

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Processor Clockrate Trend

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

More or bigger components more complicated components enable us to extract Instruction Level Parallelism from programs, enabling us to work on several instructions in parallel or to take less numbers of cycles when doing certain operations Examples: register files with multiple read ports advanced adder designs dedicated multiplier

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Mathematically speaking...

Which terms are affected?

CPUExecutionTime =

CPUclockcycles Clockrate

CPUClockCycles = numberofinstructions ∗ CPI

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Taken together

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Outline

1

Moore’s Law

2

Two Fronts

3

The Rationale for Multicore

4

Parallel Computing

5

The Problem with Multicore

6

Importance

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Interesting fact #2 about transistors

As the switching rate of transistors increase, their power consumption increases.

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

The Heat/Power Limit

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

The Heat/Power Limit

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

The CPU Cooling Challenge: Heat Sink and Fan

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

The CPU Cooling Challenge: Bigger Heat Sink and Fan

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

The CPU Cooling Challenge: Water Cooling

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Two trends due to Moore’s Law Moore’s Law opens up two fronts of attack in our bid to improve generation after generation of processors higher frequency more or bigger components

CPUExecutionTime =

CPUclockcycles Clockrate

CPUClockCycles = numberofinstructions ∗ CPI We lost one of the two!

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

I do hope you’re asking...

Couldn’t we just keep on building bigger and bigger processors?

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

I do hope you’re asking...

Couldn’t we just keep on building bigger and bigger processors?

Yes, we can but...

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Interesting fact about gates and wires

As transistors get smaller, switches(hence, gates) get faster.

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Interesting fact about gates and wires

As transistors get smaller, switches(hence, gates) get faster.

But the wires that connect those switches and gates, do not get faster.

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Interesting fact about gates and wires

As transistors get smaller, switches(hence, gates) get faster.

But the wires that connect those switches and gates, do not get faster.

In fact, at higher frequencies we spend more time waiting for the signals to arrive than actually working on the signals.

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

In other words...

More time waiting translates to more cycles, which means higher CPI.

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

In other words...

More time waiting translates to more cycles, which means higher CPI. Meaning, we have to minimize the distance that a signal travels within the processor.

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

In other words...

More time waiting translates to more cycles, which means higher CPI. Meaning, we have to minimize the distance that a signal travels within the processor.

So we could not keep on simply keep on building bigger and more complex processors.

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

A review...

We could no longer ramp up the clock rate because of power consumption issues

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

A review...

We could no longer ramp up the clock rate because of power consumption issues Transistors would still get smaller, giving us more transistors to work with as time goes by

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

A review...

We could no longer ramp up the clock rate because of power consumption issues Transistors would still get smaller, giving us more transistors to work with as time goes by But we could no longer build more and more complex processors because of the disparity between gate delays and wire delays

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

A review...

We could no longer ramp up the clock rate because of power consumption issues Transistors would still get smaller, giving us more transistors to work with as time goes by But we could no longer build more and more complex processors because of the disparity between gate delays and wire delays Given these conditions, how would we improve our processors with each new transistor tecnhnology generation?

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The solution

We keep the clockrate constant...

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The solution

We keep the clockrate constant...

We keep the processor design simple...

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

The solution

We keep the clockrate constant...

We keep the processor design simple...

We use our numerous transistors in putting multiple copies of our simple processor into our die

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

The solution

We keep the clockrate constant...

We keep the processor design simple...

We use our numerous transistors in putting multiple copies of our simple processor into our die This design is known as the multicore processor.

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The new trend in processor design

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Change in plans

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

The Dual Core Opteron

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Multicore processors and parallel computing

Using multicore processors belongs to the paradigm of parallel computing.

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Multicore processors and parallel computing

Using multicore processors belongs to the paradigm of parallel computing.

Is parallel computing new?

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Multicore processors and parallel computing

Using multicore processors belongs to the paradigm of parallel computing.

Is parallel computing new?

Not really.

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Outline

1

Moore’s Law

2

Two Fronts

3

The Rationale for Multicore

4

Parallel Computing

5

The Problem with Multicore

6

Importance

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Cluster Computers Beowulf Cluster 1 commercially available computers connected via a network doing parallel computations 2 utilizes F.O.S.S. such as Linux 3 nodes pass messages containing data to one another using MPI of PVM

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Beowulf Cluster

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Beowulf Cluster

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

Beowulf Clusters in UP Diliman DOST-ASTI

UP CSRC

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Grid Computers 1

nodes are distributed over a large geographic area, over multiple administrative domains

2

nodes are usually not dedicated computers

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

Grid Computing Example: SETI@home

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Grid Computing Example: Folding@home

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Outline

1

Moore’s Law

2

Two Fronts

3

The Rationale for Multicore

4

Parallel Computing

5

The Problem with Multicore

6

Importance

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Differences

How is multicore different from previous parallel computing approaches? most of the traditional parallel computing approaches have been applied to scientific computations many scientific computations are naturally parallel in nature they have few dependencies and they are come in very regular data structures and runtime execution patterns

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Differences

How is multicore different from previous general purpose computing approaches? previous processors were programmed pretty much the same way programmers just assumed that with each new processor the performance of their programs would automatically improve

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

The Multicore Problem

What are the problems with the Multicore Approach?

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

The Multicore Problem

What are the problems with the Multicore Approach? legacy programs would not simply improve automatically with multicore processors

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

The Multicore Problem

What are the problems with the Multicore Approach? legacy programs would not simply improve automatically with multicore processors programs have to be explicitly designed to take advantage of multicore processors

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

The Multicore Problem

What are the problems with the Multicore Approach? legacy programs would not simply improve automatically with multicore processors programs have to be explicitly designed to take advantage of multicore processors

some programs and applications are simply serial in nature

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

The Multicore Problem

What are the problems with the Multicore Approach? legacy programs would not simply improve automatically with multicore processors programs have to be explicitly designed to take advantage of multicore processors

some programs and applications are simply serial in nature most of the programs that could be parallelized or have parallel segments are not yet coded for parallel execution

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

The Multicore Problem

What are the problems with the Multicore Approach? legacy programs would not simply improve automatically with multicore processors programs have to be explicitly designed to take advantage of multicore processors

some programs and applications are simply serial in nature most of the programs that could be parallelized or have parallel segments are not yet coded for parallel execution it is difficult to program and debug in parallel!

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Multicore Processors and Amdahl’s Law

You have a program which is 50% parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a dual core processor?

Speedup =

1 (1 − p) +

p s

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Multicore Processors and Amdahl’s Law

You have a program which is 50% parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a dual core processor?

Speedup =

Speedup =

1 (1 − p) +

1 (1 − 0.5) +

p s

0.5 2

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Multicore Processors and Amdahl’s Law

You have a program which is 50% parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a dual core processor?

Speedup =

Speedup = Speedup = 1.333

1 (1 − p) +

1 (1 − 0.5) +

p s

0.5 2

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Multicore Processors and Amdahl’s Law

You have a program which is 50% parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a quad core processor?

Speedup =

1 (1 − p) +

p s

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Multicore Processors and Amdahl’s Law

You have a program which is 50% parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a quad core processor?

Speedup =

Speedup =

1 (1 − p) +

1 (1 − 0.5) +

p s

0.5 4

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Multicore Processors and Amdahl’s Law

You have a program which is 50% parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a quad core processor?

Speedup =

Speedup = Speedup = 1.6

1 (1 − p) +

1 (1 − 0.5) +

p s

0.5 4

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Limits in parallelizability

in reality, even parallelizable sections in programs are not infinitely parallelizable degree of parallelizability is dictated by nature and design of the program, algorithm, and the size of the dataset

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Multicore Processors and Amdahl’s Law

You have a program which is 50% parallelizable. The parallelizable section however, is three-way parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a quad core processor?

Speedup =

1 (1 − p) +

p s

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Multicore Processors and Amdahl’s Law

You have a program which is 50% parallelizable. The parallelizable section however, is three-way parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a quad core processor?

Speedup =

Speedup =

1 (1 − p) +

1 (1 − 0.5) +

p s

0.5 3

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Multicore Processors and Amdahl’s Law

You have a program which is 50% parallelizable. The parallelizable section however, is three-way parallelizable. Assuming that the parallelizable section was successfully parallelized, what is the speedup when such a program is run in a quad core processor?

Speedup =

Speedup = Speedup = 1.5

1 (1 − p) +

1 (1 − 0.5) +

p s

0.5 3

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Multicore Processors and Amdahl’s Law in a nutshell

The speedup caused by adding more cores to a multicore processor depends on the fraction of the workload that is parallelizable, and the degree of parallelizability of thos sections.

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

Multicore Processors and Amdahl’s Law in a nutshell

The speedup caused by adding more cores to a multicore processor depends on the fraction of the workload that is parallelizable, and the degree of parallelizability of thos sections.

Adding more cores to a processor that runs mostly on codes that have limited parallelizable sections is useless and would not bring any improvement.

Moore’s Law

Two Fronts

The Rationale for Multicore

Outline

1

Moore’s Law

2

Two Fronts

3

The Rationale for Multicore

4

Parallel Computing

5

The Problem with Multicore

6

Importance

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Why is this important?

Parallel Computing

The Problem with Multicore

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Why is this important?

Because there is no turning back.

Importance

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

What do I want you to remember?

1

Technological advances in both fields would require that computer science and engineering collaborate - from this point on, neither side could go far alone. However...

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

What do I want you to remember?

1

Technological advances in both fields would require that computer science and engineering collaborate - from this point on, neither side could go far alone. However...

2

The greater burden still lies with the programmers and application designers.

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

What do I want you to remember?

1

Technological advances in both fields would require that computer science and engineering collaborate - from this point on, neither side could go far alone. However...

2

The greater burden still lies with the programmers and application designers.

3

The potential gains are immense...

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

What do I want you to remember?

1

Technological advances in both fields would require that computer science and engineering collaborate - from this point on, neither side could go far alone. However...

2

The greater burden still lies with the programmers and application designers.

3

The potential gains are immense... but so are the potential losses.

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

What do I want you to remember?

1

Technological advances in both fields would require that computer science and engineering collaborate - from this point on, neither side could go far alone. However...

2

The greater burden still lies with the programmers and application designers.

3

The potential gains are immense... but so are the potential losses.

4

The future of computing is definitely, irrevocably parallel.

That much is certain. But...

Moore’s Law

Two Fronts

The Rationale for Multicore

Parallel Computing

The Problem with Multicore

Importance

What do I want you to remember?

1

Technological advances in both fields would require that computer science and engineering collaborate - from this point on, neither side could go far alone. However...

2

The greater burden still lies with the programmers and application designers.

3

The potential gains are immense... but so are the potential losses.

4

The future of computing is definitely, irrevocably parallel.

That much is certain. But...a lot of details remain to be decided, so it is our duty to remain vigilant, educated, and informed.