Supercomputer and Simulation - Research in Juelich (2/2008)

The Magazine from Forschungszentrum J端lich

RESEARCH in J端lich

SUPERCOMPUTERS AND SIMULATION :: PROTEIN ORIGAMI :: SPINS: TWISTING AND TURNING :: FAST COMPUTERS FOR BETTER BLOOD PUMPS

02|2008

RESEARCH in Jülich The Magazine from Forschungszentrum Jülich

A scientist is applying a 3-D simulation to analyse the flow of breath in the human nostril. He is using the pistol-like object in his hand to fire round particles which move with the flow.

Cover illustration: This simulation shows what happens when laser pulses (orange) hit the surface of a solid (green).

Amongst other things, they generate a cloud of electrons (yellow-red) that is accelerated forwards.

2

Research in Jülich 2 | 2008

Strong Simulation Science

T

ogether with theory and experiment, computer simulations form the third pillar of research work. They enable us to obtain insights and knowledge that has been previously inaccessible for physical, technical, financial or ethical reasons. Scien­ tists use supercomputers to investigate very dif­ferent questions. How are pollutants and trace substances distributed in the atmosphere and how do they in­ fluence our climate? Under what conditions can the simplest protein molecules be created from inani­ mate matter and thus form the building blocks of life? Why does nature make a distinction between a picture and its mirror image? Into what spatial shape are thread-like protein molecules folded in the body?

Join the scientists on their journey of discovery and find the answers to these questions in this issue of “Research in Jülich”.

Apart from our own research with and on super­ computers, at Forschungszentrum Jülich we have ­also set ourselves the task of providing scientists with ­supercomputer facilities of the highest performance class in the world. You can discover what this means in concrete terms in the article on “Race for World Leadership”. The following articles show how we ­develop methods and tools in order to provide sup­ port for researchers in the various fields of the com­ putational sciences. However, large computing capa­ cities alone are not enough; you also have to know how to use them. The great added value of Jülich ­supercomputers for research arises from the com­ bination of expertise in simulation science and the ­diversified scientific environment.

Here at Jülich we depend first and foremost on human brains and not on computer processors, which is why we also play a pioneering role in training ­German researchers in the simulation sciences. To­ gether with RWTH Aachen University, we operate the “German Research School for Simulation Sciences” (GRS), which offers masters and PhD courses for

2 | 2008 Research in Jülich

outstanding students. Forschungszentrum Jülich and RWTH thus at the same time also represent a model for cooperation between the universities and nonuniversity research organizations. Cooperation with RWTH Aachen University goes far beyond GRS alone.

The Jülich-Aachen Research Alliance (JARA) together with its computer simulation section (JARA-SIM) pro­ vides an institutional framework for joint research projects on a long-term basis.

In the Gauss Centre for Supercomputing (GCS), Jülich cooperates closely with the national super­ computing centres in Munich and Stuttgart. The Ger­ man supercomputer experts therefore speak with one voice on the international stage. GCS thus repre­ sents the German interests in the Partnership for ­Advanced Computing in Europe project (PRACE), which is preparing the basis for a transboundary ­supercomputing infrastructure in Europe. By focus­ ing the partners’ know-how and coordinating the ­application of funds, researchers throughout Europe will in future have access to supercomputing facili­ ties of a top international calibre. We are, moreover, also participating in the Gauss Alliance, which in ­addition to GCS incorporates the German regional and specialist computing centres. With our commit­ ment to this Alliance, to GCS and PRACE, we are ­endeavouring to sustain Jülich’s claim to be Europe’s leading centre for supercomputing and simulation sciences which can bear comparison with the best in the world.

Prof. Dr. Achim Bachem Chairman of the Board of Directors

Dr. Sebastian M. Schmidt Member of the Board of Directors

3

10 12 :: PROTEIN ORIGAMI Depending on how we fold a sheet of paper, we can create a variety of different figures. Threadlike protein molecules are also folded into very different structures. The simulation of these folding processes on Jülich supercomputers can help us to obtain a better understanding of diseases.

14 16

29 17

SPINS: TWISTING AND TURNING Elementary magnets in a thin layer of manganese on tungsten metal are always arranged in an anticlockwise spiral and never in a clockwise one. Jülich scientists have discovered the reason for this and have now set their sights on possible applications of this curious phenomenon.

FAST COMPUTERS FOR BETTER BLOOD PUMPS Scientists use supercomputers to optimize small implantable pumps for stabilizing patients’ circulation. Jülich simulation ­experts are helping to perform the required calculations fast and effectively.

4

Research in Jülich 2 | 2008

IN THIS ISSUE

3

Editorial

:: SNAPSHOTS FROM JÜLICH 6 Research at a glance A kaleidoscope of pictures show highlights of research in Jülich ranging from investigations of the air over Beijing ­during the Olympic Games and work on strained silicon up to and including a unique instrument providing a glimpse ­into the brain.

:: FOCUS 8 Computer simulations today and tomorrow 10 Turbulent atmosphere Jülich researchers track down air pollutants all over the globe.

12 Protein origami Into what shape are threadlike protein molecules folded in the body? Simulations provide the answer.

14 Computer simulation is an art 16 Spins: twisting and turning Why nature makes a distinction between a picture and its mirror image in some magnetic materials.

18 Virtual primeval broth High pressure, hot water and minerals – proteins could have emerged under these conditions on the primeval Earth.

:: Highlights 20 Race for world leadership A glimpse behind the scenes: how the fastest civilian computer in the world was installed at Forschungszentrum Jülich.

24 The future of supercomputing at Jülich Interview with Prof. Thomas Lippert, director at the ­Institute for Advanced Simulation and head of the Jülich Supercomputing Centre 26 An inside view Scientists from the Jülich-Aachen Research Alliance immerse themselves in artificial three-dimensional worlds – in future also from different locations.

29 Fast computers for better blood pumps Jülich simulation know-how helps to optimize life-saving blood pumps.

32 The doors of perception The computing power of supercomputers is increasing continuously, as is the quality of the simulations.

34 Supercomputing news Simulations that save human life, software for distributed computing, and an elite school for young scientists.

2 | 2008 Research in Jülich

5

MICROSCOPY ENTERS A NEW DIMENSION Ultrahigh-resolution electron microscopy has enabled atomic distances to be measured down to a few picometres, as Jülich scientists reported in the high-impact journal “Science”. A pico­ metre is a billionth of a millimetre. The displacement of atoms in this order of magnitude decides on whether many material properties are technically relevant, for example the ability of semiconductors to bear nanoelectronic structures or that of ­superconductors to conduct current almost loss-free.

STRAINED SILICON An innovative material from Jülich will make electronic devices smaller, faster and more economical. Strained silicon is distin­ guished from conventional silicon by its elongated crystal lattice and improved electronic properties. The Jülich team headed by Prof. Siegfried Mantl has been granted funds totalling € 3.2 mil­ lion from the Federal Research Ministry and from industry as part of the DECISIF collaborative project in order to further the development of application-oriented technology.

Research at a Glance This magazine focuses on Jülich research using supercomputers as a “tool”. However, Jülich scientists also score great successes in other research areas. We present a few snapshots of this work here.

LINK TIP www.fz-juelich.de

6

Research in Jülich 2 | 2008

SNAPSHOTS FROM JÜLICH

HONOURS FOR THE NOBEL LAUREATE Prof. Peter Grünberg, the Nobel laureate from Jülich, was awarded the Grand Cross with Star of the Order of Merit of the Federal Republic of Germany, and the Order of Merit of the Federal State of North Rhine-Westphalia (NRW). The physicist was personally honoured by the German President, Horst Köhler, and the Prime Minister of NRW, Jürgen Rüttgers. In 1988, Prof. Grünberg ­discovered giant magnetoresistance (GMR) – which is now used every day in computers. With more than five billion read heads produced to date, statistically there is one GMR sensor for every member of the human race.

OLYMPIC ATMOSPHERE Atmospheric researchers from Jülich were at the starting line for the Olympic Games in Beijing. Prof. Andreas Wahner was involved in a German-Chinese meas­ uring campaign which investigated the air quality in the Chinese capital during the games – ranging from fine dust to pollu­ tion with aerosols, nitrogen oxides, ozone and formaldehyde. The evaluation aims to show how air pollution can be effec­ tively combated in future.

2 | 2008 Research in Jülich

RESEARCHERS COME OUT OF THEIR IVORY TOWERS “Nowadays a leading scientist must also be prepared to estab­ lish contact with the mass media and to present his research to the general public.” This is the conclusion reached by Prof. Hans Peter Peters from Jülich in a report published in “Science” on a study he headed. On the whole, scientists are more satisfied with journalistic reporting on their work than one would imagine in view of the general prejudice concerning the “ivory tower”. In fact, about 40 percent of the researchers interviewed from the five leading scientific nations find public reporting conducive to their career.

THE FRAGRANCE OF AGARWOOD There is a great demand for illegally felled agarwood. One kilo of the tropical wood fetches up to $ 10,000. The oriental ­fragrance of its resin is found in lotions and perfumes. Jülich scientists are ­working on a long-term project in Indo­ nesia to discover whether agarwood of similar quality can be cultivated.

LOOKING CLOSELY AT THE BRAIN The date is set for April 2009. After two years of preparation, a unique large-scale instrument will be put into operation at Jülich to map structures and metabolic processes in the brain with as yet un­ achievable precision. It is a combined system consisting of a magnetic reso­ nance tomograph (MRT) and a positron emission tomograph (PET). The coils of the MRT generate a magnetic field of 9.4 tesla, which is almost 200,000 times as strong as the Earth’s magnetic field.

7

8

Research in Jülich 2 | 2008

FOCUS

:: Computer simulationS TODAY AND TOMORROW Computer simulations are used by scientists from all disciplines to obtain new findings. This magazine focuses on four examples: scien­ tists investigate the atmosphere and climate, biologically important substances, basic material properties and also chemical processes that cannot be recreated in the laboratory. In doing so, they profit from the continuously increasing computing capacity of the Jülich super­ computers. In this way, scientists will be able to study more complex processes and structures in future – an advance that is represented symbolically by the diagram showing columns of figures that turn first into the molecules of simple gases and finally into the complete model of a protein.

2 | 2008 Research in Jülich

9

Turbulent atmosphere Atmospheric pollutants travel around the world. How far they get depends on how fast they are degraded in chemical processes. Jülich scientists track the gases using their supercomputers and thus determine how air pollution influences climate.

I

f the air over Europe isn’t clean then this is not necessarily the fault of the Europeans since the wind carries the exhaust gases from North American cars and factories over to us. Polluted air doesn’t stop at the borders of countries or indeed continents. “Who is therefore responsible for air pollution when and where? Until we can answer this ques­ tion, it is difficult to develop appropriate countermeasures,” says Dr. Martin Schultz. The expert from Jülich’s Institute of Chemistry and Dynamics of the Geo­ sphere (ICG) continues: “Satellite obser­ vations and other measurements give us valuable pointers, but cannot provide ­data for all locations and dates.”

The only way out of the dilemma is provided by computer simulations which model the path of the pollutant and trace gases in the atmosphere. This sounds

10

easy, but it isn’t really. The starting point is government reports on how much waste gas is emitted by each country. ­Researchers must then take into consi­ deration all of the chemical processes in the air during which pollutants are con­ verted or degraded. Furthermore, atten­ tion must also be paid to the influence of the wind, solar radiation and temperature.

FROM EUROPE TO ASIA The team headed by Schultz has there­ fore skilfully coupled an existing com­puter model for simulating chemical pro­cesses in the troposphere – the lowest layer of the atmosphere – to the global climate model ECHAM5. “In this way we are able to show how European, American or Asian emissions influence the distribution of pollutants in the atmosphere,” Schultz ­explains. Amongst other aspects, the

Weather balloons provide important data for atmospheric researchers. But these data alone do not enable ­scientists to monitor the route of ­atmospheric ­pollutants.

Research in Jülich 2 | 2008

FOCUS

This simulation shows as an example the way in which emissions of the atmospheric pollutant CO from sources in Europe (green), North America (orange), Southern Asia (blue) and Eastern Asia (yellow) are distributed in the atmosphere.

s­ cientists also investigated carbon mon­ oxide – CO. The atmospheric pollutant CO indirectly influences the concentra­ tion of greenhouse gases in the atmos­ phere and thus the climate. Since CO is only degraded after one to two months, it can be used to track the routes of trace gases. This is sufficient time for an air parcel to travel once around the world.

One result of the simulation is that CO from European sources is generally trans­ ported in atmospheric layers close to the ground across Asia towards the east, but, above all in winter, the air can also ­pollute vast expanses of the Arctic.

However, pollutant and trace gases are not only distributed from south to north and from west to east, but also from bottom to top. “In the tropics, air pollutants can reach atmospheric layers at altitudes of more than 16 kilometres which are otherwise isolated from lower layers. This is, for example, true of CFCs, which are largely responsible for the de­ velopment of the ozone hole above the poles,” explains Dr. Rolf Müller from ICG. With the aid of the Jülich computer mod­ el, ClaMS, he simulates chemical pro­ cesses in the stratosphere. In mid-lati­ tudes, this part of the atmosphere begins about ten to twelve kilometres above the surface of the Earth and thus just above the usual cruising altitude of passenger aircraft. ClaMS makes use of data from the weather services at the nodal points

2 | 2008 Research in Jülich

Simulation with the Jülich ClaMS computer model: the Asian monsoon circulation over the Himalayas can be seen with tro­pospheric air at its heart – recognizable by the high CO content (red) – which is rapidly transported upwards.

of the virtual lattice that spans the globe.

Fed with information on wind direction, temperature and pressure, the computer follows the path of individual stratospher­ ic air parcels separated from each other by a distance of about 100 kilometres.

The concentration of trace gases in the respective air parcel changes continu­ ously as a result of the simulated chemi­ cal processes.

During the evaluation of satellite data, it recently became apparent for the first time that small quantities of prussic acid (HCN) were occasionally found in the stratosphere above the tropics and CLaMS was able to successfully model the distribution of this trace gas. “The poisonous HCN has no direct effect in these low concentrations. However, if we understand how the HCN gets into the stratosphere, then we will also be able to better understand the transport routes of other trace and pollutant gases,” Müller explains. Under special wind conditions, HCN resulting from forest fires in the tropics can rise up to the lower edge of the stratosphere. The Jülich computer models point to the fact that HCN is mainly carried into the stratosphere by a process of fairly slow and continual mix­ ing – and not, for example, sporadically with the aid of gigantic tropical thunder­ clouds.

In future, the research group headed

by Müller intends to reduce the distance

between the simulated air parcels and to use CLaMS to track the input of trace gases from the tropics into the strato­ sphere over a period of several years.

“We wouldn’t be able to contemplate such a project if we had the same com­ puting power as we did ten years ago,” says Müller.

IMPROVED FORECASTS The same is true of Schultz’s team: “We have only recently become able to calculate models that include the relation between air pollution and climate – thanks to the new generation of super­ computers. And we are now expanding our model to include the chemical pro­ cesses in the stratosphere which we have not been able to consider so far due to a lack of computer power,” adds Schultz.

He is certain that this will make climate forecasts even more reliable. His opinion is shared by the other experts on the Intergovernmental Panel on Climate Change – the winner of the 2007 Nobel Peace Prize. With the results of their sim­ ulations, the Jülich researchers will once again be able to make a valuable contri­ bution to the next World Climate Report, which is planned for 2013 and will open the eyes of politicians, companies and the general public to the extent and con­ sequences of future climate change. :: Frank Frick

11

Protein origami Depending on how we fold a sheet of paper, we can create a variety of different figures: a frog, a crane, or an elephant. In the same way, three-dimensional structures are formed by folding threadlike protein molecules. In this form, proteins can take over important functions in the body. The simulation of these folding processes on Jülich supercomputers may help us to gain a better understanding of diseases and provide a starting point for new treatments.

W

hether elongated muscle pro­ teins, Y-shaped antibodies or elaborately formed enzymes with pockets and depressions – proteins have different shapes depending on their func­ tions. Yet all of them are composed of a chain of small building blocks – the amino acids. Their properties and therefore the forces that act between them, combined with the surrounding water, determine the shape of a protein molecule. The dramatic effects an error can have are shown by the example of prion proteins: incorrect folding can cause mad cow disease in cattle, scrapie in sheep and the lethal Creutzfeldt-Jakob disease in humans.

12

Experiments are a time-consuming and expensive method of determining the shape of many protein chains, even when the precise amino acid sequence is known. An easier approach is therefore often employed where the physical pro­ perties of components in a row are used to calculate the folding of the chain – ab initio, as the scientists say. However, because a protein is often composed of several hundred amino acids, such cal­ culations are extremely time-consuming.

This is where a supercomputer like JUGENE in Jülich comes into play. And of course, a clever strategy is also ­required for the simulation, such as that

developed by Prof. Adam Liwo from the University of Gdansk in Poland.

Liwo starts from a “coarse-grained” approach. This means that he does not need to incorporate each individual atom of the protein into the calculations, but initially just the most important inter­ actions of each amino acid. “You can com­ pare this with two strategies used when looking for mushrooms,” Liwo explains.

“You can either systematically comb ­ev­ery inch of the ground, including sandy areas and ponds. Or you first look under trees, where you know from experience that mushrooms grow.” The first strategy corresponds to the calculation of all the

Research in Jülich 2 | 2008

FOCUS

atoms. “It may be more precise, but it would take the entire mushroom season,” says Liwo. By limiting the search to the interesting areas, we end up with a full basket of mushrooms much faster – or with an idea of how a protein is formed.

Only then does Liwo take a closer look at important areas of the molecule in order to simulate them atom by atom. This isn’t 100 % accurate at the moment but the virtual molecules are starting to resem­ ble nature more and more closely.

The simulations are based on the prin­ ciple that under natural conditions a pro­ tein chain takes the shape requiring the least energy – similar to the way that wa­ ter collects in a hollow when left to itself instead of flowing over the edges. If the researchers want to calculate the forces that shape the protein, then they first use the fundamental laws of physics to con­ struct a field of forces. The simulation is, so to speak, trained by a few small pro­ teins whose folding sequences and natu­ ral shape are known.

When it is fed this information, the ­supercomputer runs through all possible folding sequences and predicts the shape of unknown proteins. “This has proved successful for chains of more that 300 amino acids to date,” reports Liwo. His plan is to focus on long muscle proteins next, as well as enzyme complexes made up of a number of subunits, each com­ posed of between 200 and 500 amino acids.

FABULOUS COMPUTING POWER Liwo is convinced that this is possible with a computer as powerful as JUGENE. He runs his simulations in Jülich: “Be­ cause that is where the largest multiprocessor computer in Europe is, and Jülich is an excellent research centre.”

He also works on supercomputers in the USA and a small computer in his home town of Gdansk. “The more you eat, the hungrier you get,” was his comment with regard to the constantly growing need for computing capacity. “In the 80s, for my PhD, I calculated the shape of a small peptide made up of nine amino acids in a time-consuming procedure using a Rus­ sian RIAD computer – an exercise that my students can solve in two hours ­today.” And just five years ago, he hadn’t even dreamed of simulating a protein chain with 500 links, Liwo says. Now, he

2 | 2008 Research in Jülich

thinks it will be possible to simulate whole areas of the cellular machinery – in other words, the interaction between a number of proteins. A suitable tool for this purpose could be the European peta­ flop supercomputer, which Jülich is devel­ oping together with its partners within the framework of the PRACE partnership.

JOINT PROJECTS Contact between Liwo’s research group and Forschungszentrum Jülich dates back to a comprehensive initiative in 2004. Outstanding scientists from the new EU member states were offered ac­ cess to the Jülich computers on the same

Computer calculations show how an important part of a human protein is folded. In the first picture on the left of this sequence, the folding procedure has already begun, and the picture on the right shows the final shape of the protein section.

conditions as German researchers. “In this way, we opened our doors to the very good theoretical scientists in Eastern ­Europe, and they were able to make use of our excellent infrastructure,” explains Dr. Norbert Attig from the Jülich Super­ computing Centre. “All of us in Europe profited from this move.” Large colla­ borative projects quickly developed.

“The tedious simulations of protein folding were perfect for calculations on the JUBL and JUGENE supercomputers,” recalls Norbert Attig. “On the one hand, they require a lot of computing time, and on the other, they can be easily divided up between a large number of proces­ sors.” In other words, a job for JUGENE with more than 65,000 processors. The application from the working group in

Gdansk was rated as excellent by the ­independent panel of experts for the John von Neumann Institute for Computing (NIC) – the facility that allocates comput­ ing time on the Helmholtz Association’s supercomputers. Since then the Jülich supercomputers have been shaping ­virtual protein chains.

The origami of the protein molecules on supercomputers is in no way an aca­ demic gimmick, Liwo quickly adds. Simu­ lations can help us to understand the ­coagulation of proteins in Alzheimer’s disease and to find a basis for the devel­ opment of medications. Another aim is to determine the interactions of proteins and genes – a central task, for example, in cancer research. :: Wiebke Rögener

13

1

3

2

4

Computer simulation is an art The supercomputer has developed into a universal research instrument: it is, for example, a virtual telescope, test tube and microscope all at the same time.

If you have mastered the art of using the supercomputer properly then you will be rewarded with important results and also some breathtaking pictures.

1

14

Stars move through the universe at different speeds.

The ­examples coloured red in this cluster of stars are slow while the yellow ones are fast.

2

Laser pulses (left) hit the surface of a solid (blue disc) and produce a cloud of electrons (right).

Research in Jülich 2 | 2008

FOCUS

7

8

5

6

9

7

Such simulations help researchers to understand the mechanical properties of cell membranes. Under the influence of forces, an elastic network forms a complexly folded structure.

Three wells (blue) collect water from the ground. They are fed by groundwater that flows in the direction of the green arrows. This water transports a pollutant (yellowish brown cloud) along with it.

8

Technicolor world: the colours illustrate different concen­ trations of the trace gas ozone in the atmosphere 38 kilo­ metres above the ground.

The spatial structure of a protein which results from the folding of long protein chains. Incorrect folding occurs, for example, in Alz­heimer’s disease.

9

The flow profile along the mixing chamber of a reformer that is used to obtain a hydrogen-rich gas from fuels such as diesel or kerosene. This gas can be used by fuel cells to generate electricity.

3

A simulated quark-antiquark pair under the influence of binding forces. As far as we know today, quarks are the most elementary building blocks of nature.

4

5

6

Swirling eddies: the structural details inside a turbulent flow of gas.

2 | 2008 Research in Jülich

15

Spins: twisting and turning Nature is fussy. Elementary magnets in a thin layer of manganese on tungsten metal are always arranged in an anticlockwise spiral and never in a clockwise one.

An explanation for this was found by a team of scientists from the Institute for ­Advanced Simulation (IAS) by pushing the Jülich supercomputer JUMP to its limits.

The first applications of this curious phenomenon are already in sight.

P

rof. Stefan Blügel from IAS recalls how it all began. “My colleague Mat­ thias Bode from Hamburg told me about a strange result obtained by one of his PhD students involving a magnet­ izable material that was divided into ­domains.” This means that there are ­so-called boundaries within the material where the direction of the magnetization changes abruptly. Domains are areas of identical magnetization within these boundaries.

16

The direction of the magnetic field is predetermined on an atomic level by the direction of the electron spin. To put it simply, the spin is like a tiny bar magnet which is aligned to the external magnetic field and which itself also generates a magnetic field. Between the domains, the spin directions between the adjacent electrons must therefore be twisted against each other because the direction of the magnetic field changes here.

However, the change of direction does not occur quite as abruptly as it might appear from the outside. In fact, directly adjacent spins are only rotated against each other at a small angle. Only by stringing together a large number of

neighbours does the overall angle of rota­ tion add up in such a way that it is in agreement with the magnetic orientation of the adjacent domains.

FORGOTTEN FORCE RE-EMERGES “Our colleagues from Hamburg then noticed that the spins always rotate in the same direction,” says Blügel. The sci­ entists had, in contrast, expected that the spin direction would be randomly dis­ tributed – anticlockwise in one transition wall and clockwise in another. After his conversation with Bode, Blügel vaguely recalled that about 50 years ago a Rus­ sian and a Japanese scientist had pre­ dicted an effect caused by the interaction

Research in Jülich 2 | 2008

FOCUS

Institute for Advanced Simulation (IAS) Since 1 January 2008, supercomputing and the simulation sciences have been u­ nited under the roof of IAS, which currently consists of the Jülich Supercomputing Centre and three scientific institutes that use supercomputers in order to investigate the structure and properties of various materials up to and including biological ­systems.

One of these institutes is headed by Prof. Stefan Blügel. He is convinced that: “The speed with which we obtain new findings will in future increasingly depend on the intelligent application of computers. The optimal working atmosphere for such ­endeavours is created at IAS by uniting specialists from various disciplines at one ­institute.” It is planned to integrate other working groups, for example from environ­ mental and health research.

Left-hand picture: The red and green arrows represent spins that can be visualized as small elementary magnets. The upper part of the picture shows how the spins are arranged in a thin manganese layer and below it the mirror image that does not exist in nature.

Pictures below: Spins (yellow arrows) in different materials.

between the orbital motion of an electron and its spin. To date experts had as­ sumed that the force of this effect would be too small to have an impact such as that observed in Hamburg. Blügel, how­ ever, began to doubt this assumption.

“Two things came together in this way.

Bode’s group had a high-resolution scan­ ning tunnelling microscope that was able to detect the strange preference for one spin direction and here in Jülich we had the supercomputer which we could use to ­explore the possible causes,” says Blügel.

With his computer simulations Blügel was indeed able to reconstruct and ex­ plain the observations made by the Ham­ burg team. The simulations showed how the spin direction depends on the proper­ ties of the respective material. “The fact that there is a preferred spin direction at all is a consequence of a quantum-theo­ retical property,” Blügel explains. “Where­ as in conventional mathematics three times four always gives the same result as four times three, this permutability does not exist for certain quantum-theo­ retical values.”

The Jülich team then used simulations to discover the materials in which this ­effect is particularly strong. The Hamburg

2 | 2008 Research in Jülich

group then established which of these materials is especially easy to handle in reality. Finally, the two groups came down in favour of tungsten coated with a thin layer of manganese. “Everything worked,” says Blügel. The effect was so strong that it did not occur in the domain walls but was clearly measurable in the form of spirals in the manganese layer. “This ­finally led to a publication in the high-­ impact journal Nature,” Blügel proudly adds.

SPINS AND POLARIZATION The Jülich researcher then focused on possible applications for the phenome­ non. “The spirals can be pushed aside by electric current. Their position could therefore be used for storing informa­ tion.” However, the behaviour of the spi­ rals in incident light is also promising.

The electron spins influence the polariza­ tion of the light reflected at the man­ ganese surface. Polarized light is used, amongst other things, for 3-D projections – holograms – which make it possible for us to immerse ourselves in artificial ­computer-generated worlds. :: Axel Tillemans

17

Virtual primeval broth Prof. Dominik Marx from the University of Bochum uses the Jülich supercom­puters to investigate how the simplest protein molecules could have originated more than four billion years ago – long before there was any life on Earth. High pressure, high temperatures and sulphur-containing minerals may have played a part in the origins of life.

C

harles Darwin, the father of evolu­ tionary theory, put forward the idea that the first protein mole­ cules could have originated in “some warm little pond” with all possible ammo­ nium and phosphorus salts, with light, heat and electricity. He was, however, ­unable to say how the first step could have been taken from inanimate to ani­ mate matter.

The first clues were provided in 1953 by the American Stanley Lloyd Miller.

When he reconstructed the possible con­ ditions of the primeval atmosphere in a glass flask simple organic molecules were formed – the amino acids. However, the way in which these building blocks were combined to form proteins and the part that might have been played by ­minerals containing iron and sulphur ­remained a matter of speculation.

18

A research team headed by Dominik Marx, head of the Chair of Theoretical Chemistry at Ruhr University Bochum, has now been able to show for the first time on the Jülich supercomputer that under conditions still found today at hot volcanic vents in the deep ocean it would indeed have been possible for amino ­acids to combine to form protein chains – even without the biological tools normally required for cells to produce proteins.

SIMULATION NOT SPECULATION The scenario of the primeval iron and sulphur world was not simply recon­ structed in a chemical laboratory. The “warm little pond” was created virtually on Jülich’s JUBL supercomputer. Two ­simulated amino acids – simple glycine molecules – only came into contact on the computer. “In our calculations we

proceed from the basic chemical building blocks, that is to say from the electrons and atomic nuclei of the molecules in­ volved,” Marx explains. No experimental data of any sort were used in these simu­ lations. The scientists only fed the com­ puter with information on the properties of the molecular building blocks involved.

The computer then calculated the behav­ iour of the molecules dissolved in water under various conditions.

It was found that the higher the simulated pressure and the hotter the virtual water, the more easily a peptide bond was formed between two amino acids – the fundamental process in protein synthe­ sis. The molecules were exposed to a pressure of up to 20 megapascals, which is roughly 200 times the atmospheric pressure at sea level. The temperature was raised to more than 200 degrees

Research in Jülich 2 | 2008

FOCUS A hot volcanic vent in the deep ocean. Here we still find conditions today similar to those that prevailed during the Earth’s early history which could have enabled amino acids to combine into protein chains.

Celsius. “It would be extremely difficult to perform such experiments in the labo­ ratory in a controlled manner,” Marx ex­ plains. Water has quite different proper­ ties under such extreme conditions than in a normal state and the amino acids dissolved in it react with each other more readily. When pyrite containing iron and sulphur was furthermore added to the virtual primeval broth, the protein build­ ing blocks came together even faster – the mineral acts as a catalyst accelerat­ ing the reaction. “We still don’t know exactly how it works. Pyrite is a model for a number of iron and sulphur compounds that might play a part here,” Marx is quick to add. “Nor do we know whether the first protein molecules were actually formed in the same way as in our simula­ tions. However, by using the supercom­ puter we were able to show that this path is possible.”

VAST COMPUTING TIME REQUIRED An enormous amount of computing time was required to simulate the early history of life on Earth. “If we add every­ thing up then more than 2,000 proces­ sors in JUBL were working on this for four months,” says Dr. Norbert Attig from the Jülich Supercomputing Centre. Alto­ gether only about 200 research teams are granted access to one of the Jülich supercomputers per year, and only a frac­ tion of them have access to JUGENE.

Minerals containing iron and sulphur, such as this pyrite, facilitate the formation of protein chains.

2 | 2008 Research in Jülich

The amino acid glycine (left) is activated in hot water at the interface to pyrite (centre) and then combines with another glycine molecule to form a dipeptide (right).

“In principle, every researcher in Europe could use our computing facilities,” says Attig. “However, we receive five to seven times more applications than the com­ puting time available.” An international panel of experts evaluates the applica­ tions and makes recommendations for allocating the supercomputer resources.

The evaluation is based on two criteria.

Does the project promise to provide new and relevant findings? And can it be ­efficiently processed on a large super­ computer?

Both of these criteria applied to Marx’s project. “It is quite simple an extremely interesting question,” says Attig. Scien­ tists were first able to explore this issue by using the Jülich Blue Gene computer.

The panel of experts for the John von Neumann Institute for Computing (NIC) – the institution that allocates computing time on the supercomputers of the Helm­ holtz Association – awarded the project the accolade of “Excellence Project 2008” and the first results have been published in the high-impact “Journal of the American Chemical Society”. The ­impressive findings have also received considerable media coverage.

Dominik Marx hopes that even more powerful computers will soon be avail­ able for his simulations. “JUGENE offers us fantastic new opportunities but our model assumptions are unfortunately still very idealized. More powerful com­ puters would permit us to create more realistic models.” He has reason to be optimistic for the future. “The Gauss Centre for Supercomputing – the associ­ ation of three national supercomputing centres – has recently taken over the coordination of supercomputing activities in Germany and is helping to ensure that Germany will speak with one voice inter­ nationally with respect to supercomput­ ing,” he says. Moreover, on a European level, PRACE (Partnership for Advanced Computing in Europe) will further improve the European supercomputing infra­ structure. “However, networking existing centres alone is not enough,” Marx is convinced. “We also need sustainable ­investments in new computers of the highest performance class in order to ­remain competitive with colleagues in the USA and Japan.” :: Wiebke Rögener

19

Race for world leadership Buying a computer is one thing – getting it to work is another. What is all too familiar to many long-suffering PC owners applies all the more when the computer is “super” and is expected to bring home gold in the race for the fastest computer in the world.

S

eptember 2008: an engineer from the manufacturing company hands Hartmut Peters from the Jülich ­Supercomputing Centre (JSC) the official installation manual for the Blue Gene/P supercomputer. The 73 pages contain ­details of how the cabinets the size of a tele­phone box housing the computer – the so-called racks – together with their 65,536 processors should be wired up and supplied with air and electricity. A non-event? By no means – due to the ­timing. The Jülich Blue Gene/P – known

20

as JUGENE for short – had already been in operation for more than six months.

“We were the first institution in the world to install a computer of this type and size – parts of the instructions hadn’t even been written and other parts were incor­ rect,” says Peters. Together, the experts from Forschungszentrum Jülich and the computer manufacturer IBM put the ­finishing touches to the supercomputer JUGENE. Their experience was then ­incorporated into the official version of the installation manual.

“Nowadays, no manufacturer can ­afford to first set up a computer of this size on its own premises, to operate it as a reference and then to supply customers

with machines of identical design,” ­explains the administrator Jutta Docter from JSC. The experts employed by the first purchaser of a new supercomputer generation play a key role. On top of this, a supercomputer must operate in tandem with data memories and a large number of other computers – and at Jülich also with other supercomputers and European computing centres. Some problems first arise in such an environment and cannot be foreseen by the manufacturing com­ panies in their development laboratories.

“We discover how to solve these prob­ lems in cooperation with the manufac­ turer,” says Olaf Mextorf, network expert at JSC.

Research in Jülich 2 | 2008

HIGHLIGHTS

View of the computer room in Forschungszentrum Jülich. The series of pictures on this page and the following double-page spread document the installation of the JUGENE supercomputer in October 2007.

COMING UP TRUMPS Since computer technology is devel­ oping at breakneck speed, the installa­ tion of a supercomputer is a race against the clock. The aim is that the users – in the case of Jülich, the scientists – should be able to perform computer simulations using the maximum power currently avail­ able. They thus achieve results faster than other computer research teams and can then be the first to publish them, for example in a high-impact journal. Fast supercomputers therefore give scientists a clear competitive edge. But scientific progress is also closely linked to super­ computing capacity in a quite different way. “More computing power also always means a leap in quality. The results of the simulations become more accurate and reliable,” says Prof. Thomas Lippert. The head of JSC continues: “Sometimes we even discover completely new effects.”

The supercomputer then opens up new

2 | 2008 Research in Jülich

fields of research. Since the world’s lead­ ing scientists want to perform their simu­ lations on the world’s fastest computers, computers of the highest performance class attract the best researchers to Jülich.

As far as computing power is con­ cerned, experts, funders and journalists always keep their eye on the TOP500 List of the fastest computers in the world.

This list is compiled every six months and is published each June and November.

The list influences the plans of the super­ computer manufacturers and their devel­ opment partners and leads to additional pressure of time. The situation is similar to that of a world-class athlete. He must be top fit in time for the Olympic Games otherwise he will be eliminated in the early heats and won’t get a chance to compete for a medal. Nevertheless, the athlete is still in a relatively relaxed situa­ tion compared to the computer manufac­

turer. A sprinter still has the chance to win a medal at the next Olympic Games.

Due to technological progress, however, a supercomputer will always perform worse in the next TOP500 List than in the previous one.

AROUND THE CLOCK In September 2007, JSC and IBM in­ formed the publishers of the TOP500 List of their plans to upgrade the Jülich super­ computer JUGENE. On 1 October the first computer racks arrived in Jülich. In order to be included in the TOP500 List, ­JUGENE had to be fully installed and its computing power demonstrated within four weeks. This meant that up to six technicians were working day and night.

It took them just eleven days to set up and connect the racks. The first tests were run. Meanwhile, experts in Jülich and the USA worked in three shifts to ­optimize the operating software and to

21

Stowed away in 16 racks, the more than 65,000 processors in JUGENE made it the fastest civilian computer in the world at the time of its launch.

write additional diagnostic programs. ­Finally, they attempted to start the ­Linpack benchmark that is decisive for the TOP500 List in order to measure the computing power.

“Here at JSC we were naturally all dying to get on the list, as were the IBM staff,” Jutta Docter recalls. Her office was opposite the room used by the IBM tech­ nicians working on JUGENE. “I was there­ fore not only excited myself but also felt their tension rising.” After several at­ tempts, the experts finally managed to get the first presentable result just in time. According to the Linpack test, ­JUGENE achieved a power of more than 167 trillion arithmetic operations per ­second or 167 teraflop/s. This is an al­ most unimaginable result. If every person if the world were able to perform one arithmetic operation of addition or multi­ plication per second, then the combined intellectual power of all humanity would still be working 30,000 times more slowly

22

than the computer. During the course of 2008, the performance was actually increased to 180 teraflop/s.

FASTEST COMPUTER IN EUROPE Due to this achievement, JUGENE was able to call itself the “fastest civilian super­ computer in the world” for six months. As expected, computing power ages rapidly in the fast-moving computer world, but nevertheless even in the next TOP500 List in July 2008, JUGENE was still able to maintain a good sixth place and remained the fastest computer in Europe.

Just four short weeks of – fevered – activity in order to install a supercomputer of this size doesn’t seem very much. In­ deed, this record-breaking tempo was only possible because the Jülich comput­ er specialists had already made meticu­ lous preparations well in advance. Strictly speaking, the story began five years ago.

At that time, IBM confided to Klaus Wol­ kersdorfer, head of “High Performance

Systems” in Jülich, that they had plans for a new type of supercomputer. “It was so top secret at the time that at first I couldn’t even talk to my colleagues about it,” Wolkersdorfer recalls. Instead of con­ tinuing to use relatively few extremely powerful processors, IBM intended to ­install an extremely large number of rela­ tively slow processors. Once relieved of all non-essential functions, these proces­ sors should be capable of communicating quickly with each other and rapidly ac­ cessing their working memory. “Whereas in 2004 a large number of German ex­ perts, for example on the Supercomputer Committee of the National Science Coun­ cil, remained very sceptical about this con­ cept, we regarded it with great interest,” says Thomas Lippert. The head of JSC continues: “It was immediately apparent to us that this was a chance to break down the barriers of the conventional ap­ proach with respect to electricity con­ sumption, space requirements and cost.”

Research in Jülich 2 | 2008

HIGHLIGHTS

Pressing the button officially launching JUGENE for its users (from left to right): Achim Bachem, Chairman of the Board of ­Directors of Forschungszentrum Jülich, Jürgen Rüttgers, Prime Minister of North Rhine-Westphalia, Thomas Rachel, Parlia­ mentary State Secretary of the Federal Research Ministry, and Martin Jetter, General Manager and Chairman of the Board of IBM Deutschland GmbH.

Therefore in summer 2005 the JSC ex­ perts tested a new rack design with 2,048 processors. “It was a smash hit,” says Wolkersdorf laconically. The Jülich researchers had namely discovered that part of the simulations were indeed read­ ily scalable, as it is known in the techni­ cal jargon. The more processors the sci­ entists used, the faster the calculations were finished, without the additional data flow impairing the performance. Forsch­ ungszentrum Jülich thereupon ordered an additional seven racks which, together with the existing rack, formed the super­ computer JUBL (Jülicher Blue Gene/L) from early 2006.

FULLY BOOKED SEVEN TIMES OVER Just a few months later it was clear that even JUBL would not be enough to satisfy the scientists’ demand for com­ puting time. “We were fully booked seven times over,” says Docter. Then came the information that IBM was working on a

2 | 2008 Research in Jülich

new Blue Gene generation. In mid 2006, Thomas Lippert and his team therefore began work on financial planning for ­JUGENE. Specific preparations for its ­installation in October then started in spring 2007. For example, new cable routes were required for the power sup­ ply leading from the external transformer station to the computer room. “We also had to decide on the location of the racks, calculate the resulting load on the 80-centimetre-high false floor, cut out ­access holes for the air supply at various points, and also lay twelve kilometres of network cables in the floor,” says Hartmut Peters.

After JUGENE had then successfully passed the Linpack test at the end of Oc­ tober this still didn’t mean that it was ready for the simulation scientists. “For example, the supercomputer was still not communicating with the server comput­ ers required to store the data or to allow access to the various research teams”,

says Olaf Mextorf. Furthermore, even the first tests and the Linpack test runs had shown that not all of the more than 65,000 processors were functioning as required. Only then were the JSC experts able to thoroughly analyse the faults and eradicate them. Telephone and video conferences with the American techni­ cians and managers were part of the daily routine. Many a spare part was brought to Jülich from the airport at night by taxi so as not to lose any time since research­ ers from Jülich, Germany and Europe were already impatient to feed the brand new champion computer with their scien­ tific data. The great day finally dawned in February 2008. Jürgen Rüttgers, Prime Minister of the Federal State of North Rhine-Westphalia, and Thomas Rachel, Par­liamentary State Secretary of the Federal Research Ministry, officially pressed the button launching JUGENE for its users. :: Frank Frick

23

Interview with Prof. Thomas Lippert

The future of supercomputing at Jülich The director at the Institute for Advanced Simulation (IAS) and head of the Jülich Supercomputing Centre (JSC) explains what the supercomputers of the next generation will be like and what role Forschungszentrum Jülich will play in the year 2012.

few, very large, data-intensive and highly ­scalable simulations, computers like ­JUGENE, which we can use to deal with the grand challenges facing society for instance in the fields of energy and the environment or health. And then, on the other hand, there are the flexible ­systems for somewhat smaller simu­ lations, which nevertheless make great demands on the working memory and the communication network – in future these will be cluster computers made of standard components. We need both computer architectures in order to be able to satisfy, in a cost-optimized man­ ner, the requirements of all the scientists who want to use our computing facilities in Jülich.

Question: Prof. Lippert, yesterday at a meeting discussing the Jülich super­ computer JUGENE, which has only been in operation for a few months, one of your staff had to leave early. He excused himself by saying: “I’ve got another meeting. It’s about a new supercom­ puter.” Can you tell us anything else?

24

Lippert: Actually, we are already involved in the planning for two supercomputers.

We are pursuing a dual concept in Jülich.

This is based on the realization that ­supercomputers can be divided into two classes with respect to the range of ­applications of JSC users. On the one hand, there are the computers for a

Question: But doesn’t every researcher want to process his problems on the most powerful supercomputer avail­ able? What do we actually need the cluster computer system for?

Lippert: In order to reserve the high-end supercomputer for a relatively small number of extremely challenging and in­ teresting research projects – about 30 per year – requiring enormous amounts of computing time. In contrast, on the cluster computer system, just to give you

Research in Jülich 2 | 2008

HIGHLIGHTS

Thomas Lippert: for him, Jülich is the leading name in Europe as far as computational science and engineering is concerned.

a figure, computing time is available each year for about 200 projects. You should also be aware that by no means every simulation is highly scalable, that is to say it runs faster the more processors are available. Anyway, the dual concept has proved to be extraordinarily success­ ful. In addition to JUGENE, JUMP means that we also have a computer available for, so to speak, our bread-and-butter work. Which is not to say that this work is not very important, as shown by the large number of publications in high-impact journals.

Question: What do you mean by the term “cluster computer system” and what are your specific plans for it?

Lippert: A cluster computer system is ­basically composed of computers that are equipped with a complete operating system and consist of powerful standard components – connected via a very fast communication network that is linked to the computing nodes by standard inter­ faces. Not every computer will have its own cabinet. We have been exploring such systems since 2004 and we are convinced that they offer a superior cost/performance ratio. We now intend to construct such a system on the basis

2 | 2008 Research in Jülich

of quad-core processors from Intel that will be known as JuRoPA (Jülich Research on Petaflop Architectures). We have a number of European companies on board including ParTec and Bull, as well as SUN and Intel from the USA. Question: At the beginning of the inter­ view, you mentioned that you are cur­ rently involved in the planning phase for a second supercomputer. Is this a suc­ cessor to JUGENE? Lippert: We are actually planning to ­extend JUGENE from at present 16 to at least 72 racks and thus to increase its performance to the one petaflop range, that is to say one trillion arithmetic operations per second. In parallel, we naturally want to keep our users with us on the way to more computing power.

After all, at the end of the day you must be able to use the system efficiently.

Question: Does that mean that super­ computing is not just about computing power and computing capacity but also about the know-how of the operators and users?

Lippert: The on-site expertise is what ­really counts. When I train young people today on supercomputers then they will

play a part as future experts in shaping progress in this field for the next 35 years – supercomputers, on the other hand, are replaced every four or five years. And due to the parallelization of the computer world even more expertise will be ­required in future – this goes as far as to consider how scientific theories should be formulated so that they can be handled by computers.

Question: And finally, what is your ­vision? What part will Forschungs­ zentrum Jülich play in supercomputing in 2012?

Lippert: Jülich will be the leading name in computational science and engineering in Europe and will operate the super­ computing centre as an element in the new European supercomputer infra­ structure. Our institution will bear com­ parison with the centres in the USA. A computer with a performance of about 10 petaflop/s will be installed here which leading European simulation teams will clamour to use for their calculations. We are cooperating closely with industry to develop the computer systems of the ­future. :: Interview: Frank Frick

25

An inside view Many people find a good illustration helps them to understand difficult problems better.

Scientists are no exception. But they get even more insights from computer-generated, three-dimensional worlds. Experts from the Jülich-Aachen Research ­Alliance (JARA) are therefore developing a network of visualization stations so that ­researchers at different locations can immerse themselves in the same virtual ­environment.

A

lthough he is right in the middle of a burning train, Dr. Armin Seyfried keeps his cool. This doesn’t mean he is particularly stoical or death-defying.

The train isn’t real and the scientist from the Jülich Supercomputing Centre (JSC) is in the middle of a 3-D simulation which is projected onto a three-by-eleven-metre screen. But the 3-D glasses mean that the image doesn’t stay on the wall. You really believe that you are on the train.

26

“Even today it is still not clear how much heat is released during a fire on a train,” explains Seyfried. An attempt was made in Sweden to settle the issue using an experiment in which an intercity ex­ press was set on fire in a tunnel. “That was very expensive. And both the train and the tunnel were a complete write-off afterwards. The measurements also in­ volved very great uncertainties,” says Seyfried.

With the aid of simulations developed by him and his colleagues, trains will in future be designed in such a way that the fire won’t be able to spread quickly.

F­ urthermore, it should be possible to reach the emergency exits as quickly as possible irrespective of where the fire breaks out. The sophisticated simula­ tions, which can only be implemented ­using supercomputers, enable the train to be set on fire again and again in differ­ ent places, so to speak. The 3-D visua­ lization means that the scientists can ­observe the way the fire spreads close up and from different perspectives.

Meanwhile, conditions are much colder and windier for Prof. Torsten ­Kuhlen from the Virtual Reality Centre at RWTH Aachen University. He has just

Research in Jülich 2 | 2008

HIGHLIGHTS

Left: Three scientists in front of the three-by-eleven-metre screen at Jülich.

Their 3-D glasses give them the optical impression that they are in the middle of a burning railway compartment. Top: Simulated wind flows between the skyscrapers of New Orleans.

loaded a visualization of the city of New Orleans in the Aachen “cave”, which his team developed together with the Univer­ sity of Louisiana. The “cave” is a small room where images can be projected onto the walls and floor. “As yet, you can‘t ­actually touch the objects in our cave – unlike the holodeck in the Star Trek uni­ verse. But the visual impression is already very good and is getting better with each computer generation,” says Kuhlen.

Instead of watching the simulation on a computer monitor, Kuhlen puts on what looks like sunglasses with several white balls attached to the frame and enters the cave armed with a sort of toy gun ­also covered in white balls. Like a giant he now strides through New Orleans, skirts skyscrapers and fires the gun.

THIS IS SCIENCE NOT A GAME Obviously, this simulation is not a computer game but has a serious scientific background. The “gun” replaces the computer mouse and enables the “caveman” to perform certain actions.

In the present case, Kuhlen uses the gun

2 | 2008 Research in Jülich

to release particles which move with the simulated flow of wind in New ­Or­leans.

The white balls on the gun enable a so-called tracking system consisting of two cameras to determine the exact posi­ tion of the gun in three-dimensional space. Kuhlen can, for example, release the suspended particles with the utmost precision in front of a skyscraper or in a street between the high-rise buildings in order to observe flow conditions in New Orleans. The balls on his 3-D glasses keep the system informed about Kuhlen’s position. The images projected onto the walls and floor are then continuously ­adjusted to the user’s perspective, which changes with his every move.

“An animated film could not hope to reproduce the opportunities provided by such virtual reality,” explains Kuhlen. “In an animation, a decision has to be made in advance about the user’s position. For example, he wouldn’t be able to see what’s happening behind a high-rise building from his perspective whereas in the cave he just goes around the build­

ing.” Furthermore, in a film you have to decide in advance from where in the town the suspended particles will be released.

If they are released everywhere at the same time then you won’t be able to see anything.

It is obvious that the possibilities of­ fered by such virtual reality depend on the computer capacity available. “This is precisely the reason why our colleagues from Aachen contacted us in the first place. Our supercomputer had the neces­ sary computing power,” says Dr. Herwig Zilken from the Jülich Supercomputing Centre.

ENTERING THE 3-D WORLD TOGETHER In the meantime, collaboration be­ tween RWTH Aachen University and For­ schungszentrum Jülich within the frame­ work of the JARA-SIM cooperation (see box on p. 28) has expanded far beyond the provision of computing capacity at Jülich. Zilken’s and Kuhlen’s groups are working jointly on developing what is known as ivNet (pronounced “ivy net”).

This abbreviation stands for “immersive

27

JARA-SIM

A scientist uses a sort of toy gun to release a stream of suspended particles.

A tracking system with two cameras determines the exact position of the gun in three-dimensional space.

visualization network” and refers to visu­ alization stations connected via a net­ work in which users can immerse them­ selves – as in the cave in Aachen.

“In contrast to the Aachen cave and our big 3-D demonstration screen here in Jülich, this system is transportable and can be easily taken to any research insti­ tute and installed there,” explains Zilken.

The system has all of the functions of the Aachen cave including the 3-D glasses, tracking system and computer mouse.

Only the projection area is reduced to a 120-by-90-centimetre screen in order to make it easily transportable.

Individual system stations are already operational in Aachen and Jülich. Kuhlen and Zilken are working on the network and the necessary software. The first goal is to save scientists from Aachen and Jülich who are working on a joint project from having to commute between

28

their institutions every time they want to discuss a particular problem. ivNet allows the two scientists to remain in their own institutes and meet each other in virtual reality. The crucial point is that they are both in exactly the same scenario. If one of them makes a change using the inter­ active mouse then the other can expe­ rience this directly. The ultimate goal is to make what is initially planned for the relatively short stretch between Jülich and Aachen possible across any distance in the whole world.

The ivNet project could therefore spearhead the next revolution in the field of communications and global network­ ing – in a similar manner to another ­research network that first came into ­being 15 years ago in research circles – the Internet! ::

In 2007, Forschungszentrum Jülich and RWTH Aachen University combined their cooperative projects, which had been running for many years, under the umbrella of the Jülich-Aachen Research Alliance (JARA). Not least due to this ­collaboration, RWTH was chosen as an elite university by the Federal Research Ministry’s excellence initiative. The JARASIM section of this alliance develops computer simulations as the driving force behind scientific progress and combines the expertise available at the Jülich Insti­ tute for Advanced Simulation (IAS) and RWTH. “Both research and teaching profit from JARA-SIM. Without this institutional interlinkage, the German Research School for Simulation Sciences (GRS), which is intended to train the future scientific elite in computer simulations, would be unthinkable,” says Thomas Lippert, ­director at IAS.

Axel Tillemans

Research in Jülich 2 | 2008

HIGHLIGHTS

Fast computers for better blood pumps Heart disease is the number one killer in industrialized countries.

A small implantable pump provides support for a weak heart.

Researchers make use of Jülich supercomputers and simulation know-how to optimize the flow inside this pump.

I

f you need a heart transplant then you haven‘t got much time to spare. How­ ever, it can take some time before a suitable donor is found. In such cases doctors often use a blood pump to sup­ port a weak heart. The so-called DeBakey pump manufactured by an American company is a titanium device about the size of your finger which has been im­ planted into about 440 patients so far. In some heart patients, this relief enables the cardiac muscle to recuperate. How­ ever, above all the pump helps patients with a severely damaged heart to survive until a heart is available for transplanta­ tion. Prof. Marek Behr from RWTH Aachen University is further developing these lifesaving pumps with the aid of the super­

2 | 2008 Research in Jülich

computer. Within the framework of the Jülich-Aachen Research Alliance (JARA), he and Prof. Felix Wolf from the Jülich Supercomputing Centre are optimizing the necessary simulation programs in ­order to improve the pump.

CONFLICTING DEMANDS As Goethe put it in his Faust “blood is a very special kind of juice”, which means that an artificial heart must fulfil different criteria than a water pump. The aqueous blood plasma contains various cells as well as blood platelets. About half the volume of blood consists of red blood cells which transport oxygen. This means that the pump has to fulfil very different requirements. “The pump has to be very

Marek Behr from the Jülich-Aachen ­Research Alliance uses computer simulations to develop life-saving blood pumps.

29

One result of the simulation: the regions of the blood pump shown in red experience the highest pressure during operation, and those in blue the lowest.

small so that it can be implanted into the human body,” explains Behr. “How­ ever, in order to provide sufficient sup­ port for the heart it must be able to pump several litres of blood in a minute. Strong shearing forces are the result when so much liquid flows through a small pump in such a short time.” There is thus a danger that the impeller – a component inside the pump – may squash the deli­ cate cells and blood platelets because it rotates 10,000 times a minute.

This must be avoided at all costs. If too many red blood cells are damaged then large quantities of the haemoglobin they contain will leak out. This substance

30

then has a toxic effect on the patient and the damaged blood cells can no longer transport any oxygen. Nor should the blood platelets be damaged or activated if at all possible otherwise blood clots may be easily formed which then block the blood ves­ sels resulting in a life-threatening throm­ bosis. However, trial and error cannot be used on patients in order to improve the pump in such a way that it does less dam­ age to the blood it transports. This is where calculations of flow conditions on the computer can come into play. “This is a typical case where simulations can be used. They help when experiments are

impossible, too resource-consuming or too dangerous,” explains Prof. Felix Wolf from the Jülich Supercomputing Centre.

“In the first phase up to 2001, we ­developed the simulation to such an ­extent that we can be sure that it corre­ sponds to the results of real experi­ ments.” This is how Behr describes the early stages of the project. In the second phase of development, they succeeded in revealing where the blood was damaged most while flowing through the pump.

“This occurred to a greater extent than previously assumed in the rear section of the pump, in what is known as the dif­ fuser,” Behr explains. These findings have already influenced the design of a new version of the DeBakey pump. Behr con­ tinues: “Since 2006 in the third stage of the project, we have been developing mathematical methods and programs in order to optimize the pump’s mode of ­operation.”

Blood also makes special demands on the simulation. Whereas the flow be­ haviour of water can be calculated on a normal PC, a high-performance super­ computer such as Jülich’s JUGENE is ­required to investigate the liquid flow in such a complicated pump and to study the dynamics of the mixture of liquid and

Research in Jülich 2 | 2008

HIGHLIGHTS

Felix Wolf between the racks of the Jülich supercomputer. Together with his team, he has made the calculations for the blood pump simulations faster and more effective.

particles. For example, it is very timeconsuming to simulate how the blood pump is started up. “This start-up phase of about ten revolutions until the pump is rotating smoothly only lasts a fraction of a second. But in order to simulate this phase we need about ten seconds of computing time on the supercomputer,” Behr explains. Behr turned to Felix Wolf for advice on how to speed up the calcu­ lations on the supercomputer.

In order to distribute the calculations among a large number of processors on the supercomputer the pump is, so to speak, cut up into small pieces. One processor is then responsible for each of these segments. However, the large number of individual calculations have to be put back together again. This is where one of the difficulties lies. “What happens at the edges of the segments is often ­interesting,” says Felix Wolf. The exchange of data between the processors is often quite decisive here. “If problems arise then it is no good distributing the task between even more processors. The processors then simply exchange more and more messages and sometimes many of them are simply superfluous, as if they were sending empty envelopes.”

TRACKING DOWN WEAK POINTS This is why the simulation of the flows in the DeBakey blood pump was initially no faster or more accurate when more than 900 processors were involved. Wolf used special software to get to the bot­ tom of this problem. He incorporated small sensors into the simulation pro­ gram. That is to say, little pieces of pro­ gram that record all the details of each mathematical operation. Using the pro­ gram package developed at Jülich and

2 | 2008 Research in Jülich

known as Scalasca (Scalable Perform­ ance Analysis of Large Scale Applica­ tions), he analysed these data and thus tracked down the weak points in the sim­ ulation program. On the basis of this analysis, it became possible to stem the flood of “empty envelopes”. “The simula­ tion now runs efficiently on more than 4,000 processors and is roughly five times as fast as it was before,” Wolf is pleased to say. In this way, new versions of the pump can be examined much fast­ er in a virtual test run, which means that patients also benefit more quickly. For example, a scaled-down version of the pump has been developed for children aged from five to sixteen years suffering from heart disease. “Such successes are only possible because engineers and computer scientists work hand in hand in JARA,” says Wolf.

He is hoping that the European Part­ nership for Advanced Computing in ­Europe (PRACE) initiative will lead to fur­ ther progress. Individual supercomputers are already available in Europe that are well up in the international ranking. “But we are still not in as good a position as the USA,” says Wolf. However, European plans for petaflop class computers pro­ viding 1,000 trillion floating point ope­ rations per second are already well ­advanced. “This will further improve con­ ditions for researchers in Europe who want to perform complex simulations,” Wolf confidently adds. A long-term goal may also be coming a bit closer: blood pumps which do not just keep the patient alive until a suitable donor is found but which can actually provide a permanent replacement for the human heart. :: Wiebke Rögener

31

The doors of perception The design of the doors and thus the “face” of the supercomputers at Jülich has changed over the course of the years. But the computer’s brain behind the doors has changed even more. Its computing power has increased enormously – especially with the present generation of devices. At the same time, the simulations performed on these computers have become more precise, detailed and realistic: demonstrated here by the example of the computational grid for simulating the climate over Europe showing how the meshes have become finer and finer.

10 Gigaflop/s

614 Gigaflop/s

2.6 Gigaflop/s

32

Research in Jülich 2 | 2008

HIGHLIGHTS

167,000 Gigaflop/s

2 | 2008 Research in Jülich

33

News

from Supercomputing

Highly charged computers

Powerful partners

In nature, electrical charges play a decisive part in protein fold­ ing, catalytic processes on surfaces and many other phenome­ na. Jülich algorithms tailored to supercomputers help to simu­ late complex systems of charge carriers in bigger dimensions.

Thanks to funding to the tune of € 1.5 million for the ScaFaCoS project promised by the Federal Research Ministry, scientists from Jülich are now cooperating with German partners in further developing efficient programs for physical and biological issues and in making them available to the research community.

At the first industry seminar held by the EU PRACE project in September 2008, the foundation for further cooperation was laid by top executives from leading companies in the automo­ bile, chemical and electronics sectors, scientists from 13 Euro­ pean countries, and representatives of the European Commis­ sion. “The seminar showed that very positive synergies can be achieved between research and industry in supercomputing and in simulation,” said Jean François Hamelin, head of IT at the French energy utility EDF, after the seminar. In the PRACE project (Partnership for Advanced Computing in Europe), coordi­ nated by Forschungszentrum Jülich, 16 partners are preparing for the establishment of infrastructure which will provide Euro­ pean researchers with access to supercomputer resources of the very highest level.

Software for distributed computing Experts from the Jülich Supercomputing Centre have made a thorough revision of the European grid software UNICORE and have now published Version 6. Grid computing is a form of distributed computing which enables scientists and engi­ neers to solve difficult problems requiring extensive com­ puting capacity. UNICORE provides users all over the world with safe and intuitive access to computer resources and data, also at supercomputing centres. The software is now also being used by industrial companies and public institu­ tions. www.unicore.eu

34

Research in Jülich 2 | 2008

HIGHLIGHTS

Decisive engagement

Back to school for the scientific elite

In November 2008, the Federal Government and the federal states of Baden-Württemberg, Bavaria and North Rhine-West­ phalia agreed to provide € 400 million for the step-by-step ex­ pansion of the Gauss Centre for Supercomputing (GCS) at its sites in Jülich, Garching and Stuttgart. “We are extremely grate­ ful for this support since it provides us with the unique opportu­ nity of playing a decisive role in shaping the new European infra­ structure with petaflop computers,” says Prof. Achim Bachem from Forschungszentrum Jülich on behalf of the board of direc­ tors of GCS. www.gauss-centre.eu

The German Research School for Simulation Sciences (GRS), founded by Forschungszentrum Jülich and RWTH Aachen Uni­ versity, has opened its doors. Since winter semester 2008/2009, hand-picked master’s students are being trained here in the ap­ plications and methods of computer simulation for science and engineering. In April 2008, GRS had already welcomed the first physicists and engineers working on their PhDs in Jülich and Aachen. www.grs-sim.de

Simulations save lives In future, computer models will be able to rapidly predict how people attempt to escape from the danger of a fire or terrorist attack in football stadiums, railway stations or public buildings. Developed by Jülich scientists, simula­ tions will form the backbone of a computer-assisted evacuation system. It will help to guide people to the best escape routes and to optimally deploy safety personnel and the emergency services. The Federal Research Ministry is pro­ viding about € 4.6 million to fund the Hermes project coordinated by the Jülich Supercomputing Centre, and also involving industrial partners, as part of its programme on “Research for Civil Security”.

PUBLICATION DETAILS Research in Jülich Magazine of Forschungszentrum Jülich, ISSN 1435-8514 Published by: Forschungszentrum Jülich GmbH | 52425 Jülich Conception and editorial work: Dr. Frank Frick, Dr. Angela Lindner (responsible according to German press law), Kosta Schinarakis Authors: Dr. Frank Frick, Dr. Wiebke Rögener, Dr. Axel Tillemanns Design and layout: Graphic Shop, Forschungs­ zentrum Jülich Photo credits: Forschungs­zentrum Jülich (cover illustration, pp. 8 – 9, p. 11, p. 14, p. 15 all except Fig. 6, p. 17 left, p. 21, p. 26, p. 28 bottom, pp. 32 – 33, p. 34 top left), R. U. Limbach (p. 3, p. 4 top, bottom right, p. 6, p. 7 top left, bottom centre, bottom right, p. 12, p. 17 top, p. 19 bottom, pp. 22 – 25, p. 31 right, p. 34 top right, p. 35 top right), K. Peters (p. 10 bottom), ­JARA-SIM, RWTH Aachen University (p. 2, p. 27, p. 28 top, p. 29 bottom, p. 30 top), creativ collection (p. 10 top, p. 20 top, p. 30 bottom, p. 35 bottom), MARUM, ­Uni­ver­sity of Bremen (p. 18 top), micromedcv.com (p. 29 top), University of Hamburg (p. 4 bottom left, p. 16), E. Scheiner; N.N. Nair; D. Marx; Ruhr University of Bochum (p. 19 top), C. Czaplewski; A. Liwo; University of Gdansk (p. 13), Getty Images (p. 7 bottom left), Thomas Klink (p. 7 top right), J. Schumacher (p.15 bottom left), U. Schulzweida & N.P. Noreiks; Max Planck Institute for Meteorology (map of Europe pp. 32 – 33) ­Translation: Language Services, Forschungszentrum Jülich Contact: Corporate Communications | telephone +49 2461 61 - 4661 | fax +49 2461 61 -4666 | info@fz-juelich.de Printed by: Druckerei ­Schloemer GmbH Print run: 5,000

2 | 2008 Research in Jülich

35

Member of: