The newsletter of EPCC, the supercomputing centre at the University of Edinburgh
news Issue 80 Winter 2016
In this issue ARCHER gains Knights Landing capability Forthcoming training courses Centre of Excellence in Computational Molecular Biology
Cirrus
Our new computational service for industry New Fortissimo Marketplace Image: Callum Bennetts, Maverick Photo Agency
From the Director Welcome to this edition of EPCC News. The cover of this issue shows our new SGI ICE XA system which we installed and commissioned in the early summer. The system was a joint purchase between EPCC and Edinburgh Genomics, who own over 3,000 of its cores and use it to construct whole human genomes from the data produced by the University’s genome sequencing machines. EPCC owns just over 2,000 cores of the system which it runs on behalf of both organisations. Cirrus, as we’ve named it, is the second of our dedicated HPC systems for industry. Our first, INDY, is now four years old and we are in the midst of transitioning our industry users from it to the new system. We’re sure they will find the new system much faster – in particular with regard to storage. We’ve gone to some trouble to make the transition as simple as possible with familiar compilers and job scheduling software available on the new system.
EPCC has always worked with industry and many companies have made use of the National HPC services we have operated over the past two decades including HPCx, HECToR and ARCHER. Owning our own, albeit much smaller, system gives us more flexibility to meet industry needs – there is much more memory per node and we can easily accommodate long duration jobs with small core counts. We’ll be offering time on Cirrus through the Fortissimo Marketplace which we’ve created as part of the Fortissimo Projects. You can read more about this and a number of our other European projects in this issue. EPCC is proud to be a European HPC centre and is arguing strongly for UK universities and companies to continue to be able to access Horizon 2020 and subsequent programmes. More on this in future issues no doubt! Mark Parsons m.parsons@epcc.ed.ac.uk
Want to know more about our work? Since early 2013, the EPCC blog has been reflecting the wide range of our activities, including public outreach, training, research and our work for industry. Posts are mostly written by EPCC staff, but we welcome guest contributions on any area relevant to our work. If you would like to contribute, please email Tracy Peet at: t.peet@epcc.ed.ac.uk Or you can tweet us: @EPCCed
Contents
3
Bringing HPC to business Fortissimo Marketplace launches
4
EPCC expands commercial computing service Introducing Cirrus
6
Case study: the benefits of HPC for engineering Koenigsegg builds better cars
8
MSc work experience Industrial placement report
10
Working with the HPC industry Our collaborations with Cray, Intel and SGI
12
National HPC service enhanced ARCHER gains Knights Landing capability
14
Centre of Excellence in Computational Molecular Biology BioExcel launches
15
Preparing for exascale NextGenIO’s implementation phase begins
16
Energy efficiency in HPC Adept project’s measurement and benchmarking suite
18
Annual MPI users and developers conference EuroMPI review
19
Diversity in HPC Women In HPC update
20
Promoting research software Software Sustainability Institute news
22
Matchmaking industry and academia AIMday Construction report
23
Training with EPCC Forthcoming courses
Contact us www.epcc.ed.ac.uk
info@epcc.ed.ac.uk
+44 (0)131 650 5030
@EPCCed
EPCC is a supercomputing centre based at The University of Edinburgh, which is a charitable body registered in Scotland with registration number SC005336.
2
HPC cloud-based simulation of light-aircraft aerodynamics by Pipistrel (Slovenia) Image: Pipistrel (Slovenia)
Connecting business to HPC and cloud resources Fortissimo is a collaborative EC-funded project that enables European SMEs to be more competitive globally through the use of simulation services running on a high performance computing (HPC) cloud infrastructure. The project is coordinated by EPCC and involves over 100 partners including manufacturing companies, application developers, domain experts, IT solution providers and HPC cloud service providers from 14 countries. These partners are involved in experiments that aim to solve real business problems driven by end-users by simulating industrial processes using cloudbased HPC.
Fortissimo Marketplace Activity within the Fortissimo projects led to the official launch of the Fortissimo Marketplace Ltd in October this year. EPCC led the development of the Marketplace and Prof. Mark Parsons is the CEO of the company. The Marketplace provides small to medium sized enterprises (SMEs) with costeffective access to advanced simulation and modelling services operating on a cloud infrastructure of HPC resources together with software applications, expertise, and tools. The Marketplace offers a one-stopshop of resources (software, hardware and support), primarily aimed at manufacturing SMEs but also other businesses that could
benefit from cloud-based HPC, allowing them to benefit from advanced simulation and modelling service on an HPC cloud, improving company competitiveness through the use of HPC to deliver new or improved products or services.
Carolyn Brock, EPCC c.brock@epcc.ed.ac.uk
The Marketplace creates value and opportunities for all companies in the value chain, from independent software vendors (ISV) looking for a reliable partner to implement their software-as-a-service strategy, to HPC cloud service providers wishing to provide expertise. The Marketplace attracts users and members: • Users are companies or individuals who will procure services via the Marketplace. • Members are companies that advertise, promote, offer and sell services via the Marketplace. In addition to leading the Marketplace, EPCC is also a member. The ambition of the Marketplace is to become the portal of choice for HPC and high performance data analytics expertise and service provision, delivered by Europe’s major HPC technology providers and domain experts.
The newsletter of EPCC, the supercomputing centre at the University of Edinburgh
The Fortissimo Marketplace is part of the I4MS Initiative and is funded by the European Commission within the 7th Framework Programme. Following the success of the original Fortissimo project, Fortissimo 2 was awarded, starting in November 2015. This second project is funded for a period of three years and also led by EPCC. Fortissimo Marketplace www.fortissimo-project.eu 3
Supercomputing capability straight to your desktop Introducing Cirrus: our latest computational service for industry. Our new supercomputer Cirrus can dramatically cut the time taken to bring new products and services to market. The system’s applications include helping to create efficient engineering devices, solving complex genetic calculations, and optimising the design of buildings. The high performance computing (HPC) facility can also help businesses from a range of science and engineering sectors cut the cost of product and service development. Clients will be able to use the £1 million machine to tackle their research and design challenges. The service is fully supported and clients will also have access to our consulting expertise in HPC and data analytics.
Cirrus is an SGI ICE XA supercomputer comprising more than 5,000 processor cores. The system was a joint purchase between EPCC and Edinburgh Genomics, with EPCC owning over 2,000 cores of the system which it runs on behalf of both 4
organisations. Cirrus facilitates calculations that would be impossible, or much slower, when carried out on conventional desktop computing systems, delivering results in hours to days instead of weeks or months. The fifth-generation system powers many of the largest supercomputing environments in the world. Cirrus is housed at EPCC’s state-ofthe-art Advanced Computing Facility at Easter Bush, which also hosts ARCHER, the UK national supercomputing service. George Graham, Commercial Manager of EPCC, said: “This newly installed computing power – in tandem with EPCC’s in-house expertise – means we are well placed to help businesses meet many of the computational challenges associated with developing new products and services”.
Cirrus services However Cirrus is more than just a computational platform. Users will benefit from EPCC’s proven
Cirrus is a state-of-the art SGI ICE XA cluster utilising superfast Infiniband interconnect. A total of 56 compute nodes are available for general industry use. There are 36 cores per node, providing 2016 cores in total. Hyperthreading is enabled on each node, providing a total of 72 threads per node. Each node has 256GB RAM. Three login nodes, with similar hardware to the compute nodes, are provided for general use. Local Lustre storage is provided by a single Lustre filesystem with 110TB of disk space and users will have access to EPCC’s considerable data storage and archiving services.
Image: Callum Bennetts, Maverick Photo Agency
expertise in supporting advanced HPC infrastructure. A fully provisioned, two tier, support service is provided for our users, offering assistance on a range of workflow issues such as how to access the service, how to compile and run jobs, and how to manage service resources and to generate reports on resource usage. And users will not just have access to compute cycles. By partnering with key software providers we will be able to provide our customers with easy, pay-per-use access to a range of simulation and modelling services such as the FLACS fire and explosion modelling software from GexCon and the Hyperworks CFD software from Altair.
Accelerator: supercomputing on demand Cirrus is just one part of our Accelerator service, which delivers high-performance computing capability at a fraction of the cost of buying and operating in-house HPC services.
Accelerator can be used as a: •T ransformative HPC resource accelerating development and discovery lifecycles • F lexible HPC resource smoothing out demand peaks •C ontingency over internal HPC infrastructure failure There are several routes to Accelerator adoption: •D irect access: For users conversant with using HPC platforms, direct and immediate access on a pay-per-use basis is provided •S elf-assessed free trial: We can set you up with a free trial on any of our service machines so that you can undertake your own test drive, piloting and benchmarking activities •A ssisted Adopter Programme: We will work with you to facilitate the seamless mapping of your inhouse workflows to EPCC’s HPC platforms, complete with user specific documentation and training.
The newsletter of EPCC, the supercomputing centre at the University of Edinburgh
Find out more To apply for time on Cirrus or to discuss our Accelerator service, contact George Graham at EPCC: g.graham@epcc.ed.ac.uk +44 (0) 131 651 3460 +44 (0) 777 370 8191 5
On-demand computing gives an SME a competitive edge We are committed to bringing the benefits of highperformance computing (HPC) to all companies. The example here demonstrates the direct value of HPC to Swedish car designer and manufacturer Koenigsegg. Swedish SME Koenigsegg is a leading designer and manufacturer of high-performance sports cars. Its cars must meet the exceptional standards of performance and quality that this prestige market demands.
The challenge Improving the aerodynamic performance of a hypercar is essential to Koenigsegg’s product development. Understanding the airflow patterns around the complex structure means either using extensive wind-tunnel tests or advanced computer simulation, both of which would normally be prohibitively expensive. One of the main costs of advanced computer simulation lies in owning a computer capable of doing the complex calculations involved. Typically, the cost of buying and operating such a computer cannot be justified by a small company, particularly when it is not in use all the time. 6
How we helped By using a cloud-computing model, the full aerodynamic design of a hypercar can be almost entirely conducted cost-effectively using computational fluid dynamics (CFD) with minimal road and wind-tunnel testing. Instead of having to buy and run an expensive high-performance computer, Koenigsegg buys the simulation as a service, and only pays for what it uses. This dramatically reduces the cost and allows Koenigsegg to focus on its core expertise as a hypercar designer. Building on earlier work with a supercomputing centre in Italy, Koenigsegg conducted extensive CFD modeling using OpenFOAM software running on EPCC hardware. Because it only pays for what it uses, Koenigsegg was able to access the cloud-based HPC service for a fraction of the cost of running an equivalent system in-house.
Mark Sawyer, EPCC m.sawyer@epcc.ed.ac.uk
The use of ICON’s simulation software on the Fortissimo cloudbased HPC system has enabled Koenigsegg to reduce the costs of designing its hypercars, and to bring new products to the market faster. The cloud computing model means it only pays for what it uses, with hardware expenses and maintenance costs being removed.
Image: Koenigsegg
What Koenigsegg says “The use of ICON simulation software on a cloud-based HPC system has enabled Koenigsegg to avoid expensive wind tunnel testing. Accessing powerful computing resources remotely also reduces hardware expenses and maintenance costs. Because of the accessibility, it is possible to monitor the solution at any given moment and make decisions accordingly, do reduce design period. “Before the start of this case study, Koenigsegg had only limited computer resources available in-house and little experience in HPC-based CFD. “In this case study, 100% of the aerodynamic development of the Koenigsegg One:1 has been conducted using HPC-based CFD simulations. In less than eight months, hundreds of simulations to test various configurations have been carried out and optimised for various driving setups.
“The results were an impressive 250% increase in downforce with only a 15% increase in drag at 250km/h. That’s an improvement that would have been unachievable without the use of HPC-based CFD. “Tests have shown that the use of HPC-based simulation supported by external software led to a return on investment in less than three months for the production of a new car configuration. Significant costs can be saved and transferred to other critical parts of the development and production process.”
Fortissimo Marketplace EPCC and software providers ICON are making the simulation service available through the Fortissimo Marketplace – a one-stop-shop where end-users can access modelling and simulation services across a wide range of application areas. You can learn more about the Fortissimo Marketplace in the article on page 3.
The newsletter of EPCC, the supercomputing centre at the University of Edinburgh
Fortissimo Marketplace www.fortissimo-project.eu
Accelerator service To discuss our Accelerator service of on-demand HPC for business, contact George Graham at EPCC: g.graham@epcc.ed.ac.uk +44 (0) 131 651 3460 +44 (0) 777 370 8191
7
MSc in HPC industrial placements All our MSc students are given the opportunity to work on a project with a company or academic group. Among this year’s students were Adarsh Janakiraman and Killian Tattan, who both worked at Mallzee, an Edinburgh-based SME that produces the UK’s top non-retailer shopping app. The articles below describe both sides of the experience.
Adarsh Janakiraman The opportunity to work on my dissertation project with one Edinburgh’s most exciting e-commerce startups, Mallzee, was one I did not want to pass. Mallzee is a fashion app for the smartphone, which has cornered the youth market in the UK. With over 500,000 registered users, it has been able to collect user preference information on over a million different products. The company is currently in the process of making this potential mine of information available to retailers, to better understand their customers’ behaviour. The goal of my project was to build a model to predict the sales of products showcased on the app. The project involved extracting the data, cleaning it, selecting the right features, creating more features not currently in the database and finally applying a machine learning model and evaluating its fit on the data. Overall, the experience of working on an actual industrial problem was great as it not only allowed us to build on our hard skills in model building and evaluation but also allowed us to build on the softer skills, such as business model 8
evaluation, commercial strategy and teamwork, required for working in a fast-paced startup environment. I would like to thank the department, and all involved, for making such an exciting opportunity available to students. I would also like to thank the Mallzee team for accommodating me in their office with a desk and a piping mug of tea.
Killian Tattan The industry dissertation was both intellectually and logistically challenging. The project involved improving the recommender engine currently in development by Mallzee. A recommender engine (recommender) makes suggestions to the user based on their previous preference/buying history. The recommender being developed for Mallzee would present clothing items to users based on what they liked and disliked in the past. The concept is simple, but the science, algorithms, and mathematics behind it are quite complex. The final recommender developed in the project employed a random forest classifier, a machine learning algorithm (a type of algorithm that doesn’t have predefined rules and
“The benefits for us have been multiple and scattered over several levels: we gained new knowledge on our own data, which is very large and multi-faceted, and we also experimented with novel ideas with the purpose of advancing and enhancing our business via data-driven decisions.” Martina Pugliese Mallzee
Image: Mallzee
continuously learns from data), to classify (predict) whether a new clothing item would be liked by the user or not. Based on this prediction the algorithm would either recommend the item or not. The project was successful in that the accuracy of the recommender was slightly improved from the original model!
Martina Pugliese, Data Scientist, Mallzee
I feel industry projects are an excellent way for businesses to make use of the knowledge and talent available at universities, while also giving students insight into how the theory of what they’ve been learning is applied in the real world.
Killian worked in the area of user personalisation, contributing to the realisation of improvements to the current recommender system we designed, while Adarsh tackled the question of how users respond and interact with fashion, generating new knowledge about what clothing and accessories are more prone to sell in the market.
The relationship between industry and academia is mostly mutually beneficial, but not without its differences. What was interesting for me was to mediate the relationship between industry and academia while also achieving my projects goals. Overall the project was thoroughly enjoyable; an excellent way to implement the theory of what I had learned in class to real world application while also learning about the exciting new field machine learning.
Through the EPCC students’ summer placements, we at Mallzee have been able to investigate two of the main research problems we face which deal with data and about which we employ data science methodologies.
Both projects presented challenges not only at the conceptual but also at the technical level, as data had to be polished and prepared for analysis and bespoke code created. The students utilised a mixture of statistical and machine learning techniques, and were both able to conclude the work in the time allowed by presenting us with working prototypes. They mainly used the Python data stack and contributed code which is smoothly integrated into Mallzee’s codebase.
Adarsh (top) and Killian.
We are always looking for interesting collaborative projects for our students to undertake towards the end of their course (from April/ May to August). If you want to know more, please contact Maureen Simpson m.simpson@epcc.ed.ac.uk
Find out more about our MSc in High Performance Computing and MSc in HPC with Data Science: www.epcc.ed.ac.uk/msc 9
Collaboration: a bright idea EPCC works closely with industry to develop hardware and software for the next generation of HPC systems. Below EPCC staff members discuss three examples. SGI and EPCC’s European Joint Research Centre SGI, a global leader in highperformance solutions, has chosen to collaborate with us by establishing a European Joint Research Centre at EPCC. Together with SGI we will research a diverse set of scientific applications and systems software tools that enable production supercomputing at extreme scale. “Cutting edge research requires the ability to process large amounts of data at extreme scale. SGI HPC solutions and software expertise at EPCC will allow us to gain insights faster,” said Mark Parsons, Director of EPCC. “This partnership is an exciting step in European HPC research across many sectors, from healthcare to energy. We’re committed to making it a great success for both organisations.” The joint research centre will focus on HPC and High Performance Data Analytics work that prepares SGI and EPCC for the many exciting hardware and software innovations that are on the horizon as we prepare for the Exascale age. Over the next couple of years a series of hardware advances will signpost 10
how vendors intend to get to the Exascale. Although the hardware roadmap is becoming clearer there is a vast amount of software work required to utilise the very high levels of parallelism we expect to encounter. This is one of the most exciting periods in the history of HPC. If we get it right, HPC will be transformed well into the future and we intend that the collaboration enabled by the Joint European Research Centre will play a key part in this. Maureen Simpson, EPCC
Intel Parallel Computing Center and Xeon Phi processors EPCC and Intel have collaborated for over 2 years on the exploitation of Xeon Phi processors for computational simulation. We are very excited with current work, which has switched focus from first generation Xeon Phi’s – Knights Corner (KNC) – to the latest version, Knights Landing (KNL). We are currently working to understand the MPI performance across both the many-core processor and the networks it can now be directly attached to. Furthermore, we are investigating
“We are extremely excited and proud to collaborate with one of Europe’s leading research institutions to help redefine computational science at extreme scale. By partnering with EPCC we will be able to work with Europe’s top researchers across many disciplines, to help understand and solve the software challenges that will be occurring as we move to exascale computing.” Jorge Titinger, President and CEO of SGI
the performance of a range of different applications on KNL, and are starting to see real differences opening up between different types of application on KNL and on Broadwell (we have also been benchmarking applications on EPCC’s Cirrus system). This is interesting as it shows real choice for different types of applications in the types of hardware they use. Finally, we’re very interested in the new memory systems in KNL and are currently working on some tools to help application programmers efficiently exploit the more complex memory hierarchy on the KNL. Adrian Jackson, EPCC
Cray Centre of Excellence EPCC has had a long and successful collaboration with Cray. In 2009 Cray launched its Exascale Research Initiative in Europe, which resulted in researchers being based in EPCC and working closely with EPCC’s Exascale Technology Centre. This collaboration led to a number of joint projects and opportunities, most notably the EU-funded projects CRESTA [1] and EPiGRAM [2].
CRESTA was one of three flagship projects looking at software and tools for future exascale platforms. Led by EPCC, the project resulted in a number of novel methods and technologies being developed and improved. Within the project, Cray and EPCC worked closely on developing auto-tuning technology that can address the inherent complexity of programming the latest and future computer architectures. More recently the collaboration has continued through the UK’s national HPC service, which is operated by EPCC. The Cray Centre of Excellence has dedicated staff based at EPCC, whose role is to engage with users to allow them to maximise their use of Cray technologies. One particular project is currently utilizing the auto-tuning technology developed under CRESTA and applying it to specific application codes on ARCHER. Over the years, having Cray staff based within the centre has proved very successful, with a number of interesting and novel research projects. We look forward to continued collaboration opportunities in the future. Lorna Smith, EPCC
The newsletter of EPCC, the supercomputing centre at the University of Edinburgh
References 1) CRESTA: https://www.cresta-project.eu 2) EPiGRAM: www.epigram-project.eu/project
To find out more, contact Mark Parsons at EPCC: m.parsons@epcc.ed.ac.uk 11
ARCHER gains Knights Landing capability The ARCHER national service is being enhanced by the addition of a parallel Knights Landing (KNL) system that will be available to all ARCHER users. EPSRC and Cray have recently signed an agreement to add a Cray XC40 development system with Intel Xeon Phi processors to the ARCHER service. This will be a separate 12-node platform with KNL processors, but otherwise will have a very similar environment to the main ARCHER system, including Cray’s Aries interconnect, operating system and Cray tools. This should make it as straightforward as possible for ARCHER users to try out their codes on multiple KNLs. The KNL nodes arrived in the UK at the end of September and became available to users on 20 October. To encourage users to take advantage of this opportunity, for the first month there will no limits on maximum usage of the KNL service, although there will be limits on, for example, the number of jobs any user can submit at once, so that all users will have a fair chance to use the new system. After the first month, any ARCHER user can then get a default allocation. 12
Non-ARCHER users can also get this by the completion of an online KNL Driving Test. Additional allocations will require only a lightweight application process (probably similar to Instant Access).
User support Support for users of the ARCHER KNL system will be provided in a similar way to the main system: • Helpdesk • In-depth support • Cray Centre of Excellence • eCSE programme. There will also be various hands-on training courses and virtual tutorials See: www.archer.ac.uk/training/. The ARCHER KNL system provides a timely opportunity for users to investigate how their application codes will run on multiple KNLs, preparing for possible, future such systems.
About the system The KNL, Intel’s latest Xeon Phi many-core processor, can have up
Alan Simpson and Adrian Jackson, EPCC a.simpson@epcc.ed.ac.uk a.jackson@epcc.ed.ac.uk
to 72 physical cores, each of which can run 4 threads efficiently. The cores can undertake large vector operations, meaning they can perform calculations on 512bits of data at a time, giving the processor the potential of providing over 3 TFlop/s of double precision floating point arithmetic. This is three times the peak performance of the previous Xeon Phi processor, the Knights Corner (or KNC), meaning that porting codes to this latest many-core processor has the potential for significant performance improvements for applications.
size of data set Xeon Phi processors could operate upon. Having access to the full memory of a node puts them on the same footing as standard multi-core processor based nodes. Furthermore KNL processors also come with 16 GB of high bandwidth memory stacked directly on to the chip.
Direct access to main memory
This memory, also known as MCDRAM, provides much higher bandwidth data transfers than standard main memory, albeit with a slightly higher latency in the accesses. MCDRAM has the potential to significantly improve the performance of applications that stream data.
The KNL also comes with direct access to main memory, and the ability to be self-hosting, meaning that the Xeon Phi is no longer a co-processor, and systems no longer need normal processors to host the Xeon Phi.
The Cray XC40 ARCHER system uses the 7210 version of the Intel Xeon Phi processor, running at 1.30GHz, with 64 physical cores, 16 GB on chip MCDRAM, and access to 96 GB of DDR4 main memory running at 2133 MT/s.
The access to main memory also removes previous restrictions on the The newsletter of EPCC, the supercomputing centre at the University of Edinburgh
Find out more www.archer.ac.uk
Training www.archer.ac.uk/training
13
Powering biomolecular research through advanced modelling and simulation Image: Alexandre Bonvin
BioExcel is a new Centre of Excellence in Computational Molecular Biology. Adam Carter, who is responsible for Consultancy and User Groups, explains the project’s work. EPCC is participating in a 3-year project to establish the BioExcel Centre of Excellence. Although we’re still in the first year of the project, the centre is very much open for business, and a variety of activities are already underway. The centre will support both academic and industrial researchers in the use of high-performance and high-throughput computing. Project partners include the lead developers of several important software packages (including GROMACS and HADDOCK) and those who are collaborating to extend important functionality in other packages, such as the addition of a new QM/MM interface to CPMD. While work goes on in the background to refine business models and determine future governance models in order to build a sustainable centre, much research work and software engineering is already underway.
Pilot activities Some of this is focused on pilot activities, which are being undertaken by the project’s partners and their collaborators. The aim of these pilot activities is two-fold: they are inherently useful to the wider biomolecular community in that they advance the state of the art of biomolecular modelling and simulation, but they also feed back into the work of the centre so that 14
we can learn about the wider community’s interests, priorities and challenges.
Adam Carter, EPCC a.carter@epcc.ed.ac.uk
BioExcel pilot activities • Genomics workflows •F ree energy simulations of biomolecular complexes (including high-throughput ensemble calculations) •M ulti-scale modelling of the molecular basis for odour and taste • Molecular Recognition • Virtual Screening
Interest groups Another way that the project is interacting with the various existing communities involved with biomolecular research is through interest groups. We have founded eight groups to date and are working to build engagement with them. They are: • Biomolecular Simulations for Entry Level Users • Integrative Modelling • Best Practices for Performance Tuning • Free Energy Calculations • Hybrid Methods for Biomolecular Systems • Practical Applications for Industry • Training (for BioMolecular Research) • Workflows and Pipelines.
BioExcel supports face-toface meetings for its interest groups. It also hosts webinars, supports online discussion forums and can provide other online collaboration services such as chat servers and code repositories. New interest group members are always welcome. To join, see: bioexcel.eu/interest-groups Past webinars: bioexcel.eu/webinars Online discussion forum: ask.bioexcel.eu
BioExcel website: http://bioexcel.eu
NEXTGenIO: the next exciting stage begins!
Fujitsu’s factory in Augsburg where NEXTGenIO’s hardware will be built.
NEXTGenIO was one of several EC-funded exascale projects that we started work on last year. Here’s what’s been happening since it launched. The goal of NEXTGenIO is to develop a new HPC platform around Intel’s 3D XPoint™ nonvolatile memory. “Platform” in our case does not only mean the hardware, although that is of course a big part of it, but also the entire software stack that sits on top of the hardware. Our challenge is to think of how we believe this new memory technology should be used both in current and future HPC systems to alleviate some of the I/O performance bottlenecks that affect so many applications today. The past months have therefore been spent teasing out the requirements based on a set of usage scenarios. One of these scenarios is an operational run of IFS, the main forecasting model used at the European Centre for Medium Range Weather Forecast (ECMWF). Collectively, the forecast writes 7 million fields in a one-hour operational window. As the operational timetable is tight, even small gains in runtime have a big impact on the schedule. Within NEXTGenIO, the IFS output server will be adapted to exploit the non-volatile memory to avoid writing fields to disk within the time-critical window.
Another usage scenario is that of bioinformatics computations, a relatively new field for HPC. Bioinformatics applications are often characterised by a workflow where one application may read data then produce output, which is in turn consumed by another application. Reading and writing data to be passed between applications takes a long time, so reducing this cost is important. One key requirement, and an important driver for the architecture, is the need for legacy applications to “just run” on the new platform without requiring anything more than a recompilation. We have now completed our designs of the hardware, the systemware and the data architectures for NEXTGenIO, and the implementation phase – the exciting part – is under way. In July, the consortium had the chance to visit the Fujitsu factory in Augsburg where the NEXTGenIO hardware will be built. We were shown the motherboard design layout process (which is done by hand and takes 6 to 8 weeks) and all left with a real sense of progress being made!
The newsletter of EPCC, the supercomputing centre at the University of Edinburgh
Michele Weiland m.weiland@epcc.ed.ac.uk
Find out more If you are at SC16 and want to learn more about NEXTGenIO, come to the EPCC booth (number 701). You may also consider coming along to our “Exascale IO: Challenges, Innovations and Solutions” workshop. Check out the website for the full programme: www.nextgenio.eu/events/ workshop-sc-16-exascale-iochallenges-innovations-andsolutions 15
Adept: playing the power game Energy consumption is one of the most pressing challenges facing our world today – and the world of high-performance and highthroughput computing is no different. New, more powerful machines inevitably consume more energy than their less powerful counterparts, and require vast, complex energy infrastructures to keep them running. Not only is this bad for our environment, but the costs associated with running these machines are prohibitive for many potential users. With exascale on the horizon, research into energy efficiency is thus of great interest to scientific computing. The most powerful supercomputer in service today consumes 15.4 MW of energy, yet is not capable of even 1/10th of exascale speed (93 Petaflops/s). Scaling linearly, a theoretical machine based on the same technology, capable of 1 Exaflop/s, would consume approximately 165.6 MW. At the UK average electricity cost of 13.86p per kWh, this equates to an annual cost of almost £200 million – not at all feasible. This is purely the energy cost required to run the system, not including the cooling. Clearly, this is not sustainable. Current exascale guidelines suggests a total power envelope of around 20MW – only an extra 33% power budget compared to today, despite the speed of the system increasing by a factor of more than 10. The technologies involved will therefore have to significantly improve their efficiencies in order to build a sensible exascale system. The Adept project had the goal of understanding and addressing this issue. Running from 2013–2016, Adept was a three-year European16
funded project focused on energy efficiency in parallel software and hardware. One of the major outcomes of the project was a comprehensive tools suite to evaluate, measure and predict energy and power consumption across hardware and software. Two of the major components of the tools suite have been made available open-source.
Benchmark suite The suite is designed to test systems using simple operations to determine how efficiently they carry them out. There are several levels of benchmark with increasing complexity – from simple-operation nano benchmarks to full scale applications. All benchmarks also come in a number of common implementations so they can be used on a wide variety of systems. The parallel programming models OpenMP and MPI, but also less commonly used models such as UPC or Erlang. We have also implemented a wrapper library to read the values of the RAPL (Running Average Power Limit) counters on supported Intel processors. The benchmark suite is intended to characterise systems and works in
Mirren White, EPCC m.white@epcc.ed.ac.uk
We have gathered a large amount of knowledge within the Adept project, as well as the tools required to share this knowledge with others. The project may be ending, but the advances we have made could have significant implications for the highperformance computing community.
The Adept process. The tools suite takes a ‘fingerprint’ of the software under assessment. This profile is then used to assess how the software can be improved to reduce energy consumption.
tandem with the Adept Power Measurement system to understand how a specific system’s software and hardware are working together, in order to create a power profile.
Power Measurement System This is a sophisticated tool that implements a finely-grained power infrastructure to discover how every component of a system uses power over a given time. The system can take up to 1 million samples/ second, allowing us to see power variations that coarser-grained power measurement tools will not detect. This enables us to study the impact of even small changes in either software or hardware on power and energy consumption. We are also able to measure power from all components – the Adept system reads the current and voltage from the power lines that supply the components, such as CPU, memory, accelerator, or disk.
Performance and Power Prediction tool This tool is another major development and a natural followon from the Adept power measurement tools. It uses detailed fast profiling and statistical
modelling techniques to examine a software binary, allowing it to predict how well a CPU and memory hierarchy system will perform, and how power efficient it will be. It can be used even without access to the system or if the system does not yet exist. This gives us a powerful tool in the design of new systems, as designers can see exactly how a software application will run on a theoretical system before it is built. We have created a prototype tool that allows us to explore the design space for smarter, cheaper, and more efficient systems, because a system’s performance and power behaviours can be matched to a specific workload.
Our ultimate goal The Adept tools allow developers to make sensible decisions on new implementations based on real data, rather than guesswork, and allows owners of existing systems to see where their efficiencies can be improved. This could be updating a power-hungry component, or switching to a different, more efficient implementation of software. Efficient design of efficient systems – this is the ultimate goal.
The newsletter of EPCC, the supercomputing centre at the University of Edinburgh
The Benchmark Suite The Suite is open-source and available at: www.adept-project.eu/ benchmarks.html
The Power Measurement Infrastructure The Infrastructure is opensource and available at: www.adept-project.eu/powermeasurement-designs.html
17
EUROMPI 2016 EuroMPI 2016, the 23rd annual MPI users and developers conference, was held in Edinburgh in September. Coming to Edinburgh for the first time, and to the UK for only the second time after an 18-year hiatus, we were delighted to explore the current focus of MPI and how to shape it for the future. This year’s conference highlighted issues such as the current emphasis on future developments of MPI focusing on HPC simulation. The papers and panel discussed MPI’s potential benefits to big data analytics, ultra-scale datacentres, deep learning, and popular programming frameworks like Spark.
On day two of the conference a panel of MPI experts including Bill Gropp (NCSA, The University of Illinois at Urbana Champaign), Jonathan Dursi (Ontario Institute for Cancer Research), Martin Schulz (Lawrence Livermore National Laboratory) and Tony Sjkellum (University of Alabama) discussed these challenges.
One interesting question that the event did not have a clear answer to is why is MPI not already heavily embedded in these fields? Has the MPI community simply failed to market MPI well enough? Or is there missing functionality that is holding a wider community back from using MPI?
Topics that were discussed with no clear consensus included whether we need to break backwards compatibility, and how to encourage new communities to use MPI rather than re-inventing functionality that already exists.
The conference also highlighted that MPI I/O exposes correctness issues in popular file-systems and that this could be used to promote better system procurement specifications. MPI single-sided communication is well designed but (arguably) now needs to prioritise high performance implementations. There was also a consensus that it is now urgent that the community needs to devise a credible faulttolerance or resilience option for MPI. 18
Dan Holmes, EPCC d.holmes@epcc.ed.ac.uk
What is true is that MPI is here to stay, but if it is to remain relevant and useful, there are many questions that still need convincing answers. EuroMPI 2016 highlighted many of the flaws and failures of the current MPI community, but also its many valued attributes, helping to explain why it is still dominant. Many questions were raised that the community now needs to work on to address and we hope that the discussion will continue at EuroMPI/ USA 2017, hosted by Argonne National Laboratory in the USA.
MPI Forum The MPI Forum is an open group, with representatives from many organisations, which defines and maintains the MPI standard. http://mpi-forum.org/
WHPC: building careers and connections Women in High Performance Computing (WHPC) is gearing up for another busy event schedule at Supercomputing this autumn in Salt Lake City after a packed year of events. Our summer started with a birds-ofa-feather session discussing methods to improve diversity and our fourth international Women in HPC workshop, both held at ISC High Performance in Frankfurt, Germany. In September we ran our third annual UK event alongside the annual EuroMPI conference: during ‘An evening with interesting people’ we discussed careers in HPC, how to create impact from research (currently a hot topic in UK research), and ways to be a successful research software engineer. Our keynotes Kathryn Mohror (Lawrence Livermore National Laboratory) and Marie-Christine Sawley (from Intel, our keynote sponsor) discussed their routes into HPC and experiences in their careers. This event concluded with activities designed to actively build the networks of all attendees with speed networking, followed by a
dinner where everyone was encouraged to meet someone new!
Toni Collis, EPCC acollis@epcc.ed.ac.uk
Upcoming events Women in HPC at SC16
WHPC began with a question: “Why are there not more women in HPC?” Diversity • Birds of a Feather: How to build creates stronger organisations diverse teams for more effective and a stronger HPC research, Tuesday 15 November community. WHPC’s goal is to 2016, 5.15-7.00pm support the HPC community • Birds of a Feather: Women in HPC: Intersectionality, Wednesday and the women in the 16 November 2016, 12.15-1.15pm community to engage this generation of HPC users. For details of all of our events • Workshop: Diversifying the HPC Community
please visit: www.womeninhpc.org/events
To make sure you stay up to date with all of WHPC activities and get involved in the conversation, join for free by signing up at: www.womeninhpc.org/join Everyone is welcome to join, not just women, as this is about how to encourage women to work in HPC, not just for women in HPC.
The newsletter of EPCC, the supercomputing centre at the University of Edinburgh
Women In HPC www.womeninhpc.org 19
Software Sustainability Institute: news roundup The Software Sustainability Institute is a national facility for the users and developers of research software. It is led by EPCC in collaboration with the universities of Manchester, Oxford and Southampton and is involved in the organisation of various key events related to software in research throughout the UK. Here’s a review of its recent activities, and some upcoming events.
Research Software Engineers conference
Fellowship Programme 2017
With all the excitement about research software engineers (RSEs) over the last couple of years, we knew it was the right time to run a conference to bring them together.
The Software Sustainability Institute’s Fellowship Programme aims to fund fellows over 15 months to organise or attend events and workshops related to software and software sustainability, in exchange for their expertise and knowledge.
We attracted a lot of new people, but they were interacting like old friends. We worked hard to have an inclusive event, but I think this is also representative of people feeling a part of the community. As one of the emails we received said:“This might have been my 30th conference but it was the first where I felt thematically 100% at home and understood”. The conference attracted 202 people from 14 different countries. We received 38 talks, double what we could accommodate, and ran 15 workshops. We passed on expertise on subjects from Docker to “How to be a happy RSE” and most things in between. There were almost 450 tweets! To read the full article, please visit: http://bit.ly/RSEsblog
20
Our fellows come from all areas of research, and, once they are fellows, they keep that status even after their funding and regardless of where they are based. This year’s Fellowship Programme has now closed for applications and the Institute will announce the 2017 Fellows soon!
Collaborations Workshop 2017 (CW17) 27–29 March 2017, Leeds University Business School, University of Leeds The Collaborations Workshop series brings together researchers, developers, innovators, managers, funders, publishers, leaders and
Selina Aragon, EPCC and the Software Sustainability Institute selina.aragon@ed.ac.uk Just a few of the Institute’s Fellows.
Attendees of the first Research Software Engineers conference.
educators to explore best practices and the future of research software. This year’s theme is The Internet of Things (IoT) and Open Data: Implications for research. The Internet of Things (IoT) and Open Data are two of the hottest topics at the moment. Both already have, and will continue to have, a remarkable influence on our everyday lives. But what impact do IoT and Open Data have on research, how can researchers benefit from them and work towards making their data into Open Data, and where do Open Data and IoT converge? The Software Sustainability Institute invites all members of the research software community to explore these and other questions at the Collaborations Workshop 2017. CW17 aims to gain insight into the topics of IoT and Open Data and how these impact and will impact on research. It is also an ideal opportunity to form collaborations (on average, attendees start two new
collaborations) and to discuss topics proposed by delegates. For further information and to register interest in attending, see http://bit.ly/CollabW17
Software Carpentry and Data Carpentry Most of the training coordinated by the Software Sustainability Institute happens in collaboration with the Software Carpentry (SWC) and Data Carpentry (DC) initiatives. SWC focuses on software development skills (eg automating tasks with shell, structured code development, version control), while DC teaches research data management skills (eg caveats of working with spreadsheets, data manipulation with R, Python or SQL). Our next DC workshop will take place at the University of Oxford on 5 December 2016. For further details, visit: http://bit.ly/DC12-16
The newsletter of EPCC, the supercomputing centre at the University of Edinburgh
The Software Sustainability Institute cultivates better, more sustainable, research software to enable world-class research. It helps people build better software and works with researchers, developers, funders and infrastructure providers to identify key issues and best practice in scientific software. www.software.ac.uk
Software Sustainability Institute 21
AIMdays: matchmaking industry and academia The AIMday concept (Academic Industry Meeting day) matches a company’s need for new knowledge with the University’s academic expertise. EPCC recently attended AIMday Construction. AIMdays are intended to be an opportunity for companies to gain access to academic specialists capable of identifying solutions to specific business issues. Companies attending the event pose questions relating to company challenges which are then tackled by a group of academics during a one hour session. Working together, the group aims to identify pathways to solutions for the question with the goal of establishing future collaborative research projects.
AIMday Construction In October EPCC participated in the tenth University of Edinburgh AIMday. The AIMday Construction event allowed companies to meet interdisciplinary teams of academics with significant practical experience within the construction industry. These meetings provide an opportunity to explore and address the specific challenges each company faces, to find novel and innovative outcomes, encourage future collaborative working and 22
identify possible pathways to solutions. This was the largest AIMday hosted by the University of Edinburgh to date. A total of 19 separate workshops were held on the day with over 70 people attending. Transport Scotland presented on the Queensferry Crossing and Professor Masterton, from the School of Engineering, on the Crossrail project. EPCC has extensive experience of successfully using data management techniques to improve business processes, and we participated in several of the workshops, where companies asked questions such as how big data could help support project delivery processes and help make Edinburgh a Smart City. Participating in this AIMday offered us an opportunity to meet and engage with new companies and help them consider how HPC and Data Analytics can be part of solutions to their industry-specific challenges.
Carolyn Brock, EPCC c.brock@epcc.ed.ac.uk
The next AIMday will be on AMR & Infection, taking place on 30 November, 2016: http://aimday.se/infectionedinburgh-2016/ For further information about AIMdays visit: www.research-innovation. ed.ac.uk/AIMDay
Find out how EPCC’s expertise can help your business: www.epcc.ed.ac.uk/work-us
Image: Mario Antonioletti
Forthcoming courses We are one of the major providers of training in high performance computing in Europe, and frequently deliver training through our partner projects and organisations. See below for some forthcoming courses.
PRACE Advanced Training Centre
Software and Data Carpentry
EPCC is a PRACE Advanced Training Centre. 28 Nov–29 Nov: Single Node Performance Optimisation @ EPCC at University of Manchester 12 Dec–13 Dec: Writing Scalable Parallel Applications with MPI @ EPCC at University of Manchester
Future workshops are listed on each organisation’s website.
For the full list see: www.prace-ri.eu/upcoming-patc-events
Software Carpentry: software-carpentry.org Data Carpentry: www.datacarpentry.org All UK workshops are organised by the Software Sustainability Institute, of which EPCC is a founding partner: www.software.ac.uk
ARCHER training The following courses are all offered by ARCHER training on a regular basis. Dates will be published at: www.archer.ac.uk/training Introductory (level 1) courses Data Carpentry Hands-on Introduction to High Performance Computing Introduction to Modern Fortran Introduction to Scientific Programming with Python Scientific Computing Software Carpentry Intermediate (level 2) courses Data Analytics with High Performance Computing GPU Programming with CUDA Object-Oriented Programming with Fortran Message-Passing Programming with MPI Practical Software Development The newsletter of EPCC, the supercomputing centre at the University of Edinburgh
Shared Memory Programming with OpenMP Threaded Programming Advanced (level 3) courses Performance Analysis Workshop Advanced MPI Advanced OpenMP Efficient Parallel IO on ARCHER Efficient use of ARCHER and the Knights Landing Processor Programming the Manycore Knights Landing Processor Single-Node Performance Optimisation Single-sided Communications Writing Scalable Parallel Applications with MPI 23
Master’s degrees in High Performance Computing (HPC) and in HPC with Data Science From EPCC at the University of Edinburgh EPCC is the UK’s leading supercomputing centre. We are a major provider of HPC training in Europe, and have an international reputation for excellence in HPC education and research. Our two MSc programmes have a strong practical focus and provide access to leading-edge HPC systems such as ARCHER, which is the UK’s largest, fastest and most powerful supercomputer. Through EPCC’s strong links with industry, all students are offered the opportunity to undertake an industry-based dissertation project. The University of Edinburgh is consistently ranked among the top 50 universities in the world.* * Times Higher World University Ranking
Apply now
www.epcc.ed.ac.uk/msc 24