Controlsheet 30 (March 2017)

Control Sheet Cosylab’s Newsletter Volume 30

ISSN: 1855-9255

March 2017 Table of Contents

2

On System Architecture, Software Platforms and Products: Insights from our CTO

5

Particle accelerators as medical devices

9

This issue features the next in our Cosylab Experts series of articles: Klemen Žagar talks about system architecture approaches and control system software products.

Particle accelerator development processes are very different when the accelerator is used to treat patients instead of for scientific research in a lab. In the first case, the particle accelerator must become part of a certified medical device, with all that entails.

We give some insight into the nature of medical compliance.

IPAC17

Cosylab gives away 2 full-price tickets to IPAC 17!

The 8th International Particle Accelerator Conference will be held from 14 to 19 May in Copenhagen, Denmark. We are giving away 2 full-price tickets.

10

Transient Recorder Framework

15

How many physicists does it take to program a control system?

TRF is a software framework developed by Cosylab and SLAC that simplifies the integration of digitizers into EPICS. It is particularly attractive when different digitizer models need to be supported.

Is there a place for curious creativity, the kind you need to unravel the secrets of the universe in control system software engineering?

Cosylab d.d., Teslova ulica 30, SI-1000 Ljubljana, SLOVENIA Phone: +386 1 477 66 76 Email: controlsheet@cosylab.com URL: www.cosylab.com

page:

Volume 30

Control Sheet

2

ISSN: 1855-9255

Cosylab Experts

On System Architecture, Software Platforms and Products: Insights from our CTO This issue features the next in our Cosylab Experts series of articles. We turn to our CTO Klemen for his insights on system architecture approaches and software platforms choices for control systems both in scientific and proton therapy machines. He also compares other technical aspects between the two. Finally, we move on to the subject of software productization: how Cosylab witnesses this process unfold.

Control Sheet: Hi Klemen, please tell us a bit about yourself: how long have you been with Cosylab? What are the key projects you have worked on?

What is your current role?

Klemen: While I have a background in physics, I’ve been fascinated by computer science since the age of 10. Towards the end of my studies a researcher from the Jožef Stefan Institute asked me if I would join his team to work on the control system for an accelerator. I didn’t know much about accelerators at the time but over the years I’ve gotten many chances to Klemen Žagar learn quite a bit. This researcher was of course Mark Pleško, so I’ve been with In terms of projects I’ve been workCosylab from the very beginning. I ing on, at the beginning there was took on many roles along the way: in the control system for the Atacama addition to (control system) software Large Millimeter Array (ALMA), the development, also setting up the pro- project that gave Cosylab’s initial team cesses of our project management, the courage to start the company. We and quality assurance. As Cosylab built a control system framework for grew, the work entailed more system radio astronomy applications that is level responsibilities, and my job de- still quite useful today, e.g. for the Chescription effectively evolved from soft- renkov Telescope Array (CTA). ware architect to systems architect.

I always liked distributed computer Currently, I’m chief technology officer, systems, and so I studied EPICS and its and have been for about 4 years now. Channel Access Protocol. One of my Having been in a position to see most projects was documenting this protoof the technologies Cosylab has been col. working with from a closer or larger distance allows me to evaluate their usefulness, strengths and weaknesses when applied to new projects.

When CERN was about to commission the large hadron collider (LHC), they had a few reliability issues with the CORBA based control system in-

frastructure they chose. They contacted Cosylab, since we used CORBA for the ALMA control system too. We were able to help them out quite efficiently. It was a small but very interesting project I did together with Matej Šekoranja.

I got involved with the ITER project early on in 2008, with the first control system related work there. Among others, we developed EPICS device support and Linux drivers for National Instruments boards. We set up the development procedures, called ITER CODAC, which are being continuously improved.

Control Sheet: Before the start of coding in large projects, there is a lot of system architecture and design that takes place. How do you approach that crucial task, and how does one project differ from the next?

Klemen: The architect needs to make many design choices, and they tend to have large consequences. Therefore, you have to understand what needs to be built, what the overall use cases of the big physics facility are, and what the key challenges from the control system point of view are. It is not at all obvious how choices about the facility design will affect the control system. Some applications require tight real-time control loops – for example,

page:

Volume 30

Control Sheet

fast orbit correction in light sources or plasma control in a fusion reactor.

Another might have a need to quickly reconfigure the entire machine, for example sub-second reconfiguration of set points, which is needed in hadron colliders, where multiple beams exist in parallel. Another case is that of Particle Therapy machines, where energy levels are continuously changed during a 3D scan.

has a preference upfront, for example because they have skilled staff in a certain technology, and/or significant pre-existing code in that technology, then we advise them to proceed with that technology, unless there is an impending threat of obsolescence. The reason is twofold: avoid the high cost of the switch, but also avoid the risks introduced with the debugging cycle of a system written from scratch.

Based on the identified key requirements, you can start thinking about the technologies.

If there’s no such preference upfront, it depends on what kind of system they want. Some customers want great flexibility in their system, even up to entirely repurposing the machine in the decades to come. In this case, we’re looking at what I’d call a LEGObricks solution: you can recombine parts to build something else later, and the risk associated with an occasional brick temporarily toppling over is considered acceptable. For this you want a look under-the-hood, plus the ability to rewrite some parts later on.

In this case going for a scientific opensource framework is the way to go, and what you’re looking for is to use what your friends are using. If you’re building a synchrotron beamline in Europe, why go for EPICS when the people close-by that can help you out are using TANGO? And there are similar reasons to choose EPICS in other locations.

Firstly, the basic architecture of the control system: that tends to be very similar between projects. It consists of a 3-tier architecture with operator user interfaces on one end, the hardware front-end equipment interacting with the hardware on the other end and in the middle it’s a good idea to put a service layer to not overload front-ends with user interference.

Also, background services reside there – they relieve the user of many tasks and provide continuity of operation.

Secondly, there is the choice of the control system platform. Is this EPICS, or TANGO, or something custom? And what is the hardware platform for the front-end controller (FEC)? This last choice is quite important because there are a lot of FECs per system, plus you want reliable operation for many years: it should not just do its job well, you also want to be able to buy identical hardware 10 years from now, or at least have a vendor guarantee that he will notify you in time about suitable replacements.

Control Sheet: How do you choose the software platform? There are many opinions on the subject.

Klemen: Correct. Is EPICS better then TINE? How does OPC-UA compare to either? It has indeed been a subject of much debate. Our approach at Cosylab is quite pragmatic: if the customer

On the other hand, there are also accelerator applications where the robustness requirement is much stronger than any flexibility, and there is no need for a white-box approach.

Then it is a good idea to look at what the industry is doing. Even though they have for example looser timing requirements in typical automation projects (millisecond vs. microsecond response times), the software standards used there can be made to work in an accelerator context as well. Take OPC-UA: it’s similar to EPICS Channel Access, except that it was standard-

3

ISSN: 1855-9255 ized much later, about 2010. Also on the side of operator interfaces, you can take commercial tools like WinCC OA. Yes, it is not free, but with the price comes the support that you expect to have, so you’re not forced to take care of things yourself.

Control Sheet: In the previous Cosylab Experts article we asked Gašper Pajor about the differences between Proton Therapy and Big Physics projects from a project management perspective. Could you maybe tell us about similarities and differences at the technical level?

Klemen: The technical differences, for example the choice of hardware, are a consequence of differences in two conflicting key requirements: flexibility for future adaptations versus reliability.

As I’ve said, scientific projects desire a great level of flexibility in the software that can lead to an increased risk in meeting quality standards, maturity, or functional scope of the software.

In PT this is not an acceptable strategy: there are no compromises where patient safety is concerned. Hence, there is no trade-off to be done at the side of technical reliability.

That does affect the choice of hardware. In a scientific project, we see custom hardware, PCB, and custom FPGA development. Labs have a staff to do this type of one-of-a-kind work. Particle Therapy machine engineers prefer commercial, off-the-shell components with a managed lifetime: you want a strategy to replace faulty components fast, even years down the line. You cannot depend on the knowledge of one particular staff member, can you? Take the example of front end controllers with the cards from National Instruments.

Besides the reliability, they come with a software ecosystem that makes them quite suitable to do DAQ and

page:

Volume 30 front-end control. The flip-side of the convenience of the LabVIEW graphical programming paradigm is that in large scale applications the code base can get quite messy, hard to maintain.

That is, unless you do it correctly.

At Cosylab we have 8 years of experience with using LabVIEW in large project contexts. We have raised several experienced LabVIEW architects in the process, who were able to structure the software in such a way that you can do both: enjoy the benefits of LabVIEW, which is to quickly acquire analog and digital signals, process them, and make them available on the network. And you can also do it in a maintainable way. For example, as LabVIEW gets upgraded, you can, with minimal effort, get your system up and running again. We used all this knowledge to make a software product, a platform based on LabVIEW, called MADIE. It clearly defines and constrains the task of integrating a single device into the system in a way that prevents that software module from becoming bloated.

Control Sheet: That brings us to the topic of Productization. Productization can be expected when technologies move from pure R&D to commercial applications. What can you tell us about this process as witnessed by Cosylab, in the scientific and PT areas?

Klemen: Products are a nice thing: you write them once and you collect the money for all eternity, right? :) That’s just in theory, in practice you need to find a product that people are likely to buy. And for this, you really must understand the market. Cosylab has been exploring ways of productization for quite a long time in the scientific market, before engaging with it on a larger scale in the PT market.

In 2005 we introduced the MicroIOC hardware: a box that you simply connect to the network and it speaks EP-

Control Sheet ICS out-of-the-box, integrating various kinds of devices. We learned however that it is hard to market such boxes, especially in the scientific market, where people want full insight into the inner workings of the subsystems. So, we learned that with projects using MicroIOC hardware there will always be a large software-based service that goes with it.

Next, we saw that there are software components that are used repeatedly among customers. For example, device support for integrating devices into EPICS: there is a lot of boiler-plate code to be written in that process, tasks that need to be repeated simply because EPICS misses this layer with a sufficient level of abstraction. For this we developed a framework called Nominal Device Support (NDS).

Or, we noticed that people need to integrate large systems, and to do that, they need to maintain the configuration of the control system. While people solve this problem in different ways, with different kinds of databases, they are essentially solving the same problem. Here we also developed a product: the Device Control Database (DCDB). It allows configuration to be centrally stored, and for the configuration files to be generated for individual front end controllers.

In both of these cases, when people use these products, they have additional requests, so effectively they come to serve as a baseline. For example, ITER took NDS and is now using it for most of their device support in EPICS. DCDB concepts were used as a baseline for the IOC Factory at the European Spallation Source.

We can speculate about two reasons for “real” software products not materializing in scientific projects: 1) the projects differ too much from each other, or 2) the people involved prefer to have full insight and understanding

4

ISSN: 1855-9255 of the system they are building, and they believe the best way to achieve this is by doing it themselves.

Continuing our exploration of products we go to the PT market, where the situation looks different. Here we have a bunch of products in the pipeline.

I already mentioned MADIE, which is an EPICS-like framework for LabVIEW that allows the efficient development of a distributed control system with NI hardware on the frontends.

We also have a central timing system, CTS, a product built on top of Micro Research Finland (MRF) hardware. It integrates with MADIE, so essentially you take MADIE and CTS and you have the ability to have synchronized input/ output across your entire distributed control system.

To this we’re adding a generic waveform generator, FCCP, which allows people to drive waveforms in a synchronized manner, with the timing system dictating the pace at which the waveforms are played out across the entire system. This is a requirement for synchrotron-based PT machines where you need to drive the magnet power supplies and other devices in a synchronized manner.

The combination of these and other products into a product suite is a big timesaver in projects, as even with offthe-shelf software it takes a significant amount of time to properly integrate things, like the timing into the rest of the control system. We call the whole suite Cosylab’s ACS -- Accelerator Control System.

Control Sheet: Klemen, thank you for your time!

[1] MADIE Control Sheet Vol 28, page 6

page:

Volume 30

Control Sheet

5

ISSN: 1855-9255

Particle Accelerators as Medical Devices Marko Mehle (Cosylab)

Particle accelerator development processes are very different when the accelerator is used to treat patients instead of for scientific research in a lab. In the first case, the particle accelerator must become part of a certified medical device, with all that entails. We look at safety and effectiveness, regulatory requirements and medical compliance.

Special attention goes to software-intensive systems and their verification and validation. When software quality is in an organization’s DNA, applying medical processes is a natural as breathing.

Particle accelerators have many applications; one of these that is gaining popularity is their use in medicine for cancer therapy. Proton and Carbon Ion therapy machines are used to accelerate a beam of particles, and to deliver it very accurately into the tumor area.

This kind of treatment is called Proton, Ion or simply Particle Therapy, and it presents several advantages over classic radiotherapy.

The accelerator machines involved in particle therapy are fairly modest, both in power and complexity, compared to the machines used for experimental physics research; yet the distance between the lab and the hospital is great – especially in paper work. One of the tasks that need to be fulfilled is to demonstrate that the machine is safe and effective, as required by the standards and regulations. The purpose of this article is to present a starting point for understanding what it takes to make a Particle Therapy Machine that is compliant with regulations.

Medical Devices A particle accelerator, though complicated by itself, is not sufficient to conduct ion therapy on patients, and it cannot be considered a “medical device”. Why? Because an accelerator is a

machine that generates a raw beam, without means of controlling or quantifying its dose and position, and from a medical perspective, it’s useless and even dangerous. The system that we call a “Particle Therapy Machine” is composed of different subsystems; in general: one to generate the beam, one to carry and direct it to the treatment area (for example, beamline and gantry), one to measure and control its delivery (dose delivery/scanning), one to accurately position the patient (robot and X-ray imaging), one to control the operation and execution of the treatment (treatment control system), and various systems for control and safety. They altogether constitute a “medical device”. As such it will have to be approved prior to its placement on the market. It is clear, for example, that it does not make sense to market a power supply magnet or a “dose control system” alone as medical devices, since they do not have a stand-alone use in medical treatment. And even if its purpose as a component of a therapy machine is undeniable, as in the case of a dose delivery system, it is not possible to judge its safety or effectiveness without knowing the properties of the remaining components of the system. Nonetheless, quite often manufacturers of components and parts for medical accelerators deliver

products that are “certification ready”, meaning that the component meets all applicable standards and regulations, and that all necessary documentation is provided to the manufacturer of the Particle Therapy Machine.

Safety and Effectiveness There are two distinct and equally important aspects of medical devices that every medical device needs to fulfil: 1. It must be safe, 2. It must be effective.

It’s easy to understand why a Particle Therapy system needs to be safe: it is of no use to have a machine that cures cancer, when at times it behaves differently than how the doctor instructed it to, and hurts a patient. At the same time, it is important for it to be effective, i.e. it is capable of fulfilling its medical purpose. Again, a system that is safe but does not do what it is supposed to do, is useless. Imagine a wheelchair (also a medical device, though a simple one) that is tightly screwed to the floor: it may be as safe as it gets, but it does not allow the patient to move around, as it is supposed to.

That is why the laws regulating the

page:

Volume 30

Control Sheet

6

ISSN: 1855-9255

market of medical devices take care of these two aspects. These are addressed in different ways. The manufacturer must demonstrate that the device is safe and effective, as required by the country where the device is to be sold. To prove safety, a lot of documentation must be provided: extensive test reports, detailed design documents, compliance with applicable standards, and so forth. In contrast, effectiveness is proven either by doing clinical trials and investigations involving animals or humans, or by claiming that the device in question is similar to other devices already being used, and demonstrating it by pointing to scientific articles or other literature.

Medical…?

The regulatory requirements that a medical device or component needs to fulfil are different from country to country, but quite often they end up being quite similar, since it is common that countries around the world recognize the use of international standards as a valid (and often necessary) mechanism to comply with their (local) regulations. The components of a Particle Therapy Machine vary in operation and purpose. Some come into contact with the patient or are installed within the treatment area, some don’t. Some are mechanical and have moving parts, others are just electrical control and processing units.

Some use high power or voltage, others don’t. Some contain or consist of software only, some are fully analog and others partly digital. And the list goes on. There are lots of different regulatory requirements whose applicability depends on all of these factors.

But there are general things that are required for all “medical components”.

Regardless of the aforementioned technical factors, a Quality Management System (such as ISO13485) must be established, a Risk Management process (e.g. ISO 14971) must be de-

fined and executed, and the safety and effectiveness of the component must be proven.

…or Industrial?

Do all the subsystems or components of a Particle Therapy Machine need to be designed and manufactured according to the regulatory requirements for medical devices, or are there any possible exceptions? This is an important question, because building components according to medical standards increases the cost. The rule is that some of the components or subsystems may not need to comply with the applicable medical standards, as long as it is possible to demonstrate that their malfunction can never lead to an unacceptable risk, for the patient and for other people involved. Even in cases where one of the subsystems is inherently critical to the safety or the essential performance of the whole system, it is possible to mitigate the inherent risk by implementing alternative safety measures. The logic behind this approach is that we cannot really trust industrial (“non-medical”) devices since they were not designed

and manufactured following quality processes required by medical standards. Therefore, it is assumed that they can contain bugs or can fail at any time. The mechanism to assess the initial risk and the suitability of the chosen risk mitigation measures is the Risk Management, usually at the system level.

A common example from the Particle Therapy industry is to build the beam generation subsystem (the particle accelerator itself ) as an industrial device, and then to include in the system safety components, built as medical devices, that constantly monitor the beam parameters such as energy, current and position, and shut off the beam if any of these is out of tolerance. This strategy may help to save resources, but not always, since some complex industrial subsystems require safety measures so complex, that it is necessary to build a whole redundant instance of the original system, and this only to monitor the behavior of the original component. As an example, let’s imagine an ion therapy system in which, besides the particle accelerator, the system that measures

page:

Control Sheet

Volume 30 and controls the delivery of the dose to the patient is also a non-medical device. In that case, the only possible way to mitigate the risk of patient overdose due to a failure would be to have an additional medical component that measures and monitors the delivered beam. Such redundancy would be highly uneconomical.

That is why we recommend writing the medical compliance strategy very early in the design process. (For those of you more interested in this aspect, I recommend taking a look at the standard IEC 60601-1, section 16, and the informative Annex I).

Medical Software and Verification & Validation Today, every moderately complex medical device contains at least some software. It is known that software is very different in nature and behavior from hardware. Even complex digital hardware systems (composed of discrete logic) are not comparable to software. The two main reasons for

ISSN: 1855-9255

this are the complexity of software, with its enormous number of possible internal states, and the common software development process, which is more flexible and less constrained than hardware design. The first factor determines that it is practically impossible to test a software module in all its internal states, leaving room for latent bugs. The second factor also influences reliability. The nature of software allows for late changes, changes that can introduce unnoticed side effects.

So there is a risk of overconfidence on the side of inexperienced software teams near the end of the project. If these two aspects are not addressed properly, software cannot be considered reliable enough for medical devices, especially when there is a risk to people associated with it. That’s why the regulations and standards for medical devices take special care of software: to ensure that, although the possibility of bugs cannot be totally eliminated, their probability can be reduced and its associated risk mitigated; with good processes, proper testing, risk management and validation. There is an international standard

User Needs

that deals with this: “IEC62304: Medical device software -- Software life cycle processes.”

But software cannot exist without the associated hardware, and its proper operation in the context of the whole machine cannot be judged without testing them together. Yet, in a system that is as complex as a Particle Therapy Machine, there are many different subsystems or components that may contain software, and it is impractical or impossible to execute the full set of tests for every software module, all integrated and operating in the final machine. That is why this is never done in that way, and there is always a strategy to hierarchize and segregate testing into different levels, according to the system level architecture and the “V-model” (Figure 2), in order to make testing effective and practicable.

This is the purpose of the set of activities usually referred to as “Verification and Validation”: to Verify, at different system levels, that the build system works as specified, and to Validate that it is capable of fulfilling its intended purpose. More about verification and Validated Software Device

Simplified V-model User Requirements

System Verification

Verification test specification

Subsystem specification

Subsystem verification

Subsystem test specification

Component

Module or function

Figure 2: V - model

System Validation

Validation Plan

System Design

Component tests

Unit design

7

Component level testing

Software unit test

page:

Volume 30 validation of Programmable Electrical Medical Systems can be found in the standard IEC60601-1, section 14, and informative annex H.

Purpose of Processes and Standards Sometimes I hear, mostly amongst engineers and physicists, arguments about whether following the standards, regulations and processes can really enhance safety and effectiveness, or is it just a “placebo” to create an appearance of safety, which somehow relieves the engineering conscience from the responsibility of thinking. The argument is valid and it has to be taken seriously. I find that for some people, for example, passing a list of tests automatically means that the device has no bugs, or that the relentless application of risk management can give you a 0-1 type of indication about the degree of safety of a device. As a consequence of the previous misconceptions, there are people that do what is called “safety by paperwork”, which means tweaking the documentation and doing ‘assessments’ until it is

Control Sheet proven that the device is safe enough to meet the regulations. It is always possible to fool the system, and it is not in the scope of quality systems, processes and technical requirements to completely avoid it. Yet, what standards and regulations do above anything else is to establish clear respon- sibilities and minimum requirements, so – when things go wrong – no one can say “I didn’t know”.

On the other hand, it is important to understand that a good process helps you to keep organized through the development lifecycle. When projects are late or about to go over budget, there is a natural temptation to rush and skip steps. This may appear to save some time, but can also lead to dangerous mistakes. Having a wellestablished process naturally helps to resist the temptation. Furthermore, processes define what steps to take if there is an unexpected problem. And if the processes are wisely designed, they not only enhance safety, but even make the development more efficient by keeping pace, consistency, and making sure that no aspect or requirement is left neglected.

8

ISSN: 1855-9255

Conclusion Building a Particle Therapy system is a technical challenge by itself. To make it compliant with standards and regulations is yet another challenge, which is sometimes underestimated. Our experience shows that it is best to start designing the medical compliance strategy together with the machine itself, and to already have solid processes established when the development and implementation start. Smart processes not only help to design a safe device and get through compliance; they may also increase productivity and decrease the uncertainties and defects, which may be, at the end of the day, the differentiators for staying in the game.

ABOUT THE AUTHOR Marko Mehle has a background in Electrical Engineering and started working at Cosylab in 2011 where he has been involved with LabVIEW development and integration, FPGA development, hardware architecture, documentation and standards, for example on the MedAustron project. In his spare time, Marko writes (words and music) and plays the latter.

Y A W A GIVE Cosylab gives away 2 full-price tickets to the IPAC co nference! way is easy:

give-a Entering the Send

cosylab

ontrolsheet@ an email to c

The deadline Winners will

C TICKET

subject: IPA .com with the

:59 (CET)

pril 3, 23:59 is Monday, A

l 4 by email.

esday, Apri otified on Tu

be n

IPA C TI

CKE

T

Cosylab d.d., Teslova ulica 30, SI-1000 Ljubljana, SLOVENIA Phone: +386 1 477 66 76

Email: controlsheet@cosylab.com

URL: www.cosylab.com

page:

Control Sheet

Volume 30

10

ISSN: 1855-9255

Transient Recorder Framework Ambrož Bizjak (Cosylab), Ernest Williams (SLAC)

The Transient Recorder Framework (TRF) is a software framework developed by Cosylab and SLAC, which simplifies the integration of digitizers into EPICS. The framework targets digitizers that have one or more analog inputs, and a triggering system that initiates the sampling of the analog inputs. When digitizers need to be integrated into EPICS, different software frameworks could be previously used, such as the old EPICS “gtr” module [1], the AreaDetector framework [3], or the Nominal Device Support (NDS) [5]. However, all these approaches were found non-ideal; the “gtr” module is old and hard to extend, while the AreaDetector framework is not targeted at digitizers. NDS may be the most appropriate, but SLAC had clear requirements towards backward compatibility, aiming for preservation of all PVs that were used in a “gtr” system.

The new TRF is based on AreaDetector, but defines models and features that reduce the effort needed to integrate a specific digitizer. The AreaDetector framework and the “asyn” module [2], which it is built upon, were created

by Mark Rivers, whom we have consulted in the design of the TRF.

Introduction

also able to provide a number

Accelerators typically have a wide array of analog sensors, which provide important data about the operation of the accelerator and the experiment.

Such sensors are often connected to analog-to-digital converters (ADC) that convert the signals to digital values (samples). The ADCs are part of the digitizers, which provide the digital sample values to dedicated computer systems. The digital values are then processed, stored, and analyzed in order to provide useful information.

of samples preceding the trigger. Trigger events are also important because they can serve as a time reference for the acquired data.

However, not all data from sensors is relevant, and processing all data may even be impossible with limited computational resources. This issue is often addressed by processing only data associated with a specific trigger event. The trigger event can be a simple digital input signal to the digitizer, but digitizers have different capabilities with regard to triggering. Typically, a digitizer is able to acquire a specific number of samples after a defined trigger event, but some digitizers are

For the purposes of a software framework, it is important to understand the variety of hardware, and the different features and modes of operation that may be available. Therefore, a framework should avoid targeting very specific features of digitizers, but

provide the foundation and utilities that enable and simplify the integration of different digitizers.

The TRF has been developed for the SLAC National Accelerator Laboratory.

It is open-source software, and is available online [4] with its documentation. At this time, a driver is available for the digitizer General Standards 16AI64SSA/C (see Figure 1), but a driver for CAEN DT5730 is being developed, and one for SIS 3302 is planned..

Figure 1: The General Standards 16AI64SSA digitizer

page:

Control Sheet

Volume 30

Architecture The TRF is programmed in C++. It interfaces with digitizer-specific support code, which implements the low-level details of interacting with the specific digitizer model. We will call this digitizer-specific code the (TRF) driver. The TRF driver may itself interface with a lower-level driver such as a kernel driver or software library, or it may directly interact with the hardware (in particular on embedded systems).

The main component of a TRF driver is a C++ class, which extends the framework-provided class TRBaseDriver. The driver overrides specific virtual functions that the framework calls in order to perform specific actions; many of these will be described in the subsequent sections. The framework also provides many functions which the driver can call.

Figure 2 provides a graphical representation of the components of the TRF and a TRF driver, as well as components that these interacts with.

Because the class TRBaseDriver is derived from the class asynPortDriver of the EPICS Asyn module [2], the driver can define and manage regular Asyn parameters in order to interface with the EPICS database, as any driver based on asynPortDriver would. However, in many cases, it would be better to use TRF configuration parameters, which are described in a subsequent section.

The TRF and the driver together implement an external interface through which the digitizer is controlled, and through which the acquired data can be received. As in any EPICS integration project, this external interface is represented by EPICS Process Variables (PVs), which need to be defined in the EPICS database configuration files. The TRF provides database templates for PVs supported by the framework, but the driver needs to provide its own database templates for digitizer-specific PVs.

Figure 2: : Summary of TRF components and interactions

11

ISSN: 1855-9255

Arming and Disarming One of the most important functions provided by the TRF is managing the arming and disarming of the digitizer.

Arming the digitizer means to configure it to be ready to receive triggers and acquire data, while disarming means to bring it back to an idle state.

The TRF calls the driver-provided virtual function startAcquisition in order to arm the digitizer, and the virtual function stopAcquisition to disarm it. An important characteristic of these calls is that they are made on a thread internal to the TRF and specific to the digitizer instance, so they are permitted to perform synchronous operations. The TRF implements all the necessary synchronization with the external interfaces (for example, handling a disarm request that arrives while startAcquisition is still being called). The TRF also reports the current state of arming on the external interface, including a possible error indication in the case of an error during arming or acquisition.

page:

Control Sheet

Volume 30

12

ISSN: 1855-9255 solidating the management of these settings was believed to be beneficial.

Therefore, the TRF implements the concept of generic configuration parameters. The driver defines the specific configuration parameters that are supported and is able to query their values.

Generally, each TRF configuration parameter is represented on the external interface using two PVs: ◊ The desired value PV. This is the value which is set externally, and will become effective when the digitizer is armed next time. ◊ The effective value PV. This is the value that the digitizer has been armed with.

When the digitizer is not armed, it is an invalid placeholder value.

When the digitizer is about to be armed, just before startAcquisition is called, the framework will take snapshots of the desired configuration values and will call the driver-provided function checkSettings. In this function, the driver checks the validity of the configuration parameters. When the checkSettings function returns successfully, the framework will transfer the snapshot values to the effective value PVs, allowing users to see which settings the digitizer is armed with.

Figure 3: Main interactions between the TRF and the driver

Figure 3 shows the major interactions between the TRF and a TRF driver, including arming and disarming. Some of the interactions shown (such as the read loop) are described later.

Configuration Parameters All digitizers need to be configured appropriately before they are armed.

The possible settings will differ for different digitizer models, but con-

Within the checkSettings function, the driver can mark specific configuration parameters as irrelevant. If a parameter is marked as irrelevant, the TRF will not report its value in the effective value PV but the invalid value placeholder value. Using this feature improves the user experience because the values of parameters that are meaningless due to values of other parameters are not reported. For example, consider a configuration parameter “trigger period” which is meaningful only when another configuration parameter “trigger source” is set to “periodic” instead of “external signal”.

The TRF itself provides several configuration parameters which are likely to

page:

Volume 30 be useful for many digitizers and exposes them to the driver, such as: ◊ The number of samples to acquire after a trigger event. ◊ The number of samples to acquire before a trigger event (if the digitizer supports this feature). ◊ The maximum number of trigger events to process before disarming automatically. ◊ The sample rate for acquisition. This is not just a configuration parameter, because the driver must calculate the closest sample rate which the hardware can actually achieve. This feedback is provided to the user in real time as the sample rate is entered.

While many of these could be implemented by the digitizer driver itself, it is beneficial if the TRF directly supports the most common digitizer configuration, because this reduces the effort required for specific drivers and provides a unified external interface.

Submitting Acquired Data The driver is responsible for transferring (or arranging the transfer of ) acquired data from the digitizer into the main memory as well as converting and/or repackaging the data into a form which the TRF can accept. The driver fills in data buffers provided by the TRF and provides specific metadata, but from that point on is not usually involved in the processing of that data.

The TRF provides the class TRChannelDataSubmit which is used by drivers to submit data to the framework. The process of submitting data for a single channel is as follows: ◊ An instance of TRChannelDataSubmit is created (or an old instance is reused). ◊ The method allocateArray() is called, specifying the channel number, data type and number of samples. ◊ The method data() is called, which returns a pointer to a buffer, and the

Control Sheet acquired data is written to this buffer. ◊ The method submit() is called, specifying additional metadata such as an event identifier and timestamps.

As can be seen, the TRF hides many interactions with the AreaDetector framework from the driver and presents a simplified interface for submission of acquired data. Under the hood, these functions involve allocating a suitable NDArray object and sending it into the AreaDetector system for processing.

The TRF also provides database templates which can expose acquired data using PVs (waveform records), simplifying the integration of the software into a complete IOC application.

However, real applications will typically add different data processing using AreaDetector plugins, which may transform the data or calculate various statistics. The TRF does not itself need to provide support for any specific data processing.

Data Reading Loop In the previous section, it was explained how the driver submits acquired data to the framework, but not the context in which it does that. The framework actually provides two options: ◊ The driver submits data from its own internal thread(s). This is the most general option which should be able to accommodate any digitizer. ◊ Reading and processing of data is driven by the framework as part of the data reading loop. In this case, the framework repeatedly calls virtual functions implemented by the driver to read and process acquired data.

It is foreseen that the data reading loop will be used for the majority of digitizers and will simplify driver implementations. The remainder of this section describes the data reading loop.

The data reading loop runs on the

13

ISSN: 1855-9255 same thread that calls the startAcquisition and stopAcquisition functions, more specifically it executes between those functions. Compared to not using the data reading loop, this design absolves the driver from certain (but not all) synchronization considerations.

The data reading loop consists of repeated calls of the following driver functions (in the specified order): ◊ readBurst: In this function, the driver should wait for one set of data associated with a trigger event and read it into memory. The implementation of this function is very hardware-dependent. It will most often involve waiting for an interrupt signal from the digitizer that a trigger event is completed followed by transferring the data from hardware buffers to memory, but some devices support online transfer of data as it is being acquired. ◊ checkOverflow: In this optional function, the driver can check if there has been an overflow of the digitizer’s buffers indicating loss of data. If so, the driver returns to the framework the remaining number of trigger events which may be processed before acquisition is automatically restarted. ◊ processBurstData: In this function, the driver should submit the data that was read in the previous readBurst call, using the procedure outlined in the previous section. Additionally, the driver should call the function publishBurstMetaInfo to provide the metadata associated with the trigger event.

The reading loop makes few assumptions about the digitizer. Perhaps the most important assumption is that events are processed one by one and for all (active) channels at a time. It is because of these assumptions that the reading loop is practical for driver implementations.

The optional checkOverflow functions supports automatic restarting of acquisition after an overflow of the digitizer’s hardware buffers. Its semantics are targeted at digitizers that use

page:

Volume 30 a circular buffer for storing acquired data and where data after an overflow cannot be reliably processed, but buffered data before the overflow can be.

However some digitizers do not need to restart acquisition after a buffer overflow due to their design, and this feature should not be used with such digitizers.

When the digitizer needs to be disarmed, the reading loop must terminate. Note that it is expected that the reading loop spends most of the time in the readBurst function. To arrange for the termination of the reading loop, the TRF will call the virtual function interruptReading provided by the driver from another thread. The driver must then ensure that any ongoing readBurst function call returns as soon as possible and any potential future call returns immediately. The reading loop will exit as soon as readBurst returns, and the TRF will normally proceed with disarming the digitizer by calling the driver function stopAcquisition.

The read loop can also terminate automatically after a configured number of events have been processed; without the read loop, this feature would have to be implemented by the driver,

Control Sheet which would have to call the function requestDisarmingFromDriver when the requested number of bursts have been processed. The driver can generally call this function to initiate disarming irrespective of the reason.

Conclusions The TRF is a new software framework which simplifies the integration of digitizers into EPICS. It is built using modern programming practices and is specifically focused on digitizers instead of trying to be a generic device support framework. The TRF also standardizes important parts of both the external EPICS interface for using the digitizer as well as the internal interface with the driver.

These properties should make it an attractive option when considering integration of new digitizers, especially if different digitizer models need to be supported.

14

ISSN: 1855-9255

REFERENCES [1] EPICS GTR module: http://www.aps.anl. gov/epics/modules/analog/gtr/ [2] EPICS Asyn module: http://www.aps. anl.gov/epics/modules/soft/asyn/ [3] EPICS AreaDetector software: http://cars.uchicago.edu/software/epics/areaDetector.html [4] Transient Recorder Framework: https://github.com/epics-modules/ transRecorder [5] Nominal Device Support: https://github.com/Cosylab/nds3

ABOUT THE AUTHORS AmbroĹž Bizjak AmbroĹž Bizjak graduated with a degree in Computer Science and Mathematics from the Faculty of Computer and Information Science at the University of Ljubljana in 2013. He joined Cosylab in 2014 as a software developer and has primarily worked as a developer in a project involving a large medical device, but was also the main developer in several smaller projects such as integration and customization of EPICS archivers and integration of digitizers into EPICS. In his free time, Ambroz likes to work on open-source software projects and plays with 3D-printers; indeed he develops his own firmware for 3D-printers.

Ernest Williams is an engineering physicists and electrical engineer at SLAC National Laboratory, Stanford University.

He and his team specializes in advanced robust controls and signal processing for Embedded/real-time distributed systems.

In his spare time he enjoys hiking, basketball and the original star trek series.

page:

Volume 30

Control Sheet

15

ISSN: 1855-9255

How many physicists does it take to program a control system?

Matjaž Payrits (Cosylab)

Judging by the experience of Cosylab, about a third of the team working on the project seems to be a good proportion – the company is full of developers with physics degrees. Given that many Cosylab products control particle accelerators, this may not seem too surprising, but physicist-turned-software developer Matjaž Payrits believes the target market is not the main force driving physicists among our ranks. He argues instead that it is the creativity required for the job, the same kind of curious creativity involved in unravelling the laws of Nature, which he illustrates with his own experience of walking the scientific academic path that ultimately brought him to Cosylab.

During my university education, particularly the undergraduate degree, I recall several professors telling us students about the omnipotence of The Physicist on a nearly daily basis, and how an education in physics will get us virtually any job we want. It was too early to revel in this prophecy before passing the final exams, and it was also easy to brush it off as merely the professors soothing their egos while having to lecture for the umpteenth year in a row about a subject they had mastered decades ago. Nevertheless, I have learned, from my own experience as well as the relayed experiences of former colleagues, that the job market seems to genuinely share some of this sentiment. The most common arguments for physicists being sought after by everyone from insurance firms to logistics departments of supermarket chains seems to be our supposedly excellent problem-solving skills. And no matter how brilliant an individual physicist’s problem-solving skills may be, saying that it is those skills that a company is looking for in an employee often makes the job very attractive to the applicant with a physics background. It is my impression that many of us have grown addicted to the cycle of learning new theories, or subject areas in general, and apply-

ing them to solving novel problems.

There seems to be a fairly great chance of anyone seeking out problems of a technical nature to end up in software development. Society’s demand for software production makes the field potentially the largest treasure trove of technical problems and challenges of our time. Given the physicist’s fondness for those, it is then no wonder to find that many developers have a physics background. Cosylab is a prime example, with a significant fraction of physicists among our ranks, not least amongst its founders. Nevertheless, assuming for a moment that the archetypal physicist shares the preferences outlined at the end of the previous paragraph, merely being handed coding tasks is not enough to keep him or her happy. If the tasks are repetitive and require no additional learning, it is likely to have the opposite effect. The tasks have to be challenging and call for innovative solutions. And this, I believe, is the true reason why us physicists flock to Cosylab.

To get this message across more vividly, I would like to share more on the kind of environment I came from when joining Cosylab. After finishing an undergraduate degree in physics in my hometown of Ljubljana, I en-

rolled in a Masters course, Quantum Fields and Fundamental Forces, at Imperial College London. As the course’s title might suggest, I was yearning to learn about the universe’s most meticulously concealed secrets and what makes it tick. A large part of the course was dedicated to theories of quantum gravity, particularly string theory, and their unification with the standard model of particle physics. I deepened my love of the scientific method during that time, but began to feel that the cosmological context, in which the above theories become relevant, was not for me. I decided to stay at Imperial College and do a theoretical PhD in “cold atoms,” a subfield of condensed matter physics. I found myself much more at home here as the relevant experiments could be done right here on Earth. I also grew to appreciate its technological applicability.

One of the reasons the field has caught on very well and became a hot research area, despite its name, is that it resolved some of the difficulties in studying quantum effects in materials. Real materials are marred with imperfections, often making it impossible to disentangle the physics of their primary structure from the physics of disorder in experiments. On

page:

Volume 30 the other hand, quantum phenomena are notoriously difficult to simulate on a classical computer, as the required memory grows exponentially with particle number, and simulations are often feasible only up to several tens of particles. Cold atoms, on the other hand, allow experimenters to create an array of clouds of exotic quantum states of matter, either a Bose-Einstein or a fermionic condensate, nowadays also called the fifth and sixth states of matter, respectively, and make them interact with each other in a manner that closely resembles particles in materials, e.g., electrons and ions in metals. Furthermore, using what is called an optical lattice, a special arrangement of self-interfering laser beams, one can create a perfect crystal structure, without any defects, on a much larger, more experimentally accessible scale. The field owes a lot to the advances in cooling technology from the final decades of the 20th century, which ultimately also gave it its name - the atomic clouds have to be cooled down to temperatures of microkelvins or less.

The majority of my PhD consisted of theoretically investigating properties of Bose-Einstein condensates of a particular kind of atoms. In quantum mechanics, particles have a property called spin, which in many ways behaves as the angular momentum of a rotating classical rigid body, though it usually cannot be changed. The earliest Bose-Einstein condensates were realized with atoms of zero effective spin. I was investigating condensates composed of spin-2 particles, which would offer experimenters even more versatility. The problem of describing their behaviour in the presence of a particular experimentally relevant type of magnetic field had proven particularly elusive, but building on a clever mathematical trick my supervisor had previously come up with in a simpler setting, we managed to

Control Sheet extract meaningful analytical results.

The trick was based on the observation that the behaviour of confined spin-1 condensates can be “mapped,” in a rigorously defined way, to that of a single particle living on the surface of a sphere. The mapped particle feels a force field that depends on the electromagnetic fields across the condensate. For spin-2 particles, this approach still works, but now the mapped particle does not live on the surface of a regular sphere in 3-dimensional space.

Rather, it moves across the surface of an abstract 4-sphere, embedded in 5-dimensional space. Using rather abstract and, in my mind, elegant mathematics, boiling down to Gaussian integrals over a pair of coupled 4-spheres, to obtain a measurable number that had stubbornly evaded all other methods of calculation, felt like cheating reality itself and was one of the most satisfying experiences of my life.

Calculating such a number is still a long way off from using a cold atomic system to simulate a superconductor, though potentially necessary. I see basic research as a noble endeavour, since a lot of it is immensely significant and may go on to become the essential cog in the power plant of the future or the next generation of proton therapy machines, even though the role of such cogs is rarely recognized. After finishing my PhD, I felt a pronounced need to distance myself from such basic research and go on to create things that I can eventually see fall into their place, but that will hopefully still benefit society. Finally, I was well aware that I could not do much if the problem-solving needs of my inner physicist were not met.

These desires brought me to Cosylab.

As a control systems software architect, I interact with complex systems on a daily basis, am kept captivated by a constant stream of new hardware,

16

ISSN: 1855-9255 that needs to be incorporated into projects, am motivated by a company that highly values its employees and allows them to constantly expand their development knowledge and skills, and I am, after all, contributing to the functioning of proton therapy machines, treating cancer one irradiation at a time. I believe Cosylab’s scientific origins are one of its strongest assets, as it has allowed the company to aim for and attract nothing but passionate employees, and to maintain, if not deepen, this level of passion for their work. The result, I trust, is software, crafted with care, whose creators are not afraid to take responsibility for it, and that runs well because behind every line of code there isn’t merely an amorphous corporate entity, but an enthusiastic, motivated, and proud developer. I would find it very hard to describe my own experience in any other words.

ABOUT THE AUTHOR Matjaž Payrits, PhD, is a Software Developer at Cosylab. He joined the company after finishing his doctoral studies in Mathematical Physics at Imperial College London. After four years of getting his kicks from integrating over high-dimensional Lie groups to calculate ground-state magnetizations of spinor Bose-Einstein condensates, he decided to pursue his other passion of crafting beautiful compiled code, which he views as an immensely creative process, while still supporting and staying close to his beloved mistress, Science.

page:

Volume 30

Control Sheet

17

ISSN: 1855-9255

On the lookout for software developers!

A large part of our work at Cosylab involves software and ranges from pure high-level application development (in our domain) with Windows or Linux, all the way down to driver development. At Cosylab you can really put your programming skills to the test. Having the best as your co-workers, you practice quality controlled software engineering or programming-in-the-large.

We are always on the lookout for people who: ◊ are ambitious, responsible and results oriented, ◊ are fluent in English, ◊ have exceptional written, verbal, and interpersonal communication skills, ◊ are able to work as part of a team to solve complex technical problems, ◊ are willing to travel.

We offer: ◊ opportunities for programming in various technologies, like Java, C#, C++, VHDL and LabView, ◊ development of high level applications in Windows or Linux environment, ◊ driver development, ◊ working with control system frameworks, like EPICS, Tango, FESA, etc., ◊ continuous education that upgrades the knowledge gained in college, ◊ long term employment with a one year probation period, ◊ flexible working hours, ◊ possibility of a successful career path with a competitive salary!

We are looking for different experienced levels so: ◊ If you’re either a graduate in computer science, physics or electrical engineering and you have no work experience yet, but you like programming and you’re good at it ◊ or if you already have what it takes and you’re ready for the new challenges in the world of Big Physics

Get in touch with us and join one of Slovenia’s best software companies!

Please send your letters and CVs to jobs@cosylab.com marked “Software Developer”.